Enterprise AI
Reference Architecture

Not all problems benefit from AI. Before choosing an architecture, architects should first classify the problem being solved. Four classes of problems consistently benefit from AI-powered runtimes.

Four Classes of AI Problems

Identifying an AI-friendly problem class does not automatically mean AI should be used. The next question is whether the problem characteristics justify a runtime AI architecture.

Decision Matrix: Which Problems Are AI Problems?

Enterprise AI systems introduce uncertainty, runtime costs, and non-deterministic behavior that traditional software architectures rarely encounter. These principles provide the foundation for designing systems that remain reliable, observable, governable, and economically sustainable in production.

The accompanying diagram depicts the end-to-end workflow of an enterprise agentic AI system, encompassing actor input, context enrichment, model invocation, and response evaluation.

AI prototypes are typically developed for rapid iteration, which is appropriate for initial validation. However, as a prototype transitions to production, the focus shifts from assessing response utility to ensuring the system can be secured, evaluated, governed, monitored, and trusted within an enterprise workflow.

A prevalent misconception is that linking a chatbot or agent to a model constitutes an agentic AI solution. While incorporating retrieval and memory enhances grounding, these features alone do not ensure production readiness. The system must also implement controls to validate inputs, assess context quality, evaluate outputs, and monitor for drift over time.

What Changes in Production

As the system moves toward production, the architecture must do more than just send prompts to a model. It needs to control the full interaction lifecycle:

• Before a request reaches the model: Check whether the input is safe, well-formed, and allowed
• Before the model generates a response: Ensure that retrieval and enrichment provide relevant and trustworthy context
• Before the response is returned: Evaluate the output for structure, safety, accuracy, policy compliance, and action eligibility

Key Insight: Production agentic AI extends beyond a simple chatbot or event-to-LLM gateway-to-model pipeline. It necessitates a managed control layer to validate inputs, assess context quality, evaluate outputs, monitor system behavior, and establish feedback loops for drift detection and operational alerts.

Improvement is contingent upon measurement. In production AI systems, validation should extend beyond simple pass/fail checks. The system must capture signals such as input rejection frequency, context quality deficiencies, output validation failures, and changes in these patterns over time.

A production-ready agentic system needs runtime signals from three control points:

Gateway Metrics (Tracked Separately)

Gateway metrics should be tracked separately because they reflect operational health rather than application-level reasoning quality:

Drift and Alerts

The primary value of these metrics is in detecting trends. While isolated validation failures may be tolerable, significant increases in prompt injection attempts, stale retrieval results, malformed outputs, unsafe recommendations, or human rejections warrant investigation. Such issues may originate from user behavior, upstream data quality, prompt modifications, model upgrades, retrieval adjustments, outdated documents, or changes in business processes.

The AI Gateway is the single point of entry for all model interactions. It provides unified access to multiple model providers while enforcing enterprise policies.

Core Responsibilities

• Model routing and load balancing across providers
• Rate limiting and quota management
• Request/response logging and audit trails
• Cost tracking and budget enforcement
• Fallback and retry logic across providers
• Prompt injection detection and filtering

Design Considerations

The gateway should maintain statelessness and support horizontal scalability. It must accommodate provider-specific authentication, normalize response formats, and manage streaming responses. Implement caching for identical prompts when appropriate.

The orchestration layer manages agent lifecycles, coordinates multi-agent interactions, and enforces workflow constraints. This is where deterministic control meets probabilistic reasoning.

Orchestration Patterns

State Management

Agent state should be externalized and persistently stored. In-memory state is suitable only for single-turn interactions, whereas multi-turn conversations, long-running tasks, and recoverable workflows necessitate durable state storage solutions.

The memory layer provides agents with contextual awareness across interactions. Memory is what transforms stateless model calls into coherent, context-aware systems.

Memory Types

Implementation Considerations

Memory retrieval should be both rapid and relevant. Employ vector similarity for semantic search, supplemented by metadata filtering to enhance precision. Implement memory decay and consolidation strategies to manage data growth.

The knowledge layer provides agents with access to enterprise information through retrieval-augmented generation (RAG) and structured data access. This is how agents ground their responses in organizational truth.

Knowledge Sources

• Document repositories (policies, procedures, manuals)
• Structured databases (customer records, inventory, transactions)
• Knowledge graphs (entity relationships, taxonomies)
• Real-time data feeds (market data, sensor readings, events)

Retrieval Architecture

Adopt a hybrid retrieval approach that integrates dense vectors for semantic similarity with sparse vectors (BM25) for keyword matching. Utilize re-ranking models to enhance precision, and consistently include source attribution to ensure traceability.

Tools extend agent capabilities beyond text generation. The tool integration layer manages tool discovery, invocation, authentication, and result handling.

Tool Categories

Tool Governance

Every tool must have a defined schema, permission requirements, and risk classification. Implement tool usage policies that can restrict which agents can invoke which tools under what conditions.

The evaluation layer continuously monitors agent performance, response quality, and overall system health. This process verifies whether the system operates as intended.

Evaluation Dimensions

• Task Success: Did the agent accomplish the stated goal?
• Response Quality: Accuracy, relevance, coherence, safety
• Efficiency: Token usage, latency, cost per interaction
• Safety: Harmful content, policy violations, data leakage

Evaluation Methods

Integrate automated evaluations, such as LLM-as-judge and rule-based checks, with human assessments to achieve nuanced quality evaluation. Develop evaluation datasets that mirror production scenarios, and conduct evaluations following every model or prompt modification.

Observability in AI systems extends beyond traditional application performance monitoring. It is necessary to trace the reasoning process in addition to the execution path.

Observability Pillars

• Traces: End-to-end request flow including all agent steps, tool calls, and model invocations
• Metrics: Latency distributions, token usage, cost, error rates, tool success rates
• Logs: Structured logs for every decision point, including prompts and completions
• Reasoning Trails: Chain-of-thought captures, decision explanations, confidence scores

Privacy Considerations

Observability data frequently contains sensitive information. Implement mechanisms for personally identifiable information (PII) detection and masking. Establish data retention policies and ensure the security of audit logs.

Not every agent action should be autonomous. The human approval layer defines when and how humans are brought into the loop for oversight, approval, or intervention.

Approval Patterns

Escalation Paths

Establish explicit escalation paths for scenarios involving agent uncertainty or policy violations. Implement timeout mechanisms for pending approvals and provide sufficient context to enable informed human decision-making.

Enterprise AI systems operate under regulatory, compliance, and security requirements that cannot be compromised for capability or speed.

Security Controls

• Input Validation: Prompt injection detection, input sanitization, content filtering
• Output Validation: PII detection, policy compliance checks, toxicity filtering
• Access Control: Agent-level permissions, tool-level authorization, data access policies
• Data Protection: Encryption at rest and in transit, data residency compliance, retention policies

Governance Framework

• Model inventory and version control
• Prompt management and change control
• Usage policies and acceptable use guidelines
• Incident response procedures for AI-specific failures
• Regular audits and compliance reporting