Back to Writing
Agentic AIMay 18, 202514 min read

AI Agents: Our Experiments, Key Implementation Techniques, Security Challenges & Observability Solutions

A hands-on field report: four agent implementation patterns (tool-calling, LLM-with-tools, MCP, MCP server), the top security threats, observability with OpenTelemetry, and a Level 1–5 agent maturity model built around a CRM automation agent.

Co-authored with Milind Chandramohan

Agentic AIMCPAI SecurityObservability

Beyond the chatbot: AI agents offer unprecedented autonomy — but building, deploying, and trusting them means sifting through a maze of ideas and technology: toolchains, protocols like MCP, and multi-agent (A2A) complexity. We decided to dive in.

For this discussion, “agents” refers to standalone decision-making systems that leverage LLMs. The emergence of LLMs as drivers of autonomous agents came from advances in stronger reasoning, instruction following, tool-use rationale, and code generation. Interestingly, the original ideas for autonomous software that learns by observing are 30+ years old — Pattie Maes's early-1990s work on adaptive autonomous agents defined much of what we still chase today.

Techniques of agent implementation

We looked at the four most popular patterns for building agents.

Diagram
The four most common patterns of agent deployment. The orchestrator still exists in patterns 2–4 — LLMs are reasoning engines, not execution environments.
  • Tool-calling with an external LLM — a custom tool-call chain implements the logic and integrates enterprise assets and LLMs. The orchestrator retains most of the business flow. Common for many RAG and classical AI use cases.
  • LLM with tooling — the LLM drives functionality, reasoning about a sequence of actions and using provided tools. Each enterprise endpoint gets custom tooling so the LLM can discover and use it.
  • LLM with MCP — every enterprise asset is MCP-enabled and exposes an MCP endpoint; the LLM discovers and uses them directly.
  • LLM with an MCP server — a middle ground where an MCP server, like an API gateway, exposes endpoints and abstracts the complex integrations behind it.

What we learned comparing them

  • Given an example dataset, the LLM can easily identify the right content to send to a tool. Direct tool embedding gives control but isn't architecturally clean; MCP adds necessary indirection and abstraction.
  • Google ADK is easy to install and ships with a runner and web interface to chat with the agent.
  • LangChain enforces constraints that provide more “compile-time validation,” making correct agents easier to build.
  • MCP development is not straightforward — releases are new (0.x), and with FastAPI we hit version incompatibilities that took many attempts to resolve. Writing a correct agent prompt also took several iterations.

Security of AI agents

As agents gain autonomy, take more actions, interface with more entities, and learn from history, security concerns escalate. Grouping the OWASP Agentic AI threats with other sources, we picked the top three.

Prompt injection

Most successful threats fool the LLM into interpreting a data payload as a prompt. These can get sophisticated — adversaries using LLMs to craft malicious prompts, or poisoning the vector databases behind a RAG system.

Malicious tools and data leakage

With MCP and A2A, agents may select partner tools and agents at runtime, making it easy for an attacker to pose a malicious tool as legitimate — leaking sensitive data or performing unauthorized actions. Hallucination and misunderstanding make this worse.

Model stealing

Treating the agent as a black box, an attacker sends inputs, observes outputs, and trains a surrogate model via knowledge distillation. A serious problem in LLMs generally, though less clearly so for agents.

Reducing risk

  • Explicitly separate an agent's data inputs from its prompt/instruction inputs; structure and preprocess incoming data to prevent injection and poisoning.
  • Register agents and users to enable tracing and non-repudiation — an agent gateway that validates the source.
  • Leverage sessions (stateful and contextual, like web sessions) with authentication and access controls to limit what a session can access.
  • Use a multi-agent shield — a set of models critiquing outputs — or a cheaper dedicated filter model to detect malicious queries before they reach the agent.

Observability of agents

Like ML and LLM solutions, agentic systems need monitoring, debugging, and operational oversight — especially in multi-agent situations where interactions can't be known beforehand. We chose LangFuse, and favored tools that piggyback on OpenTelemetry (LangFuse, TraceLoop's OpenLLMetry). Most enterprises already run OpenTelemetry-compatible infrastructure, so compatibility matters more than raw feature count.

An illustrative example: a CRM automation agent

To demonstrate increasingly complex agents, we built CRM agents with Python, Google ADK, LangChain, Ollama, Llama3/Gemma/Gemini-Flash, and HubSpot. We classify agents from Level 1 to Level 5 by problem/solution complexity and autonomy.

  • TaskBot (Level 1) — automates simple, rule-based tasks. Our CRM client agent reads emails from potential leads, extracts relevant data, and calls tools (create_lead(), create_meeting()) to update HubSpot, then records the lead in Postgres. We built two versions: direct tool-calling via Google ADK, and the same via a LangChain agent using an MCP server.
  • FlowBot (Level 2) — a multi-step flow: beyond entering the lead, it finds a good time on the lead's calendar and can gather more data, transcribe calls, and add sentiment to the CRM.
  • InsightBot (Level 3) — plans a strategy per lead, does background research, and proposes a plan for human approval before executing. An ambitious stretch goal.
  • NeuroBot (Level 4) and AGI (Level 5) — increasingly autonomous agents that coordinate across teams, learn and adapt, and (theoretically, at Level 5) run end-to-end with full autonomy.

Further topics to explore

  • Cost analysis — agents trade speed, agility, and elasticity against the risk of bad decisions and computational complexity.
  • Ethics and responsible AI — a topic of enormous complexity with no black-and-white answers.
  • Human-agent interfaces — how humans build trust in and collaborate with agents.
  • Role of SLMs — smaller, targeted models can cut cost, latency, and footprint for routine operations.
  • A2A complexity — multi-agent systems amplify classic distributed-computing challenges: consensus, Byzantine faults, stabilization.
Many design principles of building systems apply directly to agent-based automation. Should one agent do the whole workflow, or many cooperate — and how? (Compare orchestration and choreography in microservices.)