This time last year, we built an MCP server connecting Claude to our data lake. It was an internal experiment but we started to see huge amounts of value showing up immediately. Teams across pricing, underwriting and customer success could suddenly answer customer queries and prepare for meetings by just asking questions of the data in plain English. We started to think about the application and value it could bring to the customer and Jay came to life.
This is the first in a series of blogs about how we built it. We'll stay at the high level architecture here, with a follow-up post coming soon that goes much deeper with full architecture diagrams, the technology choices behind them, and what we learned building it.
What is Jay?
Jay is an agentic AI assistant embedded in Flock's fleet portal. Fleet managers ask questions in plain English about their fleet's safety performance: trips, risky driving, speeding, fatigue, telematics connectivity, claims and Jay answers with live data, inline charts, interactive trip maps, and downloadable reports.
Why build an agent instead of another dashboard?
Flock sits on top of an extremely rich data asset: telematics data and claims data. Like most software companies showing data and analytics to customers, we have traditionally done this through dashboards, notifications and static charts.
We have always had a wide ranging customer base, and those customers want very different things from their data. For product teams, that made it hard to know what to build and what to prioritise, with the huge variety in user needs and customer pains.
Customers do not have the time to be scrolling through dashboards and spending hours on portals, they need solutions that fit into their everyday jobs and ways of working. Building an agent has given customers the power to get what they need out of their data in the simplest and low touch way possible.
How does Jay work technically?
Jay runs as a containerised agent service on Amazon Bedrock AgentCore Runtime, built on the Vercel AI SDK for the model loop and streaming. Claude Opus 4.7 does the reasoning, called through a cost-tracked, cross-region EU inference profile; a cheaper Claude Haiku 4.5 handles the lightweight side jobs, conversation titles, suggested follow-up prompts. The expensive model is reserved for turns that need it.
Jay is built from four core pieces:
- Data tools. ~20 MCP tools (and expanding), served by a dedicated MCP server that sits between the agent and Flock's internal fleet-data services, covering fleet and vehicle safety metrics, trip search, connectivity and policy information. We kept this as a separate server rather than building the tools straight into the agent, so the same data tools can be reused by other agents and interfaces down the line, not just Jay's chat window.
- Rendering tools. The model emits structured payloads that the frontend materialises as real UI: six chart types, interactive trip maps and CSV, markdown and JSON downloads. Customers see their data in whatever form suits the question, not whatever form we pre-built.
- Skills. Detailed domain playbooks for things like trip investigation, safety scores, fatigue, connectivity and claims. These are stored as markdown and loaded on demand via a
get_skilltool, so the base prompt stays lean and cache friendly. - Memory. Long-term, per-user memory (semantic facts, running summaries, stated preferences) persists across conversations, so Jay can pick up context (a named vehicle, a preferred unit system) without the customer repeating themselves. It's scoped per user and switched off entirely for internal staff sessions, and it's the foundation for the personalisation work described below.
A question, start to finish. A fleet manager asks something in the chat. Claude decides which of the data tools it needs, calls them, and streams the answer back as it's produced: text, a chart, a trip map, rather than waiting for everything to be ready before showing anything. If a question needs several tools, they get called in sequence as Claude reasons about what it's learned so far, not from a fixed script we wrote in advance.
What does production grade mean?
Outside of the carefully thought through design of the agent, from authorisation, to tool scoping to memory, we followed rigorous evaluation, observability and red-team testing. This is essential in order to deploy an agent into production that has access to sensitive customer data and includes the following:
- Authorisation the model can't touch. Every request carries the customer's own auth token, verified twice: once at the agent runtime's front door, once again in-process. That same token is forwarded on to the MCP server and the underlying fleet-data services, so every data access is enforced by the same service-level checks as the rest of the platform. The model itself holds no elevated access of its own; it can't expand its scope by asking nicely, or through a prompt injection. Each conversation is hard-scoped to a single policy.
- Guardrails against the agent itself misbehaving. The tool-calling loop is capped, and a repeated-call guard stops Jay if it ever tries the same tool call over and over, so a confused agent fails safely and visibly, instead of looping silently.
- Prompt-level guardrails. An explicit cannot-do list (no disciplinary recommendations, no claims actions, no pricing), backed by a red team eval suite covering tenant isolation, prompt injection and jailbreaks, system prompt extraction, impersonation and hallucination.
- Evals on every change. Quality is enforced with Promptfoo suites run from CI against live deployed endpoints: smoke tests, then a ~50 case golden suite, then red teaming. An LLM judge parses the actual response stream and verifies which tools were really called, so "sounded right but made it up" fails the eval.
- Full observability. Every invocation is a Datadog LLM Observability workflow span, tagged with session, user and policy, with full token and prompt-cache accounting per turn. We can inspect cost and behaviour conversation by conversation, and we do.
The loop that keeps improving Jay
Our aim is for Jay to be a truly agentic system, and the feedback loop is where that starts. With each captured conversation, we run analysis on what customers asked and how Jay responded.
That lets us quickly identify gaps in features, capabilities and tools. Those gaps become fully specc'd Linear tickets, picked up by our dev team for now, and soon by agents.
We have released 15 updates to Jay through this loop so far, and we only expect that pace to increase as our approach to agentic software development matures.
Where can we take this?
We are working on hyper-personalisation for each customer (building on the long-term memory already in place) and on Jay taking more proactive action beyond the chat window.
That's the high level picture. The next post in this series goes much deeper: full architecture diagrams, the technology choices behind every piece above, and the technical learnings, good and bad, from getting Jay into production.