Context
I work at a crop insurance carrier. Answers to everyday questions live in long policy documents, guidelines, and HR material, and finding the right paragraph takes time.
The assistant lets employees ask those questions in plain language and get an answer with links to the documents it came from. Under the hood it runs on AWS Bedrock, searches company documents stored in Postgres, and is built with LangGraph (an approach known as retrieval-augmented generation, or RAG).
Expanding what it knows
The first version covered two subject areas, each handled by its own specialist AI agent, with a coordinator agent deciding which specialist should answer. I expanded it to six subject areas, each with its own set of documents, and built the logic that decides which ones a question needs.
I also co-built the HR area end to end: preparing the documents, loading them in automatically from cloud storage (AWS S3), and connecting it to the rest of the assistant.
The redesign
With a coordinator in the middle, every question took extra steps: the coordinator planned, handed off to a specialist, and the specialist searched its documents. That made sense when AI models struggled to choose between many sources on their own. Newer models don't, so the extra layer mostly added time and cost.
I redesigned it on my own as a single assistant that searches all six document sets directly. It shipped alongside a model upgrade, so the testing process had to show quality held steady through both changes.
Measuring quality
We keep a set of 120 test questions with known-good answers, plus an admin screen for reviewing results. An AI grader scores every run on whether the answer is correct, whether it used the right documents, and how close it is to the expected answer. Every model or design change has to pass before it ships.
Results
The redesign and the model upgrade shipped together, so these results reflect both.
| Measure | Change |
|---|---|
| Time to answer | Noticeably faster |
| Cost per question | Substantially lower |
| AI processing per question | A fraction of before |
| Answer quality | Held steady |
Running it
- Three environments (development, testing, production), all set up as code with Terraform.
- Monitoring through Datadog and AWS logs.
- Safety guardrails: I learned how AWS Bedrock guardrails work, tested the options, and put them in place for production.
- Usage reporting: reports to leadership every two weeks and every month. The assistant is available to 3,000+ people across the company.
- I work with an outside development team on new features.
What I learned
A design that makes sense for one generation of AI models can become overhead with the next. The coordinator earned its place at the start. Once models could choose sources reliably on their own, removing it was the biggest single cost and speed win.
The test questions are what made that change safe to ship. Without a fixed set of known answers, we couldn't have shown that a simpler system answered just as well.