Eight weeks ago, a client came to us with a challenge: their data team was stretched, and they wanted a high-performing, customer-facing agent that could parse a database and return accurate answers-answers that currently required too many clicks and filters in Power BI. It seemed simple. LLMs have progressed so quickly that, in theory, you could just point one at the other. Several major data lake and warehousing platforms, like Snowflake and Databricks, even claim this works out of the box.
Yet as anyone who has worked in an enterprise data team knows, transforming large volumes of raw data into accurate, query-performant tables can be complex. It takes deep proficiency in Python and SQL, some form of data compression tooling (Timescale / TigerData), pipeline management via dbt (if you're lucky), ideally 3-6 months of business context, and usually a few beers after work with the ex-DBA who left two years ago. Our client had also fallen into a fairly common trap: building a hornet's nest of stored variables and functions directly into the visualisation layer, which slowed reports and made the figures teams relied on difficult to reproduce.
The additional challenge here was data segregation - since there was a single DB and the agent needed to serve multiple customers, we had to route all agent queries through authenticated MCP server endpoints on the backend. The agent wasn't allowed to write raw SQL against the database because of the risk of one customer seeing another customer's data.
But in the LLM era, this should be simple right?
Where it broke down
We began. The data was intriguing - IoT sensor data from factory lines producing familiar products: tubs of yoghurt, boxes of beef patties, and trays of rusks. Lines are measured on composite efficiency - essentially a combination of uptime, performance, and quality - so we needed to account for rejects, sensor heartbeats, products, line processes, downtime reasons, shift scheduling, and, of course, outputs.
Like most AI-based POCs, it was quick to get something reasonably impressive. But the agent hit a wall fairly soon:
The key question in augmenting data analyst & engineering roles is that a human typically has a "scent" for the right data shape - judgment that helps them naturally triangulate questionable answers or intuitively choose a more readable grain to group by. It was clear (as we expected!) that letting an agent loose on a schema doesn't work, and no amount of prompt refinement would fix it. So the actual challenge was to build a degree of trust you'd normally get when sending / receiving a request to your data team - we just had to figure out how to encode that judgment and measure whether it was actually working.
Enter evals
That's where evaluations (or evals) come in. Evals are essentially a set of controlled tests where you've defined the right answers in advance. Running the eval suite involves asking the agent to answer the question and grading the response according to a rubric - is it correctly formatted, does it produce the right answer, does it give a reasonable textual interpretation of the results etc. It's surprisingly like a school exam actually - the agent ends up with a score like 75%, and some feedback on how it can improve. Here's how a couple of the items look like:
We decided to go with Langfuse, one of the leading open-source LLM eval frameworks, and implemented a set of 24 questions provided by our client, grounded against three of the largest customers on two separate reference days.
The first hard part: establishing ground truth
This gets complex quickly because you have to figure out what "correct" even means. To encode this, we'd go back to the live data from a particular day, pull the raw numbers, and manually verify them. This often meant loading everything into a spreadsheet, doing the calculations ourselves and cross-referencing against what the reporting tools said.
You're basically becoming the source of truth yourself because you can't trust that the existing reports are correct - dates are wrong, product codes are shared across lines, timezones don't match. Typically there are always strange things configured at the application layer and then nested workarounds in the data that have been there for years. We took a week just getting our benchmark data clean enough to use it.
The second hard part: running evals repeatedly on the right infrastructure
Most organisations have a traditional QA gate-you build something, you hand it to QA, they test it, and you iterate. But with prompts constantly changing-and with multiple LLM calls wired together in an agent pipeline-that model breaks down. Embedding evals directly in the CLI was essential for a tight feedback loop, so that the agent could grade itself against an external standard.
The whole pipeline lived there-prompt versioning, query tracing, and eval scoring. You'd tweak one prompt in the pipeline, run it in the terminal, and instantly see not just the agent's answer, but which prompts it hit, how it routed, and whether it passed the eval. On some tool calls, we went through forty or fifty prompt versions. Managing that without proper version control would have been completely unworkable. With some tight prompt revisions and sharper tool definitions, we managed to get the agent up to around 50% on the eval suite.
Layered on top of that were multiple environments. We ran MCP servers in dev, staging, and production, each with separate auth and gating. If you change something upstream in one environment and get it wrong, evals can end up running against different data than your customers are seeing.
The third hard part: evals expose upstream problems
The final chunk of agent correctness was the hardest part. Once you start scoring answers against ground truth, you realise the agent isn't always the issue. Sometimes a failing eval traces back to the data itself, where the gold view layer had stale logic or a materialised table was always being calculated wrong. Fixing those problems and getting our final 45% of accuracy meant going upstream - rewriting views, consolidating and reshaping tables, so that the agent could draw on clean, reliable, unambiguous data. It turns out (surprise surprise) that if your house is already in order, your returns from AI investments are much higher. Also - once your heavy read tables are at the appropriate grain, the performance changes are a nice upside, and further changes are far smoother.
So did we answer the question?
We started by asking how a data agent changes the data analyst and data engineer role. The honest answer is: it depends on which problem you're solving.
Can it do what a mid-level analyst does? Mostly, yes - but not on day one, and not without unglamorous work to get there. Forty or fifty prompt iterations on some tool calls alone. Hundreds of queries manually verified against raw sensor readings before we trusted a single eval score.
The agent is most helpful for absorbing the relentless volume of ad hoc requests that eat up a data team's time - the "can you just pull last month's output by production line split by SKU for these 3 customers" queries that are technically answerable but take someone twenty minutes to pull together. A well-built data agent handles that queue, and yes, your junior data analysts won't be learning SQL anytime soon, but it does give senior engineers their time back for work that actually requires them.
What we didn't expect was how much the agent changed how we approached the data engineering work. The failing evals became a prioritisation tool - showing where to focus, which materialised views to revisit, where to build data quality scoring. The diagnostic loop ran upstream into the raw tables as much as it ran from customer prompts, and that was incredibly helpful.
On reflection, the thing that shifted my thinking was treating the agent like a new colleague. You wouldn't give someone no feedback and hope they would figure it out. You'd watch their work, flag when something looked wrong, and iterate. Evals are just that process made systematic - building judgment over time, providing evidence for the known correct answer and grading outputs progressively. Over time, you'd also appreciate it when they point out your gaps too!
>_ Written By
James Reid