FRC ERIE SUMMIT  ·  KEYNOTE RECAP

Confidently Wrong: Why AI Agents Fail on Government Data

Jay Schuren on the architecture that turns unreliable AI into a system you can trust.

A recap of Jay Schuren’s featured session at the FRC Erie Summit. Jay Schuren is President of AtScale.

At the FRC Erie Summit, AtScale President Jay Schuren delivered a warning wrapped in a solution. Complex AI can unlock enormous value, but only when it is engineered to be reliable. Left on its own, an AI agent asked to handle real government data will often be confidently wrong, and in the public sector that is a problem you cannot afford to ship.

The Speed Advantage Has Shifted to the Attacker

Schuren opened with a statistic that reframes the stakes. Citing a report drawn from 500,000 incident reports, he noted that the median time from initial access to an active, coordinated attack dropped from eight hours to 22 seconds, a roughly 1300x improvement. That leap did not come from more people or manual effort. It came from adversaries automating their work with AI. His pointed question to the room: have any of us gotten 1300x better at our own jobs? The people trying to do good with AI, run programs, protect data, serve the public, are not moving nearly that fast, and that imbalance is the real risk.

From Ask Mode to Agent Mode

The industry is shifting from ask mode, where a person uses AI as a smarter search tool and decides what to do with the answer, to agent mode, where the system reasons about how to approach a problem and then acts on it. That shift is powerful, but it changes the tolerance for error. When a person is in the loop, a wrong answer is a nuisance. When an agent is chaining steps and making decisions, a wrong answer early on quietly corrupts everything downstream.

The Core Problem: Confidently Wrong

Whenever AI touches something that has to be a fact, Schuren explained, it can fail in a very specific way: it is just confidently wrong. He used the simplest question he could think of, how much did we spend last month? A finance team means the prior calendar month, not a rolling 30 days. If the model guesses wrong and returns 4.7 instead of 4.1, nothing breaks and no error is thrown. Now imagine that number is the first step in an automated process that decides whether to pay someone. This, he said, is what scares him most about deploying AI on anything that truly matters. In government, a wrong figure on a budget or a grant is not a rounding issue; it erodes public trust.

“You can’t just prompt your way to a better result. You have to structurally add the context to get there.”

Jay Schuren, FRC Erie Summit

More Data Is Not the Fix

The common instinct is to feed the model more: every past query, every dashboard, every report, and trust that volume will produce accuracy. Schuren pointed to research from Anthropic on their own data. Out of the gate, the model answered correctly only 21 percent of the time. Piling on all of those queries and dashboards improved it by a single percentage point. What took them to 95 percent was not more data. It was a semantic layer.

The Semantic Layer: Structure Over Guessing

A semantic layer is simply the logical relationships in your business data, the definitions that make an answer correct. It tells the system that a question about last month always means the prior calendar month, never a rolling window, and it does so for every metric each domain team cares about. The teams that own each number document it once, and the model is then forced to reason with those definitions instead of guessing at them. Every organization has its own nuance, and no general-purpose model was trained on it. The same model, given the right structure, goes from unreliable to dependable.

Reasoning Plus Deterministic Tools

Schuren’s architectural principle is to play to each part’s strength. Modern models are excellent at granular reasoning, so let them reason. But do not ask a probabilistic model to write an eight-page SQL query or add two numbers; by the time it is a page in, it is likely doing something you never intended. Pair the model’s reasoning with tools that do the deterministic heavy lifting, a calculator for math, a semantic layer for data, so the pieces that must be exact are never left to a guess. It reasons; the tools return the facts.

Building an Agent You Can Trust

Making an agent safe in production comes down to a few requirements. The agent needs governed access to your tools and data, typically through the model context protocol, which introduces the same security concerns you have always managed, including prompt injection. Beyond that, three things matter: deterministic answers, so exact questions get exact results; security policy applied in context, so an agent respects what each user is actually allowed to see rather than assuming one posture fits all; and full auditability, a reproducible trace of who asked what, when, and with which tools. A probabilistic system can return different answers on different runs, so logging every step is what makes the process defensible.

What This Looks Like in Government

Schuren pointed to Army and Defense Innovation Unit work on unmatched transactions, where each mismatch once required hours of manual investigation and years of tedious machine-learning models to address at scale, ultimately saving the DoD over a billion dollars. With today’s agentic AI working intelligently against the right data, that class of work could be done in minutes. The same pattern applies well below DoD scale: unspent grant funds, purchase orders, and unobligated money at year-end are all problems that good architecture can surface in time to act.

Where to Start

His advice for teams early on the journey was practical. First, take a census; most organizations already have someone building something with AI that simply has not surfaced yet. Then prioritize the way you would any initiative, weighing risk against reward and choosing the projects that build momentum. Ideas fail when they are not useful, so start where the value is real and the path is clear. The technology is genuinely useful, and the other side of the security equation is already using it, so the goal is to adopt it safely rather than sit it out.

The Takeaway

Schuren closed on a hopeful note grounded in engineering. We can use this technology safely, but safety comes from the infrastructure and design, not from clever prompting. Give the model the structure and context it needs, pair its reasoning with deterministic tools, and govern it with policy and auditability. Do that, and confidently wrong becomes reliably right.

Key Takeaways

• AI has made attackers dramatically faster; defenders and government programs need to close the gap safely.
• On anything that must be a fact, AI’s failure mode is being confidently wrong with no error to catch it.
• More data barely helps; structure does. A semantic layer took accuracy from 21% to 95%.
• Let models reason, but hand exact work to deterministic tools like a semantic layer.
• Trustworthy agents require deterministic answers, in-context security policy, and full auditability.

Explore more highlights from Jay Schuren’s Session: https://fedresources.com/erie-summit-sessions/#jay-schuren