Is Your AI Analyst Lying? Uncomfortable Truths Revealed

Webinar recap: Andy Cotgreave (co-founder, How to Speak Data) and Francois Lopitaux (SVP of Product, ThoughtSpot) on hallucinations, accountability, and what it actually takes to trust AI with your data.

If you've ever watched a colleague confidently present a number that turned out to be wrong (and only realized it after the decision was made), you already understand the tension at the heart of AI analytics. 

The tools are fast, fluent, and convincing; that's exactly the problem.

That tension ran through a recent ThoughtSpot session, Is Your AI Analyst Lying to You?, where Andy Cotgreave traded views with Francois Lopitaux, SVP of Product at ThoughtSpot. 

Andy spent 15 years at Tableau as Senior Data Evangelist before going independent; he co-authored The Big Book of Dashboards, runs the How To Speak Data newsletter, and is one of the most influential voices in the field on what AI is actually doing to analytics. 

So this wasn't a vendor-versus-skeptic standoff. Both agree AI is reshaping the discipline. The interesting disagreements were about how—and about who's left holding the bag when the AI gets it wrong.

Here are the five sharpest takeaways for anyone building, buying, or relying on AI-powered analytics.

How do you know if your AI analytics tool is giving you the right numbers?

Andy opened with a story. He'd asked an AI tool to build a dashboard from a US working population dataset. It produced something polished and completely authentic-looking. It wasn't until later, during a live stream, of all things, that someone pointed out the big KPI at the top was simply straight-up wrong. The model had hard-coded a hallucinated number and built calculations around it.

His point wasn't that the tool is useless. It's that we're wired to trust things that look right.

A rigorous analyst running something for production would verify those numbers. But that's not the scenario that worries him. 

The scenario that worries him is the CMO who needs a slide in the next hour, and the business user moving fast who has no reason to second-guess a clean-looking chart. “That's the variability that terrifies me,” Andy says. “The data analysts can do that rigorous stuff. But how do we help the people who are not data analysts, business users, who are working quick?”

That's the real exposure with AI analytics. It’s not that it can't produce a good answer, but that a wrong answer looks identical to a right one.

So who's responsible when AI gets it wrong?

This was the most debate-worthy moment of the conversation. Andy pressed the question directly: a business user asks something, gets a wrong number, and makes a consequential decision on it. Who owns that?

Francois didn't hedge: "Who is responsible is the person who asked the question,” he says, “because you have to be curious about where the answer is coming from and you need to be able to validate where the answer is coming from.”

That can sound uncomfortable if you adopted AI tools precisely so you wouldn't have to interrogate every number. But Francois's point isn't that you should distrust AI by default. It's that any platform worth deploying has to let you check its work.  And if it doesn't, the responsibility framing falls apart.

Andy sharpened the stakes. The whole promise of AI is speed: short questions, fast answers. But speed is exactly what makes the accountability question hard.

The implication for data leaders is concrete. Before you roll an AI analytics tool out across the organization, ask whether it shows its reasoning. Can a business user see how the answer was generated and trace it back to the underlying data? 

If not, you're handing your teams a liability dressed up as productivity. This is a governance policy question, not a tooling preference.

"It's a solved problem", but only if you change the architecture

Francois made one of the session's boldest claims: that for ThoughtSpot, the trust problem is already solved. The reasoning is worth unpacking, because it's an architecture argument, not a marketing one.

The key move is to keep the LLM away from the step where it's most dangerous: generating the query that pulls your data. Instead, the LLM interprets what you're asking, and ThoughtSpot's own technology converts that intent into a query. The result is greater determinism:

He framed the broader philosophy memorably: use the LLM "like we are using a CPU"—a smart component inside a larger architecture, not the entire system. 

The validation piece matters here, too. Since ThoughtSpot's search tokens render a query in plain language, Francois's test is that his own mother could read one and understand what was asked. In short: can non-technical users verify an answer without knowing SQL?

"Single source of truth" oversimplifies a nuanced reality

If you've spent any time in data strategy conversations, you've probably heard "single source of truth" so often it's lost all meaning. Andy admits the phrase drives him nuts because it obscures the human sitting at the end of the process: “You can have the best database and a great semantic layer on top of that, but at the very top that last mile is an insight and communication layer where for me, traditionally that is a human element,” he says.

Francois agreed with a reframe. A semantic layer doesn't pretend all your data lives in one place. In a modern stack, it's a composition of signals—business logic, definitions, company context, even how people actually use the data—living across multiple systems. Think of it as functionally one layer; technically many.

And even when you agree on the data, interpretation introduces variability. Andy's example was simple: ask for average sales by product by country, and two people might reach for different aggregations. “You and I could choose different statistical aggregations like mean, median, or mode to get totally different answers,” Andy says.

The lesson isn't that one answer is better: it's that the "truth" can be quietly shaped by the tool. So the goal shouldn't be one source of truth, but one shared, governed definition of what the words mean before the AI starts answering.

How is the data analyst role changing because of AI?

Andy raised the existential question that his audience is already feeling. With business users talking straight to AI and engineers maintaining the infrastructure, what's left for the analyst in the middle?

Francois pushed back first by questioning the premise: Analysts don't vanish; they become the people who make the agents trustworthy in the first place. Less dashboard builder, more agent manager.

In practice, that means owning the semantic layer: the business logic and definitions that ground every answer. Analysts become, in his phrase, "the guardian of this castle." 

Andy named the bittersweet part honestly: some of the joy of the job was crafting something beautiful, and managing agents at scale isn't the same craft. But Francois argued the trade is a promotion, not a demotion: “You are enabling the entire company to thrive inside. So I think it's much more rewarding,” he says.

The skills that make a great analyst—curiosity, domain knowledge, a healthy skepticism about a number that looks off—are exactly the skills that make a great agent manager.

Are dashboards still relevant now that AI can answer questions directly?

Andy raised the obvious risk: if anyone who can write a prompt can build a dashboard, do we just recreate "Excel hell" and "dashboard hell" all over again—an AI dashboard hell? 

Francois's answer was governance by design. Because generated output is grounded in the semantic layer, what you create is validated before it exists, rather than a free-floating result you can't trace.

Across ThoughtSpot's usage, the split between dashboard consumption and conversational analytics sits at roughly 50/50. People still want a Monday-morning artifact they can glance at. 

What's changing is the form factor - now the standard is dashboards that are personalized and dynamic, shaped by your role and your recent questions, rather than static reports everyone shares.

Where is AI analytics heading next?

The throughline across all of it is the same: AI analytics is only as trustworthy as the architecture and governance behind it. 

The organizations that get this right won't be the ones with the flashiest AI—they'll be the ones that fostered a scalable governance policy and a culture of AI and data literacy, 

Watch the full session on demand, or book your personalized ThoughtSpot demo today.