Why Most Enterprise AI Pilots Fail: Lessons on Trust and Deployment

Episode Overview

Understand how to close the gap between AI experimentation and enterprise production. Shub Agarwal, Founder of the AI Trust Lab at USC and author of Successful AI Product Creation: A Nine-Step Framework, shares his AI product management framework for taking enterprise AI strategy from demo to production, drawing on two decades of product leadership at Amazon and Fortune 50 firms. He breaks down why experimentation must tie directly to business OKRs, the four mindset shifts leaders need to scale AI responsibly, and how the AI Trust Lab is building a benchmark evaluation framework for AI model trust and governance.

Key Moments:

Why 80% of AI Projects Never Reach Production (02:13): Shub traces the root cause of stalled AI programs to a missing system for moving from demo to deployment. Most teams have no repeatable path to production.
Shub's Nine-Step Framework for Building AI Products (06:00): Most AI projects start with a cool model instead of a painful problem. Shub walks through the three phases of his framework: discovery, execution, and excellence.
The Case Against "Fix Your Data First" (12:41): Conventional wisdom says clean your data before building AI. Shub challenges that, arguing modern LLMs offer far more flexibility with imperfect data.
Four Mindset Shifts for Scaling Enterprise AI (16:35): Shub outlines the four shifts separating organizations that scale AI from those that stall, from measuring AI performance differently to embedding trust from day one.
Inside Shub's AI Trust Lab at USC (23:54): Major foundation models are already being benchmarked on trust and safety. Shub explains the lab's mission to build a standardized evaluation framework for AI model governance.
Why Enterprise AI Governance Needs Multiple Disciplines (28:36): AI models can be sycophantic, manipulative, or lack candor. Shub argues that building trustworthy AI demands an interdisciplinary approach.

Key Quotes:

“I think the fundamental problem that organizations are facing today… is not that they have a lack of experimentation in the demo aspect. The challenge is they don't know how to take those demos to production, and that is where I saw the gap.” - Shub Agarwal

“I do think data is the fuel for AI… But I think today organizations are crippled by this ‘fix your data, and then we'll build AI’, and they never build AI. They never build use cases that are adding value." - Shub Agarwal

"There's no FICO scores for models, so I decided to create one. I built this lab… bringing the computer scientists, the researchers, the applied AI researchers, the policy, and the communication people together to think of what is trust, define it, and ultimately measure and evaluate it." - Shub Agarwal

Mentions:

Guest Bios:

Shub Agarwal is an associate professor of professional practice at the University of Southern California, an industry executive, and an advisor to start-ups and academic institutions. He holds an MBA from the University of California, Los Angeles (UCLA), and an MS from Carnegie Mellon University (CMU). He is the author of two books: Solve Catch-22 of Product Management and Successful AI Product Creation: A 9-Step Framework. He has made significant contributions to the fields of artificial intelligence and machine learning, holding several U.S. and global patents for his work, and is also a published author of several technical research papers.

BACK TO DATA & AI CHIEF