10 data and analytics
trends for 2024

Instead of bringing data-driven ideas to life, we were hardcoding SQL statements, wading through technical debt, and sitting on silos of fragmented data—unable to extract timely insights. The powerful experts were left to tell a blessed version of the truth. 

While we’ve experienced bursts of progress, the dream of truly democratized data remained elusive. But Generative AI (Gen AI) has sparked the rebirth of our industry, breaking down technical barriers and firmly placing your business, people, and ideas at the center of data and analytics. 

This is the Data Renaissance.

By embracing the opportunities of this moment, you can empower your entire community—your team, colleagues, customers, and partners—to reimagine every part of their world, making the impossible possible.

That doesn’t mean it will be an easy course to chart. This opportunity only exists for those willing to step up and embrace it. The names that go down in history will be the ones who are bold, first movers. Leaders who seek to understand not only the opportunities of Gen AI, but also the limitations and pitfalls, will set their organizations up to thrive in this new world.

In the following chapters, we’ll share insights, resources, and actionable resolutions—covering everything from legislation and ecosystem innovation to career paths and product building. This guide is built to illuminate how you can make the most of data and analytics in 2024 and beyond, but we know it’s just the beginning. And we want to hear your thoughts. Join the conversation.

Gen AI is transforming the entire data and analytics ecosystem, kicking off the fifth wave of analytics and BI—a wave that will be bigger, faster, and more furious than any of its predecessors. As we predicted in February 2023, the first movers to embrace Gen AI with ethics and trust will have an outlasting advantage. Lack of imagination and technical debt remain the key blockers to unlocking this opportunity.

Right now, organizations are so mired in creating dashboard widgets or waiting for data loads it’s hard to fully reimagine a radically new way of going from collected data to insight to action, without all the current plumbing and processes.

Think about 1997: Before high-speed Internet and smartphones, did you ever imagine shopping from your phone, trying those trendy jeans on virtually, having them magically appear on your doorstep within hours, paying with a click of your watch or phone, and receiving a photo as proof of your delivery? In 1997, you would have laughed.

We need to stop micro-optimizing the archaic processes and start reimagining them in a Gen AI world—across the whole data-to-insight value chain.

Reimagining the future of healthcare

A reimagined Gen AI future in healthcare, for example, might include medical data shared from a patient’s Fitbit and smartphone with all their doctors. Or, health alerts that are simultaneously sent to the patient and doctor, benchmarked against a population, with recommended actions. If the alert was blood pressure related, the recommended actions might include a class at a yoga studio, lower sodium foods from the grocery store, or a refill of your medication.

Customers buying and deploying analytics and BI products in this wave must consider:

• Trust with human-in-the-loop, always.

• Flexibility of LLMs used to balance accuracy, cost, and provider safeguards. This may include walled-garden options and industry-specific LLMs. 

• Degree that products are built for the cloud, have open APIs, interoperability across data ecosystem components, and provide the ability to leverage all data, including semi-structured.

For as long as we’ve heard about the potential of big data, we’ve also heard about the potential of unstructured data.

Each year, the predictions have been the same:

Unstructured data—the information contained in text documents, images, videos, and other formats, as opposed to the structured data contained in precisely defined tables of numbers—holds a vast amount of useful insight. Technological advances will soon help us unlock it. 

So far, the results have been underwhelming. While there have been pockets of progress, most businesses are making relatively light use of their unstructured data. Chances are, if someone tells you that they’ve uncovered an interesting insight while analyzing data, you’ll assume they’ve been working with numbers and spreadsheets—i.e., structured data.

In other words, for most companies, “data” is synonymous with “structured data.”

Why haven’t we done more with unstructured data? Often, it’s still too hard to work with. To extract meaningful value from thousands of images and text files, we can’t just search through them or parse their contents and summarize them into something useful.

With structured data, we can combine millions of individual data points into metrics and aggregate summary statistics. But how do you condense millions of emails or audio files in the same way? Though some vendors provide ways to extract customer sentiment from social media posts or common themes from sales calls, these tools can only answer a limited number of questions on specific datasets.

The process of working with more general datasets has been more or less manual: someone reads the text, learns from it, mentally aggregates it, and then summarizes what it means for other people. This is extremely expensive, relatively slow, and prone to all sorts of errors. But last year, that changed.

There are two major ways these technologies will profoundly impact business:

All that said, LLMs are still a new and rapidly evolving technology. Most companies are just starting to work with them. Moreover, in order to make use of unstructured data, companies still have to collect it, build reliable pipelines to ingest it, and figure out how to deal with a range of complex security and compliance concerns. These are not trivial pieces of data infrastructure, and will take time—years, probably—for some companies to build.

Though we’re not there yet, the future is promising. Until recently, it’s taken some degree of faith to believe in the value of unstructured data. We’ve been told it’s in there, like some precious ore buried deep inside of an impenetrable rock. LLMs are getting us closer to the center.

Recent research by Experian makes a startling claim: companies typically lose between 15% to 25% of their revenue due to poor data quality.

These losses manifest in various forms, including:

• Time and resources spent in correcting errors

• Verifying data through alternative sources

• Adhering to compliance

• Mitigating consequences of ensuing inaccuracies

Gartner's analysis underscores that poor data quality significantly impairs an organization's competitive edge, impeding crucial business goals. Similarly, a Harvard Business Review report citing IBM reveals a broader economic impact, where the U.S. economy annually incurs a staggering $3.1 trillion loss owing to data-related inefficiencies. These losses are attributed to reduced productivity, frequent system outages, and increased maintenance expenses.

The research clearly demonstrates that prioritizing high-quality, reliable data is not only a technical necessity but also a strategic imperative for Gen AI to achieve its maximum potential. Enter data mesh and data contracts.

Zhamak Dehghani, CEO of NextData and author of “Data Mesh,” states that “Trusted data still isn’t accessible enough to support the AI revolution. Decentralized data is the future.”

Chad Sanderson, CEO and co-founder of Gable.ai, is creating a data collaboration platform based on data contracts to meet the growing needs of BI and AI. The emergence of startups focused on data contracts and data mesh indicates a growing interest in these trends, but data quality issues have plagued the industry for decades. Leaving many to question: Is the solution data contracts or data mesh?

Is the solution data contracts or data mesh?

Both sociotechnical approaches strive to enhance the agility, efficiency, and value derived from data in large enterprises. Data mesh focuses on a decentralized, domain-oriented data management and architecture approach; data as a product is a key concept as opposed to a centralized data warehouse.

Data contracts focus more on greater collaboration to decentralize data ownership, manage schema evolution, provide data quality measures, and enforce SLA, enabling data to reach its fullest value.

Sounds amazing, right? Yet, the controversy still remains. Here’s why:

• Aren’t data contracts just a sub-component of a data quality tool?

• That are the standards, tooling, and processes for data mesh and data contracts?

• Aren’t data contracts just a practical application of the underlying data mesh concepts, not a product or practice unto itself?

• Will they become a standard or die a slow and silent death like other approaches? 

Even in the face of these questions, NextData and Gable.ai raised funds in the Fall of 2023. Zhamak led NextData to acquire $12 million in funding, and Chad Sanderson raised $7 million to start Gable.ai. Clearly, the investors see value in both data mesh and data contracts. 

The demand for quality data at scale from AI and BI coupled with the industry-wide data quality issues suggest that there is potential for data mesh and data contracts to revolutionize data management in BI and AI.

However, the future remains uncertain, with their widespread acceptance and efficacy yet to be determined.

In 2023, we saw data literacy and data fluency initiatives rise on the list of CEO priorities. In 2024, data fluency will evolve to include AI fluency, and organizations that already had a jump start with their data fluency programs will be in a better position to leverage AI.

Organizations have made missteps in conflating data fluency with technical literacy. The two are very different. Technical literacy is about using the technologies whether SQL, Python, Tableau, Mode, ThoughtSpot, Alteryx, and so on. Data fluency is about understanding where data originates, business definitions of data elements, statistical concepts, visual storytelling, and interpretation.

Gen AI will substantially lower the technical proficiency required to interact with data. True natural language processing makes it easier to interact with data. GPT-like interfaces, such as ThoughtSpot Sage, enable business users to more easily ask questions of their data, generate insights in a few clicks, and automatically summarize visualizations and anomalies through text-based explanations. These innovations allow organizations to balance their once-tech-heavy literacy programs and focus more on the language of the business, an understanding of incomplete or biased data, mathematics, and basic statistics.

With AI becoming infused in our daily lives and business processes, key aspects of AI literacy include:

• Recognizing when AI is involved and AI-generated content

• Understanding how digital interactions fuel AI models

• Being aware of a model’s degree of accuracy

• Understanding inputs to algorithms including training data and emphasis of features/attributes

• Realizing how using public AI platforms may contribute to data unintentionally becoming part of the training data set

For example, when Hurricane Ian was approaching the West Coast of Florida in Fall 2022, many citizens within the cone of uncertainty did not prepare or evacuate. The storm veered slightly south, devastating Fort Myers.

Public officials only realized later that too many people misinterpreted the cone of uncertainty. The whole concept of a prediction continues to remain lost on many citizens—whether in hurricanes, snowstorms, or even how long your Uber ride will take.

As Gen AI starts being used in classrooms—with schools having wide-ranging differences in policies—students, parents, and teachers are all faced with AI literacy challenges. Popular tools to check for plagiarism are also confounded with Gen AI content, producing false positives. Teachers who better understand how the tools are trained, and why they are less accurate for students where English is a second language, are less likely to falsely accuse a student of plagiarism.

Regulatory requirements to label AI-generated content may help with identifying such content. But that alone isn’t enough. In a survey of over 1500 Americans, conducted by the Allen Institute for AI, 84% failed basic AI literacy skills.

Will the adoption of standalone semantic layers take off in 2024? Probably not, at least if history is any guide. Despite the recent releases by dbt Labs, Looker, and their industry competitors’ efforts to rejuvenate these systems, the impact of semantic layers in 2024 appears grim. Combined with the disruptive force of Gen AI, this suggests that code-centric and cube-centric semantic layers will soon be regarded as the gatekeepers of a bygone analytics era.

The semantic layer promises to bridge the gap between complex data structures and downstream stakeholders—like analysts, data scientists, and non-technical business users. Self-service analytics is a core capability of that promise. This is achieved by presenting data in a format that is not only accessible and consistent but, most importantly, easy enough for a business user to understand and use in the context of their day-to-day work.

While interest in the promise of semantic layers has grown and the need has not diminished, recent incarnations of the standalone semantic layer must overcome the longstanding and still unresolved challenges.

Despite the recent releases by dbt Labs, Looker, and their industry competitors’ efforts to rejuvenate these systems, the impact of semantic layers in 2024 appears grim.

Four longstanding challenges to the semantic layer include:

Throughout its evolution, the semantic layer's core mission has remained constant, even as the demand to be more dynamic, intelligent, and user-centric has increased.

With the emergence of cloud computing and Gen AI—the semantic layer still promises to bridge the gap between complex data structures and downstream stakeholders. More broadly, the idea that every solution should have a conversational AI coupled with human-in-the-loop capabilities is seeping into the collective consciousness of data leaders, having the effect of ‘freezing’ the market for products that don’t yet have these capabilities. On the upside, LLMs enhanced with a semantic layer could potentially resolve the issue of hallucinations.

The concept of the semantic layer as a valuable component of a data stack is not obsolete. However, the future lies in agile, AI-driven systems that offer real-time insights with minimal human overhead. And in the future, the standalone universal semantic layers may find themselves conspicuously obsolete.

FinOps will expand into managing the costs of operating and optimizing the value of LLMs. Evolution is a critical concept in FinOps. Executives and data leaders have learned, first-hand, the value of FinOps through their experiences with cloud migrations and digital transformations. By now, we know how Adobe racked up an $80K-per-day cloud bill without noticing for over a week, resulting in unexpected costs of over $500K. Adobe is not the only one; the list includes notable companies like Intuit, Pinterest, and Coinbase. And it’s not just the big names wracking up big bills. 1 in 5 companies face an unexpected cloud bill, according to a recent article in Technology Magazine.

1 in 5 companies

face an unexpected cloud bill, according to a recent article in Technology Magazine.

AI services are appearing daily, bring-your-own-LLM (BYOL) is rising, and training AI on private 1st party data is becoming critical to creating a competitive advantage.

It’s easy to jump into the deep end before you realize the water is over your head. But today’s leaders are clear-eyed about the potential cost of new AI offerings.

The enthusiasm for Gen AI and LLMs has not detracted leaders from their commitment to responsible cloud cost management. The lessons learned from the COVID-19-fueled adoption of cloud computing and digital transformation are still fresh in their minds and wallets. According to McKinsey & Company, 89% of large companies have ongoing digital and AI transformations, with only 31% of companies capturing the expected revenue lift and only 25% realizing the expected cost savings.

The underwhelming outcomes of digital transformations have driven the maturation of FinOps over the past five years, leading to a sharp increase in its adoption among top-performing organizations.

That doesn’t deter the onslaught of real and perceived AI hype in 2024. However, the organizations that come out on top will be proactive in making FinOps for LLMs part of their AI initiatives.

2024 is the year of the analytics engineer, AI engineer, prompt engineer, Chief AI Officer who knows?

Back in the early 2010s, the taxonomy of a data team was fairly simple.

Data analysts worked with lines of business to build reports and solve business problems. Data scientists—who often had PhDs in statistics, physics, or economics—built complex predictive models to create forecasts or build recommendation engines. And data engineers built technical systems that would collect, store, and process data for the data analysts and data scientists. The lines weren’t perfectly clear, but they were reasonably well understood. Back then, data leaders could’ve drawn boundaries around the roles in the same way that residents of a city draw lines around a neighborhood: They might not line up perfectly, but everybody would roughly agree on where the Upper West Side is.

Then, in 2012, data scientists—not data analysts or data engineers, but data scientists—were famously declared the sexist job of the 21st century. And all hell broke loose.

Companies became enamored with the possibilities of data science, and data scientists became unicorns that everyone wanted to hire. In 2016, Glassdoor declared it the best job in the United States. Understandably, job seekers started looking for data science roles, favoring them over their lowly cousin—the data analyst.

Wanting to hire the best talent, companies responded by rebranding analysts as data scientists. Lyft explained it all in a 2018 article.

Nicholas Chamandy, then Director of Data Science, Mapping at Lyft, wrote:

“Even if only a minority of the tech industry has shifted their definition of Data Science towards the business analytics end of the spectrum, the presence of even one or two major players in that group (and there are several) immediately puts us on the back foot.

For this reason, we’re shifting our data analysts over to the Data Scientist title. To maintain a functional distinction between the roles, we’re also rebranding our data scientist job title to Research Scientist.”

Things have only gotten more confusing since then. Analytics engineering burst onto the scene in 2019 , occupying a role somewhere between analytics and data engineering. By 2022, it had replaced data science as the trendiest job in the industry. If you asked data leaders to draw role boundaries this year, many of them would say the same thing that the residents of Staten Island said about a few blocks caught in no man’s land: “Nobody knows what to call this neighborhood.”

All of this will create further confusion about what we should call data teams and what they should do. There are solutions, though, so long as data leaders find a balance between experimenting with the latest trends without getting caught up in them. 

Organizations have always wished for one BI tool that would do it all. Meanwhile, users have always wanted a BI tool that is ideal for them, based on their skills and task at hand. 

Now, you throw Gen AI into the mix. It’s a race to bring in this innovation without throwing out all the plumbing of whatever analytics and BI platforms are already deployed. What is a CIO or CDAO to do? Buy five or six different analytics tools optimized for each user persona? Or, settle for good-enough capabilities across a few use cases and limit how many specialty products are added to the mix?

It’s a fool's game. I find myself thinking about family cars and modes of transportation. Settling for good enough is like using one of those Duck Boats and assuming it will work equally well on snowy roads and high-speed highways as it does on city streets and the Boston harbor. But then again, how often does a family buy a different vehicle for each use case? And even if they did, is that a responsible use of resources? Some things need to be shared and common, like electric car chargers, and then the mode of transportation can be optimized for the use case and driving conditions.

There was a time when analytics and BI platforms had a safe investment period of three to five years. Earlier in the first wave of BI, that investment horizon was easily ten years. The waves of innovation (as discussed in the first chapter) have accelerated, but the mindsets of central BI buyers have not. They should.

No longer do organizations have to sign long-term contracts with big upfront perpetual licensing commitments. Nowadays, with pay-as-you-go, consumption-based pricing, procurement approaches should be more flexible. This shift in licensing models is an industry shift that enables every analytics persona to have their own optimized experience, without sacrifice. License the product for a month or a year, and if your needs evolve or a competitor innovates faster, switch. That’s the theory. 

In reality, interoperability across the analytics ecosystem and even within a single analytics platform remains a work in progress. Even when an organization has implemented a cloud data platform, legacy processes, and visualization tools, they continue to extract, replicate, and aggregate that data into their own proprietary stores—often unnecessarily so. Metadata between different modules may not work together, and newer capabilities may not work consistently.

Just look at Tableau’s dbt integration that only works in Tableau desktop; not within Tableau Server or Online. Or consider Power BI’s smart narratives that work differently within the desktop and the browser, with constraints in updating visuals.

Optimized experiences by persona, using a shared foundation and flexible licensing, solve the wishes of both the users and procurement team. ThoughtSpot was already executing on this vision. For example, ThoughtSpot for Sheets uses the same shared semantic model as ThoughtSpot Liveboards. Multi-experience was also the vision behind the ThoughtSpot and Mode merger–let the data analysts have their code environment with the business users having their no-code environment, but the data model can be via dbt and visualizations created in one environment can be surfaced in another.

This multi-modal experience extends to embedding analytics into business applications like Salesforce and ServiceNow—either as Liveboards or as natural language search interfaces. It’s unclear how excited the market is about extending the data experiences to virtual reality with glasses like XReal or HoloLens. But, I can imagine field workers appreciating this hands-free ability to call up critical KPIs.

If these predictions were written eight years ago, we might’ve said that the future of data and AI was on the blockchain. This revolutionary new technology, I might’ve continued, will change tens of thousands of businesses. Adopt it, I’d conclude, or get left behind.

Though people made these sorts of forecasts—according to one survey, more than half of global business leaders thought that ten percent of global GDP would be stored on the blockchain by 2027—nobody got left because they ignored the warnings. Instead, the opposite happened: A number of companies chased the hype, pushed blockchain-powered products out the door that were more sizzle than substance, and shut them down a few years later.

Now, the technology world is turning its attention away from Bitcoin and towards AI. A majority of CEOs say that Gen AI is one of their top priorities, and everyone from small startups to multinational enterprises like Walmart and PwC are rushing AI-backed products and tools out to their customers and employees. Every day it seems a SaaS vendor launches a new chatbot.

The problem, however, is that building a chatbot is relatively easy. Building a good chatbot—one that’s more than demoware and genuinely improves the experience of using a product—is hard. 

Simple wrappers around GPT and Anthropic aren’t enough. Vendors have to build and maintain much more than that—think:

• Integrations into metadata repositories

• Infrastructure for storing that metadata and evaluating its usefulness

• Systems for evaluating the accuracy of results

• Tests for hundreds of versions of different prompts

• Feedback loops to help models improve

But that’s not all. Vendors might also have to train their own LLM, or choose from what is quickly becoming a huge catalog of domain-specific options. And most of all, they have to figure out if you need a chatbot at all, or if AI is better integrated into their products in a completely different way. 

We know these intricacies intimately because we encountered them when building and refining ThoughtSpot Sage. For example, we couldn’t just send a question like “How many accounts did we have in Asia last quarter?” to a LLM and return an accurate SQL query.

Companies that do this deep-thinking work will be rewarded with more useful products and happier customers.

But the companies that don’t—the companies chasing AI because it’s the latest fad—will actually end up making their products worse. Their bolted-on, half-baked chatbot will work for simple requests, but its inconsistency will constantly frustrate customers. People won’t use the feature, and product teams will stop investing in its success. Over time, that chatbot will slowly decay into an awkward barnacle that everyone learns to ignore.

In other words, in 2024, AI will transform thousands of products—some for better, and some for worse.

While the global AI innovation race speeds ahead, regulators are working to mitigate potential harm to humanity. The range of risk includes full annihilation to more insidious harms.

No matter where you stand on the AI fear spectrum, the impacts to businesses and individuals are real. The recent drama of OpenAI’s dismissal, then rehiring, of Sam Altman shows the high-stakes and extreme philosophical differences that exist. To all, the reality is clear: regulation risks stifling innovation—a risk no country in a global, digital economy wants to take.

While the playing field is not level globally—China, for example, will take a different view on AI regulation than the USA—tech innovators want the playing field to be level within regional economies. Gen AI is evolving so quickly that there is zero chance regulation can keep pace with the ever-expanding capabilities.

Recent examples of AI-induced harms:

• A Gen AI chatbot gave diet advice to people with eating disorders.

• An algorithm to prioritize kidney donations gave black patients a lower priority.

• Teenage boys in a New Jersey high school used Gen AI to create fake nude photos of female classmates.

• A Meta whistle-blower and Instagram safety consultant revealed how his own daughter faced sexual harassment that Meta’s safety protocols failed to prevent. 

• Scammers use AI-generated voice bots to trick family members into sending money.

These efforts may be useful in raising awareness and building consensus, but builders and deployers of AI models must be more proactive right now.

Cindi Howson is the Chief Data Strategy Officer at ThoughtSpot and host of the award winning The Data Chief podcast. Cindi is an analytics and BI thought leader and expert with a flair for bridging business needs with technology. In her role at ThoughtSpot, she advises top clients on data strategy and best practices to becoming data-driven and influences ThoughtSpot’s product strategy. She was awarded Motivator of the Year by Women Leaders in Data and AI and named one of the most influential people in data by DataIQ100.

Sonny Rivera is a Senior Analytics Evangelist at ThoughtSpot. Sonny is a modern data stack thought leader and expert. He brings with him over 25 years of experience in delivering data solutions that drive business value and increase the speed to insights. He also has a long history of bridging the gap between business needs and technical capabilities.

Benn Stancil is the Field CTO at ThoughtSpot. He joined in 2023 as part of its acquisition of Mode, where he was a Co-Founder and CTO. While at Mode, Benn held roles leading Mode’s data, product, marketing, and executive teams. He regularly writes about data and technology at benn.substack.com. Prior to founding Mode, Benn worked on analytics teams at Microsoft and Yammer.