- Plan beyond storage by assigning clear roles to metadata, governance, processing, and analytics.
- Decide how people will find, understand, and use data before the repository grows.
- Choose schema and storage patterns around real reporting, exploration, data science, and AI workloads.
- Compare lake, warehouse, and lakehouse options as planning choices rather than competing defaults.
A data lake stores structured, semi-structured, and unstructured data in raw form, so you can prepare and analyze it for different purposes later. Microsoft's data lake definition describes it as a storage repository that holds large volumes of data in its native, raw format.
When you plan a data lake, storage is only one responsibility. You also need a clear approach to discovery, governance, context, and analytics if people will rely on the data.
What Is a Data Lake?
A data lake is a centralized repository for data that you may not want to structure before storing. AWS data lake overview describes a data lake as a centralized repository for structured and unstructured data at any scale.
That scope can include relational tables, CSV files, JSON, XML, logs, clickstream events, Internet of Things data, images, audio, video, and documents. With schema-on-read, you apply structure when you query or process data for a particular workload.
Use that flexibility deliberately. Before collecting a new source, identify why you want to retain it, who may use it, and what context they will need.
Treat the repository as one responsibility within a broader plan. Set expectations for ingestion, metadata, processing, governance, and consumption alongside your storage choice. Record those decisions in one place, and review them when a new source or workload enters the plan.
What Kinds of Data Can a Data Lake Store?
Start with an inventory of the structured, semi-structured, and unstructured data you expect to collect. Group those sources by business purpose, sensitivity, ownership, update pattern, and likely downstream use.
Raw retention can preserve options for later analysis, but do not use raw as a synonym for unmanaged. Assign an owner, descriptive metadata, access rules, and quality expectations to each high-value source.
When a source may support several purposes, document those possibilities without forcing one future use. Then prepare the right slice when a reporting, analytics, modeling, or AI need becomes clear.
How Does a Data Lake Work?
To plan how a data lake will work, map the path from source collection through storage and use. This article uses a six-layer planning framework to help you organize the decisions involved:
Sources: List the business apps, databases, event streams, files, devices, and feeds you intend to collect.
Ingestion: Choose batch or streaming patterns and record each source's origin, timing, and owner.
Storage: Decide how you will organize raw data, refined datasets, and approved outputs.
Catalog and metadata: Define how you will record names, formats, ownership, lineage, freshness, and business meaning.
Processing and governance: Select preparation methods and set policies for access, quality, privacy, retention, and usage.
Consumption: Identify the analytics platforms, notebooks, applications, or machine learning systems that will use each dataset.
When you use schema-on-read, document how structure will be applied for each workload. Use shared metadata and governance to keep interpretations consistent across those uses.
Your analytics layer can connect with data stored across the broader estate. ThoughtSpot can connect data lakes and analytics across business apps, databases, data warehouses, data lakes, files, and feeds.
Why Metadata and Governance Matter from Day One
Do not rely on file names or folder paths to carry meaning, ownership, freshness, or reliability. To avoid an unmanaged lake, define metadata and governance practices before the collection expands. Specify which details are required, who maintains them, and how often each record should be reviewed.
Use this checklist as a direct starting point:
Catalog high-value datasets and the fields people use most often.
Assign an accountable owner to each important data product or domain.
Classify sensitive data and document appropriate handling requirements.
Record quality checks, update schedules, lineage, and known limitations.
Apply access policies based on roles, purpose, and data sensitivity.
Review storage, compute, retention, and usage patterns over time.
Definitions matter just as much as file access. ThoughtSpot Models can add business logic, calculated metrics, synonyms, filters, and governance controls, giving people governed reusable data models for analysis.
Data Lake vs. Data Warehouse vs. Data Lakehouse
Use this comparison as a rule-of-thumb decision framework. Focus on when you want to apply structure, which responsibilities you want the storage layer to carry, and what each workload needs.
| Planning Question | Data Lake | Data Warehouse | Lakehouse |
|---|---|---|---|
| What data state will you manage? | Plan for raw and refined data in several formats | Plan for cleaned, modeled, and structured data | Plan for raw and refined data with managed tables |
| When will you apply schema? | Apply schema when a workload reads or processes data | Apply schema before loading data for planned use | Choose where flexible intake or managed structure matters |
| What will you prioritize? | Retention, exploration, data science, or varied processing | Consistent BI, reporting, or governed analysis | BI, analytics, or data science on shared lake storage |
| Which workload question comes first? | Do you need room for varied or changing data? | Do you need standardized metrics and reporting? | Do you need lake storage with more table management? |
Choose schema-on-write planning when your reporting process calls for data to be cleaned and organized before loading. This choice can support agreed definitions and established reporting requirements.
Choose schema-on-read planning when you want to retain varied data and apply structure for a defined use. For further reading on the decision, see data lake versus data warehouse.
Consider a lakehouse when your plan calls for lake storage plus more table management for analytics. This data lakehouse architecture guide offers further reading on that option.
When Should You Use a Data Lake?
Consider a data lake when your plan includes varied or fast-arriving data, long-term retention, exploratory analytics, data science, or machine learning. Define the specific workload first, then test whether flexible retention supports it better than a pre-modeled approach.
For standardized reporting on clean, modeled data, compare the lake plan with a warehouse. If you want BI and data science to work from shared lake storage, include a governed lakehouse in your evaluation.
You can also assign different responsibilities to operational databases, a warehouse, and a lake. Document where each dataset belongs, how it moves, and which system should answer each type of question. If the same data appears in several places, name the approved source for each reporting or analytics use.
What Are the Benefits and Limitations of a Data Lake?
Treat flexibility as a potential benefit to evaluate against your workload. Retaining varied data before every use is known may support later exploration, advanced analytics, or machine learning.
You can also plan storage and processing as separate responsibilities. Decide which engines will handle preparation, querying, and modeling, then document where approved source data should remain.
Balance that flexibility with management requirements. Set standards for formats, metadata, ownership, and quality context so people have a defined basis for evaluating a dataset.
Review security at the file, zone, and derived-dataset levels. Define access policies, lifecycle rules, and handling requirements for sensitive information before broader use.
Model storage, movement, duplication, retention, and compute as separate cost questions. Review usage patterns and decide which data needs to remain immediately available.
Test performance against representative queries instead of assuming one storage pattern will meet every need. Your plan may call for refined zones, managed tables, indexes, caching, or modeled data at the point of use.
How Do You Turn Data Lake Storage into Trusted Analytics?
Treat storage and analytics as separate responsibilities. For every priority dataset, define how people will find it, understand it, request access, and evaluate whether it is appropriate for their question.
Keep the data layer focused on storage and preparation choices. In the analytics layer, define the business context, metric logic, filters, and governance needed for each approved use.
The ThoughtSpot connection and model examples above show two planning points: connect the relevant sources, then give people governed context for analysis. Apply the same test to any analytics approach you evaluate.
Ask who needs the data, what question they are answering, and which definitions should guide the work. Then document those decisions close to the dataset and the analytics experience. Include examples of approved questions, metric definitions, and access boundaries so reviewers can evaluate the planned use.
A Practical Starting Point
You do not need to design every future workload before you begin. The following five-step sequence is one suggested way to start with a specific decision or use case:
Choose a priority decision or workload. Define the question, user, required freshness, and acceptable response time.
Map sources and owners. Identify where the data comes from, who understands it, and who approves access.
Choose ingestion and storage patterns. Decide what arrives in batches or streams, plus how raw and refined data will be organized.
Set catalog, access, and quality controls. Document meaning, lineage, sensitivity, freshness, tests, and retention before usage grows.
Publish governed models and test real questions. Give people shared metrics and definitions, then note where context or data quality needs more work.
Use what you learn to refine the plan before adding another workload. Keep each new decision tied to a named owner, a defined question, and documented governance requirements. Review the plan with data owners and intended users before expanding the scope.
Conclusion
A data lake is flexible storage for raw, diverse data, but storage alone does not create useful insight. Your durable architecture pairs the right storage pattern with metadata, governance, business context, and an accessible way to analyze trusted information.
Whether you use a lake, warehouse, lakehouse, or a combination, start with the questions people need to answer. Then explore ThoughtSpot analytics for AI-native, governed exploration across connected data.
