Best Practices

BI Leaders—Make it Simple with Data Abstraction

Consider an online retailer who has questions like: When should we roll out specific marketing campaigns?  Which products should we push?  Which products should we discontinue?  These decisions all impact cash flow and profit.  

So they pull together a planning team.  First, the business folks show up with some static Excel reports and spreadsheets pulled from the data warehouse. Then the data scientists bring in their OLAP analysis report from their analysis and forecasting system.  And another manager comes in with with data from the ad-hoc query tool.  And the planning falls apart because none of the data agrees. 

But why do these team members get different answers?!  It’s because different interfaces, even to the same data, can result in different views of the data.  

Consider creating a data abstraction layer if you find yourself in this situation.  An abstraction layer ensures a single interface, or business logic, is the foundation for all users to create  consistent analysis.  

Most organizations, even smaller ones, have many different sources of business data.  They usually have accounting systems (managerial and tax), HR systems, inventory systems, point-of-sale (POS) systems, etc.  And the sources can be databases, web services, big data storage, etc.  All of these systems have their own sets of data, data structures, and ways of representing data.  And they may not all agree.  

So, what is a data abstraction layer?

A data abstraction layer is simply a view of the data that is easy to understand, easy to query, and that leads to consistent answers no matter who is using the system.  Obviously definition is easier than implementation, but a definition does describe the goal.  

Consider our retailer above with their CRM system that has information about the customers.  They have data from POS systems about customer purchases.  They have a separate system recording online customer transactions. 

Instead of directly pulling this data from the source systems into the POS system, they can pull it into an intermediate system. This system is designed for ease of analysis.  They extend this system by pulling in customer data from the CRM and the sales systems. This data then becomes a customer abstraction.  They introduce information from both sales systems about the sales into a single sales area.  

Now all their analysis tools share a single, unified view of customers and their purchases and the planning team gets consistent analytics.

Making the abstract concrete

How do you build an abstraction? First, decide which technology you want to use.  Some companies choose a traditional data warehouse or data marts.  These companies pull data into a data repository for analysis and work from there.  Others purchase custom solutions that assist in abstracting the data and serving it up in a number of different formats (SQL, web services, etc.). 

After you’ve decided on a technology, you can start creating the data abstraction.  An incremental approach is best.  Identify the analysis you want to support.  For example, maybe you want to analyze sales by customers, channels, and product lines.  Pull  the data from the different systems into your new “sales” abstraction layer and create your first data abstraction.  Have a few different teams work with the data to proof the abstraction.

Don’t forget to get business users involved early.  A useful technique is to ask them to provide a list of questions they want answered.  Use these to identify the data and business logic needed to provide those answers. Then, model your abstraction layer to suit.  You may not get all of the analysis supported initially, but you are sure to get real value for end-users.

After that, incrementally add data to the abstraction.  Always makes sure to include your business users as you go into other areas of analysis,.  The benefit of an incremental approach is that you can get value from your implementation quickly, while also learning how to structure the data so that it serves all your teams. 

Using abstraction in the real world

I recently had the opportunity to apply these principles on a project.  We had a customer who had a complex transactional system.  Creating correct answers required a deep understanding of the transactional system and encoding business logic into queries.  We created a simplified abstraction of their transactional system for analytics and user were able to do their own analysis and get correct answers.  This effort focused on one area and only took a couple of weeks to accomplish, start to finish.  It made the business users happy because they can now do their own analysis with confidence and the BI team is happy because they can move on to new areas and more complex problems.

Businesses rely on data to get answers to business problems.  But given the different number of sources of data and access methods, it’s common to have different systems give you different answers that are hard to reconcile.  By implementing a data abstraction layer, you can make data easier to access and provide consistent results for analysis.  And by using an incremental approach, you can get value from the process quickly.