SAP BI and the Conservation of Complexity

You may remember a rather obscure law of science from your high school days in physics called the Conservation of Energy:

The total amount of energy in a closed system remains constant. In other words, energy can be converted from one form to another, but it cannot be created or destroyed.

I don't claim to understand this principle fully, but essentially it's saying there is a finite amount of energy in any given system and that amount cannot be changed. All you can do is change the way it's packaged. I guess it's a little like the mess in my boys' room – it never really goes away; it only changes form now and then.

I would like to offer a corollary to the Conservation of Energy that I call the "Conservation of Complexity", which states:

The total amount of complexity in an information system remains constant. In other words, complexity can be transferred from one party to another, but it cannot be created or destroyed.

This principle states that in any given information system there is a finite amount of complexity that cannot be removed. All you can do is determine who deals with the complexity. With any given business intelligence system the options have traditionally been the BI software/system vendor, IT "back end" resources, IT "front end" resources, and the end user.

Going back in to ancient history (10 – 20 years ago), the following graphic shows how the typical transactional system might have distributed complexity:

I experienced this arrangement first hand when working at my first real job after college in the mid to late 80's. I worked at the Data Processing Center at Texas A&M and part of my job was to produce IBM VM/SP (mainframe) reports for college staff. I can assure you that there was a LOT more demand for information than I could possibly keep up with. And while the end user was certainly shielded from much of the complexity of the system, they certainly experienced more than their share of frustration waiting for their ration of data from the mainframe gods (this of course leads us to yet another corollary principle, the "Conservation of Frustration", which we'll save for a future discussion).

Once SQL-based relational systems came of age, one of the first attempts to resolve this issue was to provide the end-user with their own query and/or report writing tools with the hopes that they would become more self-sufficient ("ad-hoc" queries). The distribution in this type of system looks something like:

Essentially, this arrangement eliminated the bottleneck created by the IT front end resource. It also shifted some of the complexity to the back end resources (primarily due to the need for additional and more complex views and reporting structures). The reason why this was a colossal failure for the vast majority of end users is easy to understand (at least now): the typical end user proved incapable of dealing with such a high level of complexity. It quickly became obvious that shifting the bulk of the complexity onto the end user was not the path to take.

Once again, I experienced this misguided effort first hand. I worked with more than one client in the mid to late 90's who had attempted to put reporting/query tools in the hands of their end users and almost died trying. I remember one particular client, a large hospital in Denver, CO, that had purchased over 400 copies of Crystal Reports and installed them on user's PCs in the hopes they would somehow manage to find their way to their data. Wishful thinking. Fortunately for them we were able to later work a trade for a true enterprise reporting system. But back then a lot of end users simply put their "off the shelf" software back on the shelf. 

About this same time the data warehouse was born. The idea behind the data warehouse was to provide a back end data structure that essentially would absorb much of the complexity associated with traditional SQL transaction systems. The distribution then became something like this:

In this scenario, the warehouse vendor takes on some additional complexity by providing "out of the box" data structures and tools for gathering, cleansing, and organizing the data. In most cases it still requires IT front end resources to create queries (in the case of SAP BW) and other custom interfaces to allow users to access data stored in the data warehouse.

You may notice the end user is shown dealing with more complexity than the IT front end resource. On the surface, this contradicts one of the stated benefits of a data warehouse – that the end user can easily access the data in the warehouse via a simple GUI interface. In other words, the user can be self-servicing. The reason for this is we are making the assumption that many end users are not only interested in accessing the data but also in presenting the data.

Using the standard BEx interface it is fairly simple for an end user to access SAP BW data. All the user needs to do is open the query, respond to the necessary prompts, and then run the query. The data is then retrieved into the Excel spreadsheet. If this were the end of the story there would be very little complexity for the end user to deal with.

However, in many cases the end user does not simply want to leave the data in the spreadsheet "as is". The most common need is to produce printed output. Because the BEx interface is basically a "dump" of data into a spreadsheet, it requires a considerable amount of work to format the output to make it "fit for print". Something as simple as page headers and footers are quite difficult to accomplish. Pagination and labeling are also a challenge. Therefore, when the user requires formatted output of SAP BW data, the level of complexity that the user must deal with is often quite high.

In the typical rollout of SAP BW the initial end user community is quite small and unusually advanced in both their data access requirements as well as capabilities. These are the "power users". While the standard BEx interface may prove quite useful for these more self-sufficient users, it is almost a certainty that as the reports are rolled out to a wider base of users that many will find BEx to be insufficient in meeting their needs. For these users, the distribution of complexity needs to be more like the following:

This shift in complexity is primarily the result of moving the responsibility for the finished (or formatted) output from the end user back to IT front end resources, or as it says in the above graphic, the "Information Broker". I use this term deliberately to get away from the mindset that it is up to official IT "front end" resources to help users produce finished output.

I've worked on a lot of BI projects throughout the years and one thing that's always puzzled me a bit is how little companies invest into front end resources in IT. In almost every case, even with the largest companies, you can count the front end resources on one hand – or finger. Even then they are almost never dedicated to assisting the end user in accessing and formatting data. On the contrary, they typically have other responsibilities that dominate their time and attention. The end users get the leftovers. And they get a lot of complexity dumped in their laps.

I'll save this idea of an "Information Broker" for a later installment. This is partly because I've got to have some time to think it through a bit. For now, my point is simply this: if your business users are looking to IT "front end" resources to take on some of the complexity they are dealing with, they are likely going to be waiting a long time. There has to be another way.

At first glance, this can appear to be a step back to the days of transactional reporting where the bulk of the complexity fell upon the IT front end resource, and in some ways this is true. I've been reading a lot lately about the need for real ROI with the tremendous investments companies have been making in their information infrastructure, in particular with data warehouses like SAP BW. In order to insure maximum ROI it is imperative that companies find creative ways to reduce the amount of complexity presented to the broader end user base. I am convinced that a little extra investment in the right resources can, in this case, reap tremendous bottom-line benefits.

Written by Mike Garrett, Data Management Group Senior Consultant