Better BI

Chris Gerrard – Exploring the world of Better Business Intelligence

Beware “The Single Version of The Truth”

leave a comment »

This post started out as a response to the LinkedIn Business Intelligence Professionals group discussion:

Start with data warehouse or datamart? A customer wants start a BI project.
Would you advice to start with data mart or data warehouse?

I’ve long been vexed by the promotion of “The Single Version of The Truth” as the be-all and end-all of BI. This is such a ridiculous notion that it’s always been a mystery to me why otherwise reasonable, rational people would fall for it as a serious first principle.

When I started in BI in 1985 one of my first lessons was: “all data is local”, meaning that business data is perfectly relevant to the area within which the systems that capture it operate. As a FOCUS consultant it was my job to help business people understand their data, and in doing so understand their business area. Simple, straightforward, and true.

And then, out of somewhere, came this idea that there is only ONE version of the truth, and that only The Single Version of The Truth was worth knowing, worth putting in the time and effort to understand. The immediate consequence of this idea was that only by building fully consolidated data warehouses could an organization possibly be able to make sense out of its data. So Business Intelligence became synonymous with BI/DW, which has caused enormous grief, strife, and lost opportunities in the past twenty years.

So, on to the LinkedIn response:

Now we’re getting into the meat of it.

The single biggest fraud committed upon businesses that need to understand their data in the past 20 years has been the fabrication that BI is necessarily, and only, valuable insofar as the business can, through the grace of their BI technorati, grok the “Single Version of The Truth”.

This “Single Version of The Truth” is, in any organization with more than one data source, at best a red herring, at worst a pernicious propaganda that pollutes the business’s mindscape, and almost always a justification on the part of Big BI vendors for charging huge amounts of money in the pursuit of a mirage.

OK, that’s harsh.

The truth is that all data is true. Or, all data has truth in it. And all these truths are equally valid. Every business manager who needs to know what’s going on in his or her area or responsibility needs to know first and foremost what his/her data has to say about the state of that area. All BI is local.

Aha! you say. What about dirty data? What about data that isn’t the same across systems? What about data that doesn’t conform to corporate standards? Good questions all, to which I say: “Good questions. Why do these things matter? Do they negate the need to understand the data-from a business perspective-where it lives?”

This isn’t to say that collected, organized, homogenized, regularized, annotated, consolidated, updated, collated, and masticated data that’s all nice and tidy and wrapped up into the corporate abstraction of the multitude of individual truths isn’t important. It surely is. But that’s the surface of the business’s datascape, not the totality of it.

Perhaps the badness is in the term “Single Version of The Truth”. I don’t know its exact origins, but it smells of marketing-speak, created (or hijacked) for the purpose of selling ever-larger BI technology to the top corporate executives. The pitch goes something along the lines of: “You, Mr./Ms. CXO, need to know the half-dozen essential KPIs, metrics, etc. that will tell you exactly how your company is performing, internally and in the competitive arena. In order to do this you need to know “The Single Version of The Truth” and the way to achieving enlightenment is to buy our very expensive BI technology, engage our very expensive delivery teams (not to worry, we use a remote development model to keep costs down), be very, very, very patient, and then, one day in the future we’ll unwrap your big, new, shiny, oh! so very nice Truth.”

What’s a better name than TSVoTT? How about something like: “The Abstracted Companywide Data That’s Been Compiled Into The Cross-Company Common Business Context?” Or maybe not, it doesn’t have quite the same illumination of the secrets of the universe ring to it.

It might come as a surprise that I’m not against consolidated data warehouses that can present the company’s data in a form that preserves it’s salient truths within a common contextual framework. I believe that they’re good things and every company should be on the road to having one.

But the road is long and winding and littered with the corpses of the multitude of failed projects that set out to build them, and deliver TSVoTT as the first outcome.

Written by Chris Gerrard

December 28, 2011 at 7:54 pm

Posted in Uncategorized

What it takes to be a BI professional.

leave a comment »

Someone recently posted this question as a LinkedIn discussion topic here, if you’re on LinkedIn:

Hi everyone! i want to become a business analyist and want to enter into the business intelligence.
can anybody guide which steps should i take + which useful resources should i study to well achieve my targets?

Since I’ve been seeing a lot of these types of questions recently, I thought I’d post my response here, with some elaborations;

these will be inset and in blue, like this.

Martyn’s right, Business Intelligence is a profession. Like engineering, medicine, architecture, science, or being a lawyer/solicitor/barrister. As such it takes a very substantial investment of time, the expenditure of a lot of effort, and mastering a broad body of knowledge.

There’s no easy path, no certification in this or that technology, no checklist of tasks that announces: “Here is a BI professional.”

I am, admittedly, biased.

I confess: it bothers me that BI is becoming seen as the ‘hot’ area and is attracting amateurs, poseurs, carpetbaggers, charlatans, and other bad actors who are concerned with using BI as a vehicle to enrich themselves rather than as a way to help people understand their business through achieving information and insights from their data.
see: Psst – Wanna Buy a Dashboard?
and: How Big BI Fails To Deliver Business Value

I’ve had classical training in the theory and practice of programming and data processing; have been active in everything from the evolution of programming languages, and data processing spanning pre-relational databases through data warehousing, and now the post-DW, post-SQL world.

My programming and data processing started with courses in FORTRAN and COBOL programming at the University of Guelph, in Canada, in 1974-75, where my first “BI” project was analyzing bird banding data under the direction of my professor, Antonio Salvadori. I earned my undergraduate degree in Business Administration, majoring in Computer Information Systems, from Georgia State University, Atlanta, Georgia in 1985, following which I spent almost a decade working for Information Builders, Inc., first as a FOCUS consultant delivering innovative, highly responsive and adaptive reporting systems for our clients, then as a product manager developing and delivering PC and Unix FOCUS versions.

I’m not young, have been programming since the days when you had to be properly trained before they let you near a computer.
I’ve also never been satisfied with the status quo; my entire career has been spent trying to find and use better ways to do what’s possible.

I have an MBA, earned in 2007, augmenting my BBA, and have worked on and led dozens and dozens of strategic business-enabling applications, from operational line–of-business apps to top-level strategic analytics systems providing the top KPIs to executive management. For the past five years I’ve been concentrating on employing the innovations enabled by the recent emergence of direct-contact, minimal-friction, high productivity data analysis technologies and tools in the pursuit of improving the velocity and quality, and lowering the barriers and cost, of the delivery of critical business information and insights to the people who need them.

I’ve kept up with the advances as they occur.
And suffered through the backwards movements as they’ve gained dominance.

My complaint, and again I believe I agree with Martyn, is that too many people look at BI and think “Gee, that seems popular. What minimum set of claims to I need to be able to make to get someone to pay me to do that?”

This attitude reflects and has been created by the dominant paradigm that software development in general, and BI particularly, are industrial production processes that are perfectly well conducted by masses of minimally skilled technical people who frequently possess only the most rudimentary single-version single-product skills “certification-level knob-twiddling skills (below)”.

The reasons for this are beyond the scope of this post, but this mechanistic, industrial model is of particularly pernicious effect in BI.

Back to the original question. You cannot become a business analysts without making a commitment to becoming someone who lives to understand how business works, what it needs to know and do, and acquire the required variety of abilities, knowledge, and skills. Beyond that, you cannot, let me repeat that-CANNOT-be a BI professional without being a business analyst who also understands the nature and practice of analyzing data to explore it, identify patterns and anomalies, and design and implement the various media that communicate the information and insights achieved from the data; then, and only then, must you understand the technology used to implement the automated systems that are the machinery of delivering the information to the human people who need it.

Mastering BI is hard. It’s particularly hard for IT people, largely because they ARE IT people.

Of course, it’s possible to become a top-notch data warehouse builder without understanding the business context, or anything beyond the DW boundaries. BI needs these people just as architects need plumbers and surgeons need nurses. The same holds for ETL developers.

IT has a place in BI. It’s how we get things done. However, it’s not the point.

And BI platform technicians hold a special place in the BI world. I’ve worked with many, many people with certification-level knob-twiddling skills in BO, Cognos, and other platforms who are unable to grasp the basic principles of analytic information design, even of basic application user interface design, and who produce mountains of terrible analytics that fall far short of effectively communicating useful information to real human people.

This holds particularly true for remote teams who are thrown together because they can “work” BO or Cognos. There are stories to be told, but they’re best kept for dark stormy nights.

If you’re looking for a wage-earning job, by all means learn one of the technologies. Learn an ETL tool. Learn Inmon, or Kimball, preferably both. Learn Oracle, SQL Server, SQL or one of it’s dialects. Learn Business Objects, Cognos, MicroStrategy, or another platform. We need plenty of people who can do that stuff.

But please, if you do, identify yourself as someone with those skills. It’s only fair. After all, the person who shows up in the ambulance to take you to the hospital isn’t likely to tell you: “It’s OK, you’re going to be fine. I’m a Doctor”

When I was a teenage boy I met one of my friend’s Father for he first time. He was a well know and highly respected doctor. Upon meeting him I said: “I’m pleased to meet you, Mr. Dockerill.”

To which he replied: “My FATHER was MISTER Dockerill. I’m DOCTOR Dockerill.”

As I said, I’m biased.

I’m a BI professional.

Oh, one last thing. If I’m leading a BI project and you say to me “I’ll load that data into SQL.” when you mean “I’ll load that data into SQL Server.” I’m going to have a very poor opinion of your abilities. (and the same goes for using text message spelling like “u” for “you”)

The point here being: if you don’t make a distinction between a programming language and a commercial database product, I’m not likely to trust you to make distinctions between business concepts, considerations, and concerns in an area that’s outside your area of supposed competence.

Written by Chris Gerrard

December 24, 2011 at 4:43 pm

Posted in Uncategorized

BI Maxims

leave a comment »

A simple collection of lessons learned in 25 years of BI.

“Above all, show the data”—Edward Tufte

“If your statistics are boring then you’ve got the wrong numbers”—Edward Tufte (1983)

BI and people
• All BI is local.
• People are smart.
• Work with purpose.
• Deliver value early and often.
• What’s the point?
• Better information leads to better business.
• Business data owners know their data best.
• Closeness counts.
• All data is true; all truths count.

The nature of BI
• BI does not make decisions, it supports decision makers.
• BI is concerned with that area that will always be beyond the horizon of automation systems.
• BI not equal to data warehouses.

Building BI solutions
• It’s not the tool, it’s the talent that matters.
• It’s design all the way down.
• Usability matters.
• Do the (right) first thing first.
• Don’t BI blind.
• You can’t start with everything—because (if you try) you’ll never have anything.
• Beware BI’s fatal attraction: the seduction of “one version of the truth”.

The nature of (business) intelligence
Thinking is not trivial or obvious—if it was simple anyone could do it. Presumably, decision makers are in their positions because of their skill in making good decisions. It is therefore incumbent upon BI professionals to provide the best quality information to the decision makers.

Written by Chris Gerrard

December 22, 2011 at 12:54 pm

Posted in Uncategorized

Psst – Wanna Buy a Dashboard?

with one comment

Have I got a deal for you. This Dashboard is specifically tailored to people just like you, with your pressing business issues, concerns, information needs, and analytical and reporting requirements.

Our Highly Paid Consultants created this Dashboard based on their Highly Remunerative Engagements with Executives and Business Leaders in your industry, including some of your partners and competitors.

We’re confident that the money they paid us was more than adequate to cover the cost of building this Dashboard in their environments and we’re pleased to be able to offer you the opportunity to leverage our Highly Paid Experience in their employ and acquire a Dashboard Just Like Theirs that will provide Strategic Business Value to you in your Executive Capacity.

By leveraging our Industry Leading Experience and employing our Standard Methodological Process via our Institutionalized Templated Project Plans we will be able to install your Dashboard in short order, with only the wiring up of the complex, impenetrable data backplane to your organization’s unique mix of custom operational systems left to do.

And we have just the staff of regimented, organized, fully industrialized workers to perform this Too Complex For You custom wiring, with no need for your people to get their hands dirty. At the Low, Low price of How Much Do You Have To Spend?

The best part of all this is that your new dashboard is guaranteed to be the envy of Boardrooms and Executive Suites across your industry. We leverage all the full spectrum of marketware best practices in designing the Incredibly Attractive Visualizations that are guaranteed to produce the Wow! factor that only the highest-placed executives can truly appreciate.

You will have

  • Archive Quality Photo Realistic Bulb Thermometers!
  • Dial Gauges!
    licensed from the leading Italian Automotive Design Studios!
  • Glistening, dripping wet-look 4D Lozenge Torus Charts!
    skinned with Authentic Southwest Arizona Diamondback scale patterns!
  • Fully Immersive Holodeck Multi Dimensional Data Flyby Zones!
  • Many more too numerous to mention!

Step right up.
Step right up.

Written by Chris Gerrard

November 30, 2011 at 10:19 am

Posted in Bad BI

Tagged with

A note on Tableau’s place in BI history.

leave a comment »

I began this note as a response to Eric Paul McNeil, Sr’s comment on my earlier “BI Bis is dead…” post, here, in which he notes that history repeats itself, relating the emergence of Tableau to Client/Server RDBMS’ effect in the mainframe-dominated 80s.

I agree. History does repeat itself, although I see a different dimension, one more directly tied to business data analysis, the essence of BI.

Back when mainframes ruled business computing data processing shops controlled access to businesses’ data, doling it out in dribs and drabs. Sometimes grudgingly and only under duress, sometimes simply because the barriers to delivery were high, with lots of friction in the gears. Designing and writing COBOL to access and prepare data for green bar printing or terminal display was laborious and difficult.

Fast forward to the present and the modern Enterprise Data Warehouse is today’s data processing shop. Monolithic, gigantic, all-swallowing, the data goes in but business information has a very hard time getting out, for largely the same reasons: the technology is enormous, and enormously complex, requiring legions of specialized technologists to prime the pumps, control the valves, man the switches, and receive their sacrifices. Size and Complexity are universally accompanied by their partners Command and Control. Simply put, the engagement of the resources and personnel required to get anything out of modern Big BI data warehouse-based systems demands the imposition of processes and procedures nominally in place to ensure manageability and accountability, but pretty much guaranteeing that delivering the information required by the business stakeholders takes a back seat. Sad but true. As a very wise man once said: “Beware the perils of process.”

The parallels continue. In the way-back some very clever fellows realized that there was a better way. Instead of mechanically writing the same structural COBOL code to munge data into the row-oriented, nested sorting with aggregations reports in demand, they abstracted the common operations into English, resulting in the ability to write this:

with the expected results. This was the birth of the 4GL. Ramis begat FOCUS, with NOMAD in the wings. All was good in the world. Businesses could get immediate, direct access to the information in their data without needing to offer the appropriate sacrifices to the DP high priests.

Then came the dark years. The clarity was lost, the ability for business people to directly access their own business data diminished as the data became locked first in impossible-to-comprehend atomic data tables (sometimes even relational), and then in an irony too delicious for words, the massive dimensionally modeled monsters that were supposed to save the day but instead imposed themselves between the business people and their data.

And now we have the emergence of Tableau which, along with its cousins, delivers the same paradigm-changing ability to short-circuit the data guardians and establish the intimate connection between business people and their data.

-Their- data.

Once again, as BI professionals, and I consider myself lucky to have been one, albeit under different names, for 25 years, we have the opportunity to observe and honor one of the maxims of our profession:

All BI is local.

In the five years I’ve been using Tableau I’ve been watching it, and wondering what trajectory it would follow. Would it preserve its essential beauty and grace as it evolved, or would it choose a path that led it into darkness, like its conceptual ancestors. I watched FOCUS stumble and lose its way, and haven’t wanted to see it happen to Tableau. So far, so good. Tableau has preserved its integrity, its nature, even as it has grown and adopted new features. There have been a few rough edges but the company seems truly committed to continuing to create and deliver the absolute best product possible. One that helps, supports, and lives to serve as unobtrusively as possible instead of demanding that people make concessions to the machinery.

Sometimes history knows when enough is enough.

Written by Chris Gerrard

November 14, 2011 at 10:47 pm

Posted in Uncategorized

Mapping Tableau Workbooks

leave a comment »

Map of the Tableau Finance sample workbook

Map of the Tableau Finance sample workbook

As Tableau workbooks grow it becomes difficult to keep track of their various elements—e.g. Dashboards, Worksheets, and Data Sources—and the relationships between them.

I’ve found that mapping out these elements is helpful in identifying and understanding them, particularly with workbooks that I didn’t create. Benefits include understanding complex relationships that aren’t surfaced well in the Tableau UI, such as the connections between worksheets and data sources, including the scope of global filters; spotting hidden worksheets; and identifying anomalies like unused data sources, dashboards that reference no worksheets, even orphaned worksheets.

When I began working with Tableau five years ago I started out hand-drawing maps as I worked in the workbooks, recording the parts as I encountered them.

The discovery that Tableau workbooks are XML(ish) was a big step forward, making it possible to use straightforward text handling techniques to identify the parts, and some of the obvious relationships between them. It was straightforward to use grep and awk, sometimes a dash or perl, to help tease out the information. Changing a workbook’s file extension to xml and viewing it in an XML-aware tool was really helpful-Firefox is very useful for this.

Still, there were difficulties. Some of the relationships are not straightforward, and require deeper inspection and interpretation.

So I built the map-making ability into TWIS, the app I wrote to inventory Tableau workbooks, and the results are pretty satisfying. The map of the Tableau Finance sample workbook is shown rendered in PNG; clicking on it will display the PDF rendering, with much better resolution and suitable for printing. I’ve also posted links to a wide variety of Tableau workbook maps here. For more information on TWIS, its home page is here.

Orphaned Worksheets

This link is to the PDF of the Tableau Sample Workbook Science.twb (v6.0). Interestingly, it shows the existence of several worksheets that can’t be found using Tableau. Orphaned worksheets, like these, were at one time included in a Dashboard, and left stranded when the last Dashboard that referenced them was deleted. As far as I know, this is the only presentation of orphaned worksheets. As of v6.1 Tableau identifies this last-Dashboard-reference to a worksheet when deleting the last Dashboard and asks whether or not it should proceed, making the ongoing orphaning of worksheets much less likely. As of this writing it’s unclear whether or how legacy orphaned worksheets will be handled in Tableau v6.1 and beyond, but at least there’s now a way to identify them.

I’ll be posting the results of a more thorough investigation into orphaned workbooks soon.

Written by Chris Gerrard

November 14, 2011 at 10:36 am

Posted in Tableau, TWIS

Big BI is dead. But it’ll twitch for awhile.

with 2 comments

The end came last week in Las Vegas at the annual Tableau Customer Conference.

Big BI put up a valiant struggle, but it’s been unwell for some time, sputtering along, living on past glories real and imagined.

Its passing was inevitable. Big, bloated, and overripe, it never lived up to its promise of being the path to data enlightenment. Although its nominal goals were good and proper, in practice it failed to deliver. Much has been written on Big BI’s problems, including here, here, and here.

Big BI flourished, and then held on in spite of its flaws. Partly through inertia—a lot of money and time had been spent over its two decades of dominance. Partly through pervasiveness—Big BI’s proponents have been hugely successful at promoting it as the one true way. Partly through the absence of the disruptive technology that would upend the BI universe.

Big BI is brain-dead, but support systems are keeping the corpse warm.

Like all empires on the wane, its inhabitants,and sponsors haven’t realized it yet. Or, cynically, those who have are milking it for what they can get while the getting is still good. Like many empires, its demise comes not with a big bang, but with a nearly silent revolution that upends the established order—even as the Big BI promoters and beneficiaries are flush, fat, and happy their base of influence and position, wealth and power, has eroded away, leaving them dining on memories.

Big BI’s fundamental premise was always deeply flawed, erroneous when it wasn’t disingenuous or worse. The paradigm held that the only approach to achieving Business Intelligence within an organization was through the consolidation of the enterprise’s business data into data warehouses from which a comprehensive, all-encompassing single version of the truth could be achieved.

The only points of differentiation and discussion in the Big BI universe were squabbles about ultimately minor aspects of the core reality. Data Warehouses vs Data Marts. Inmon vs Kimball (Google “Inmon Kimball”). Dimensionally modeled analytical databases are relational vs no they’re not. And so on and so forth and such like.

The baseline concept remained the same: Business Intelligence is achieved by collecting, cleansing, transforming and loading business information into the integrated, homogenized, consolidated data store (Mart or Warehouse), where it then, and only then, can be fronted by a large, complex, complicated “Enterprise BI Platform” that provides a business semantic facade for the dimensional data and is the only channel that can be used to code up, create, and deliver reports, graphs, charts, dashboards, strategy maps, and the like to the business people who need to understand the state of their area of responsibility and make data-based decisions.

The overarching goal is completely correct: delivering information (intelligence) to the real human people who need it. But the reality is that Big BI has abjectly failed to deliver. With an eye to history, and another to the evolution of technology the possible end of Big BI has been in sight for some time. The history of BI is deep and rich, encompassing much more than Big BI. A brief history (mine) is here.

What happened? Why now? Why Tableau?

A number of years ago novel products appeared, sharing the concept that data access and analysis should be easy and straightforward, that people should be able to conduct meaningful, highly valuable investigations into their data with a minimum of fuss and bother.

Tableau was the best of these products, squarely aimed at making it as simple and straightforward as possible to visualize data. This simple principle is the lever that has ultimately toppled Big BI by removing the barriers other technologies impose between real human people and their data, and the friction they impose, making data access and analysis a chore instead of an invigorating and rewarding experience.

But Big BI was well established. There were institutes, academics, huge vendors to sell you their databases and Enterprise BI platforms, and huge consultancies to help you wrangle the technology.

And there was a whole generation indoctrinated in the One True Way that held as articles of faith that there is only One Version Of The Truth, that only Enterprise-consolidated data carries real business value, that business data is too important to be left to its owners: like Dickensian orphans it needs to be institutionalized, homogenized, and cleaned up before it can be seen in public.

Tableau has a different philosophy. Data is in and of itself valuable. Data analysis is the right and privilege of its owners. Data analysis should be fast, easy, straightforward, and rewarding. There is truth in all data, and all those truths are valuable.

Still, the Big BI advocates found ways to block the radical upstarts, the data democratizers. “But what about consistency across units?” “How can we trust that two (or more) people’s analyses are equivalent?”

And the most damning of all: “Pretty pictures are nice toys, but we need the big, brawny, he-man industrial controls to ensure that we at the top know that we’re getting the straight poop.” There’s so much wrong with this last one that it will take several essays to unwind it. (to be continued)

Distilled to their essence, the objections of the Big BI proponents to the use of Tableau as a valuable, meaningful, essential tool for helping achieve the real goal of getting the essential information out of business data into the minds of those who need it as fast as possible, as well as possible, are these:

Point: There must be single-point, trusted sources of the data that’s used to make critical business decisions.

Subtext: Local data analysis is all fine and good but until we can have point control over our data we’re not going to believe anything.

Subsubtext: This is an erroneous perspective, and ultimately harmful to an organization, but that’s another story, the reality is that even as misguided as it is in the entire context, there is a need to have single-source authoritative data.

Tableau’s response: the upcoming Tableau version 7 provides the ability to publish managed, authoritative data sources to the Tableau Server, available for use by all Tableau products. This feature provides the single trusted data source capability required for Enterprise data confidence.

Point: There must be absolute confidence that similarly named analyses, e.g. Profit, are in fact comparable.

Subtext: As long as people have the opportunity to conduct their own data analysis we suspect that there will be shenanigans going on and we need a way to make sure that things are what they claim to be.

The Tableau reality: Tableau’s data manipulations are, if not transparent, not deliberately obfuscated. Every transformation, including calculations, e.g. Profit, is visible within the Tableau workbook that it’s part of. There are two ways to ensure that multiple analyses are conveying the same data in the same way: the workbooks containing the analyses can be manually inspected; or the workbooks can be inventoried with a tool designed for the purpose, the results of which is an database of the elements in the workbooks, through to and including field calculations, making the cross-comparisons a simple matter of data analysis with Tableau.

Tableau does not itself possess the self-inventorying ability. There is, however, such a tool available: the Tableau Workbook Inventory System (TWIS), available from Better BI, details are available here.

So Big BI’s day is done. The interesting part will be watching how long will it take before it’s grip on business data analysis—Business Intelligence—loosens and enterprises of all types and sizes really begin to garner the benefits of being able to hear the stories their data has to tell.

Written by Chris Gerrard

October 25, 2011 at 10:44 am

Posted in Uncategorized

Tagged with , ,