In the last post, I describe what I call the Model-Driven iteration, where we take thin requirements from the end-user in the form of a user story and generate the access and performance layer, or our star schema, logically using the OBIEE semantic model. Our first several iterations will likely be Model-Driven as we work with the end user to fine-tune the content he or she wants to see on the OBIEE dashboards. As user stories are opened, completed and validated throughout the project, end users are prioritizing them for the development team to work on. Eventually, there will come a time when an end user opens a story that is difficult to model in the semantic layer. Processes to correct data quality issues are a good example, and despite having the power of Exadata at our disposal, we may find ourselves in a performance hole that even the Database Machine can’t dig us out of. In these situations, we reflect on our overall solution and consider the maxim of Agile methodology: “refactoring”, or “rework”.

For Extreme BI, the main form of refactoring is ETL. The pessimist might say: “Well, now we have to do ETL development, what a waste of time all that RPD modeling was.” But is that the case? First off… think about our users. They have been running dashboards for some time now with at least a portion of the content they need to get their jobs done. As the die-hard Agile proponent will tell you… some is better than none. But also… the process of doing the Model-Driven iteration puts our data modelers and our ETL developers in a favorable position. We’ve eliminated the exhaustive data modeling process, because we already have our logical model in the Business Model and Mapping layer (BMM).

But we have more than that. We also have our source-to-target information documented in the semantic metadata layer. We can see that information using the Admin Tool, as depicted below, or we can also use the “Repository Documentation” option to generate some documented source-to-target mappings.

When embarking on ETL development, it’s common to do SQL prototyping before starting the actual mappings to make sure we understand the particulars of granularity. However, we already have these SQL prototypes in the nqquery.log file… all we have to do is look at it. The combination of the source-to-target-mapping and the SQL prototypes provide all the artifacts necessary to get started with the ETL.

When using ETL processing to “instantiate” our logical model into the physical world, we can’t abandon our Agile imperatives: we must still deliver the new content, and corresponding rework, within a single iteration. So whether the end user is opening the user story because the data quality is abysmal, or because the performance is just not good enough, we must vow to deliver the ETL Iteration time-boxed, in exactly the same manner that we delivered the Model-Driven Iteration. So, if we imagine that our user opens a story about data quality in our Customer and Product dimensions, and we decide that all we have time for in this iteration are those two dimension tables, does it make sense for us to deliver those items in a vacuum? With the image below depicting the process flow for an entire subject area, can we deliver it piecemeal instead of all at once?

The answer, of course, is that we can. We’ll develop the model and ETL exactly as we would if our goal was to plug the dimensions into a complete subject area. We use surrogate keys as the primary key for each dimension table, facilitating joining our dimension tables to completed fact tables. But we don’t have completed fact tables at this point in our project… instead we have a series of transaction tables that work together to form the basis of a logical fact table. How can we use a dimension table with a surrogate key to join to our transactional “fact” table that doesn’t yet have these surrogate keys?

We fake it. Along with surrogate keys, the long-standing best practice of dimension table delivery has been to include the source system natural key, as well as effective dates, in all our dimension tables. These attributes are usually included to facilitate slowly-changing dimension (SCD) processing, but we’ll exploit them for our Agile piecemeal approach as well. So in our example below, we have a properly formed Customer dimension that we want to join to our logical fact table, as depicted below:

We start by creating aliases to our transactional “fact” tables (called POS_TRANS_HYBRID and POS_TRANS_HEADER_HYBRID in the example above), because we don’t want to upset the logical table source (LTS) that we are already using for the pure transactional version of the logical fact table. We create a complex join between the customer source system natural key and transaction date in our hybrid alias, and the natural key and effective dates in the dimension table. We use the effective dates as well to make sure we grab the correct version of the customer entity in question in situations where we have enabled Type 2 SCD’s (the usual standard) in our dimension table.

This complex logic of using the natural key and effective dates is identical to the logic we would use in what Ralph Kimball calls the “surrogate pipeline”: the ETL processing used to replace natural keys with surrogate keys when loading a proper fact table. Using Customer and Sales attributes in an analysis, we can see the actual SQL that’s generated:

We can view this hybrid approach as an intermediate step, but there is also nothing wrong with this as a long-term approach if the users are happy and Exadata makes our queries scream. If you think about it… a surrogate key is an easy was of representing the natural key of the table, which is the source system natural key plus the unique effective dates for the entity. A surrogate key makes this relationship much easier to envision, and certainly code using SQL, but when we are insulated from the ugliness of the join with Extreme Metadata, do we really care? If our end users ever open a story asking for rework of the fact table, we may consider manifesting that table physically as well. Once complete, we would need to create another LTS for the Customer dimension (using an alias to keep it separate from the table that joins to the transactional tables). This alias would be configured to join directly to the new Sales fact table across the surrogate key… exactly how we would expect a traditional data warehouse to be modeled in the BMM. The physical model will look nearly identical to our logical model, and the generated SQL will be less interesting:

Now that I’ve described the Model-Driven and ETL Iterations, it’s time to discuss what I call the Combined Iteration, which is likely what most of the iterations will look like when the project has achieved some maturity. In Combined Iterations, we work on adding new or refactored RPD content alongside new or refactored ETL content in the same iteration. Now the project really makes sense to the end user. We allow the user community–those who are actually consuming the content–to dictate to the developers with user stories what they want the developers to work on in the next iteration. The users will constantly open new stories, some asking for new content, and others requesting modifications to existing content. All Agile methodologies put the burden of prioritizing user stories squarely on the shoulders of the user community. Why should IT dictate to the user community where priorities lie? If we have delivered fabulous content sourced with the Model-Driven paradigm, and Exadata provides the performance necessary to make this “real” content, then there is no reason for the implementors to dictate to the users the need to manifest that model physically with ETL when they haven’t asked for it. If whole portions of our data warehouse are never implemented physically with ETL… do we care? The users are happy with what they have, and they think performance is fine… do we still force a “best practice” of a physical star schema on users who clearly don’t want it?

So that’s it for the Extreme BI methodology. At the onset of this series… I thought it would require five blog posts to make the case, but I was able to do it in four instead. So even when delivering blog posts, I can’t help but rework as I go along. Long live Agile!

If you haven’t already deduced from my first post — or if you haven’t already seen me speak on this topic — what I am recommending is bypassing, either temporarily or permanently, the inhibitors specific to data warehousing projects which limit our ability to deliver working software quickly. Specifically, I’m recommending that we wait to build and populate physical star schemas until a later phase, if at all. Remember the two reasons that we build dimensional models: model simplicity and performance. With our Extreme BI solution, we have tools to counter both of those reasons. We have OBIEE 11g, with a rich metadata layer that presents our underlying data model, even if it is transactional, as a star schema to the end user. This removes our dependency on a simplistic physical model to provide a simplistic logical model to end users. We also have Exadata, which delivers world-class performance against any type of model, and can bridge the performance gap afforded by star schemas. With these tools at our disposal, we can postpone the long process of building dimensional models, at least for the first few iterations. This is the only way to get working software in front of the end user in a single iteration, and, as I will argue, this is the best way to collaborate with an end user and deliver the content they are expecting.

Of the puzzle pieces we need to deliver this model, the first is the Oracle Next-Generation Reference DW Architecture (we need an acronym for that), which Mark has already written about in-depth here. As you browse through this post, pay special attention to his formulation of the foundation layer, which is the most important layer for delivering Extreme BI.

Oracle Next-Generation Reference DW Architecture

Foundation Layer

This is our “process-neutral” layer, which means simply that it isn’t imbued with requirements about what users want and how they want it. Instead, the foundation layer has one job and one job only: tracking what happened in our source systems. Typically, the foundation layer logical model looks identical to the source systems, except that we have a few additional metadata columns on each record such as commit timestamps and Oracle Database system change numbers (SCN’s). There are other, more complex solutions for modeling the foundation layer when the 3NF from the source system or systems is not sufficient, such as data vault. Our foundation layer is generally “insert-only”, meaning we track all history so that we are insulated from changing user requirements in the near and distant futures.

UPDATE: Kent Graziano, a major data vault evangelist, has started blogging. Perhaps with some pressure from the public, we could “encourage” him to blog on what data vault would look like in a standard foundation layer.

Capturing Change

Also required for delivering Extreme BI is a process for capturing change from the source systems and rapidly applying it to the foundation layer, which I described briefly in one of my posts on real-time data warehousing. We have a bit of a tug-of-war at this point between Oracle Streams and Oracle GoldenGate. GoldenGate is the stated platform of the future because it’s a simple, flexible, powerful and resilient replication technology. However, it does not yet have powerful change data capture functionality specific to data warehouses, such as easy subscriptions to raw changed data, or support for multiple subscription groups. You can, in general, work around these limitations using the INSERTALLRECORDS parameter and some custom code (perhaps fodder for a future blog post). Regardless of the technology, Extreme BI requires a process for capturing and applying source system changes quickly and efficiently to the foundation layer on the Exadata Database Machine.

Extreme Performance

Although I’ll drill into more detail in the next post, the reason we need Extreme Performance is to offset the performance gains we usually get from star schemas, since we won’t be building those, at least not in the initial iterations. Although Rittman Mead has deployed a variant of this methodology sans Exadata using a powerful Oracle Database RAC instead, there is no substitute for Exadata. Although the hardware on the Database Machine is superb, it’s really the software that is a game-changer. The most extraordinary features include smart scan and storage indexes, as well as hybrid columnar compression, which Mark talks about here and references an article by Arup Nanda found here. For years now, with standard Oracle data warehouses, we’ve pushed the architecture to it’s limits trying to reduce IO contention at the cost of CPU utilization, using database features such as partitioning, parallel query and basic block compression. But Exadata Storage can eliminate the IO boogeyman using combinations of these standard features plus the Exadata-only features to elevate the query performance against 3NF schemas on par with traditional star schemas and beyond.

Extreme Metadata

Extreme performance is only half the battle… we also need Extreme Metadata to provide us the proper level of abstraction so that report and dashboard developers still have a simplistic model to report against. This is what OBIEE 11g brings to the table. We have also delivered a variant of this methodology without OBIEE, using Cognos instead, which has a metadata layer called Framework Manager. As with Exadata, the BI Server has no equal in the metadata department, so my advice… don’t substitute ingredients.

Consider, for a moment, the evolution of dimensional modeling in deploying a data warehouse. Not too long ago, we had to solve most data warehousing issues with the logical model because BI tools were simplistic. Generally… there was no abstraction of the physical into the logical, unless you categorize the renaming of columns as abstraction. As these tools evolved, we often found ourselves with a choice: solve some user need in the logical model, or solve it with the feature set of the BI tool. The use of aggregation in data warehousing is a perfect example of this evolution. Designing aggregate tables used to be just another part of the logical modeling exercise, and were generally represented in the published data model for the EDW. But now, building aggregates is more of a technical implementation than a logical one, as either the BI Server or the Oracle Database can handle the transparent navigation to aggregate tables.

The metadata that OBIEE provides adds two necessary features for Agile delivery. First, we are able to report against complex transactional schemas, but still expose those schemas as simplified dimensional models. This allows us to bypass the complex ETL process at least initially so that we can get new subject areas into the users hands in a single iteration. But OBIEE’s capability to map multiple Logical Table Sources (LTS’s) for the same logical table makes it easy to modify — or “remap” — the source of our logical tables over time. So, in later iterations, if we decide that it’s necessary to embark upon complex ETL processes to complete user stories, we can do this in the metadata layer without affecting our reports and dashboards, or changing the logical model that report developers are used to seeing.

Flow of Data Through the Three-Layer Semantic Model

More to Come…

In the next post, I’ll describe what I call the Model-Driven Iteration, where we use OBIEE against the foundation layer to expose new subject areas in a single iteration. After that, I’ll describe ETL Iterations, where we transform a portion of our model iteratively using ETL tools such as ODI, OWB or Informatica. Finally, I’ll describe what I call Combined Iterations, where both Model-Driven activity and ETL activity are going on at the same time.

In a standard data warehouse implementation, whether we are walking in the Inmon or Kimball camps, some portion of our data model will be dimensional in nature; a star schema with facts and dimensions. So let me pose a question, which I think will lend itself well to diving into the Extreme BI discussion: Why do we build dimensional models? The first reason is simplicity. We want to model our reporting structures in a way that makes sense to the business user. The standard OLTP data model that takes two of the four walls in the conference room to display is just never going to make sense to your average business user. At the end of a logical modeling exercise, I expect the end-user to have a look at a completed dimensional model and say: “Yep… that’s our business alright”. The second reason we build dimensional models is for performance. Denormalizing highly complex transactional models into simplified star schemas generally produces tremendous performance gains.

So my follow-up question: can the combination of Exadata and OBIEE, or Extreme BI, actually change the way we deliver projects? We’ve all seen the Exadata performance numbers that Oracle publishes, and I can tell you first hand the performance is impressive. Can this Extreme Performance combined with the Extreme Metadata that OBIEE provides give us a more compelling case for delivering data warehouses using Agile methodologies?

To start with, I’d like to paint a picture of what the typical waterfall data warehousing project looks like. The tasks we usually have to complete, in order, are the following:

1. User interviews
2. Construct requirement documents
3. Create logical data model
4. SQL prototyping of source transactional models
5. Document source-to-target mappings
6. ETL development
7. Front-end development (analyses and dashboards)
8. Performance tuning

Raise your hand if this looks familiar. We would have to go through all these steps, which could take months, before end users can see the fruits of our labor. To mitigate this scenario, organizations will attempt to deliver data warehouses using “Agile” methodologies. What this usually means, from my experience, is a simple repackaging of the same waterfall project plan into “iterations” or “sprints”, so that the project can be delivered iteratively. So the process might look like the following:

1. Iteration 1: Interviews and user requirements
2. Iteration 2: Logical modeling
3. Iteration 3: ETL Development
4. Iteration 4: Front-end development

But this, ladies and gentlemen, is not Agile. To get an understanding of what lies at the heart of Agile development, we need to look no further than the Agile Manifesto, or the history of the Agile Movement. When examining the different methodologies, there is one major theme that permeates all of them: working software delivered iteratively. It’s not enough to simply deliver the same old waterfall methodology in “sprints” or “iterations”, because, at the end of those iterations, we don’t have any working software… software that end users can actually use to improve their job or help them make better decisions. In the example above, we still require four iterations before we get any usable content. It doesn’t matter if we’ve written some complex ETL to load a fact table if the end user doesn’t have a working dashboard to go along with it.

To apply the Agile Manifesto to data warehouse delivery, it’s the following key elements that are required for us to deliver with a true Agile spirit:

1. User stories instead of requirements documents: a user asks for particular content through a narrative process, and includes in that story whatever process they currently use to generate that content.
2. Time-boxed iterations: iterations always have a standard length, and we choose one or more user stories to complete in that iteration.
3. Rework is part of the game: there aren’t any missed requirements… only those that haven’t been addressed yet.

I’ve been conscious not to prescribe any distinct Agile methodology, though I can’t help using more Scrum-like concepts in this formulation. However, I think this list is generic enough to apply to most methodologies. Over the next few posts, I’ll discuss the necessary puzzle pieces to engage in Extreme BI, as well as how we might implement new subject area content in a single iteration. Additionally, I’ll discuss how these implementations might be reworked, or “refactored”, over several iterations to produce data warehouses that respond to user stories: what users want and when they want it.

Follow-up Posts

Agile Data Warehousing with Exadata and OBIEE: Puzzle Pieces

Agile Data Warehousing with Exadata and OBIEE: Model-Driven Iteration

Agile Data Warehousing with Exadata and OBIEE: ETL Iteration

Taking a step back and looking at the ETL process we felt there was a lot of complexity unnecessarily created by decomposing the process into a number of program units or mappings. In our view this process creates the following problems:

• A large amount of processing time was wasted on the inter-communication of these mappings.
• Unnessary temporary storage objects and created and populated in the database.
• A separate technology is required to orchestrate all the mappings.
• It encouraged multiple developers to work on the ETL process thereby increasing the risk of mis-communication and mis-aligned interfaces.

In response to this Rittman Mead have developed the One Mapping Paradigm. We believe that you should put all your ETL code into one mapping, and as such have called this approach the One Mapping Paradigm (OMP). The goal of this approach is to encapsulate your entire ETL routine into one mapping or program unit.

We feel this approach adheres to some of the fundamental tennets of software development: encapsulation (everything is in the one mapping) and decoupling (there are no external dependencies). Further it completely negates the need for old bugbear re-usability, you now don’t even need to re-use code, just use it once, all in the same mapping. Most importantly OMP will also provides a reduction in development costs: you now only need one developer.

Our extensive research has also developed a series of steps you can follow to deliver your One Mapping. You should note that the One Mapping that OMP generates will be extremely complex, only by following these can you address the complexity of the mapping that will be generated.

OMP follows a black hole development approach where it is crucial for the developer to do as much development as possible without any outside interfere from either peers or the business. This allows the developer to focus solely on the development task in hand, which is a must when developing extremely complex code. It is also essential that the developer is allowed to proceed as far through the process as possible without stopping for other distracting activities like testing. In order to follow the OMP I have built the following example using Oracle Warehouse Builder.

• Step 1: source objects – create new mapping a drag all your source objects onto the canvas – it is important to arrange these in a straight line on the left hand side of the canvas.
• Step 2: add all your join operators to combine the data. A couple of tips here, (1) add predicates into the join conditions to avoid using filter operators (2) keep the data transition lines as straight as possible for performance reasons.
• Step 3: add any expression or transformational operators required – these should really be added to the middle of the canvas.
• Step 4: add all your target tables – these are added to the right hand side of your canvas. You are in the home straight now, but you may find this the trickiest part and we recommend using at least a 29″ monitor to complete this process.
• Step 5: unit test – note there is no orchestration or integration required, as you only have One Mapping.
• Step 6: release to production – you can just release you mapping straight into production, overwriting whatever was there before. There is no system or integration testing required as there is only one piece of code. UAT is further bypassed as your unit testing verifies whether the entire ETL process works or not.

We are looking for beta testers for this concept, so if you want to try the OMP for your ETL code, please contact me at omp@rittmanmead.com.

The Balancing Act

Put simply, if BI is delivered well it should generate two things…..firstly, answers [no pun intended] to the questions that the business know need to be answered….but then quickly followed by questions (the silent tidal wave of all the things that the business always wanted to do/know/ask, but just never had the capability).  The up side to this is that, given a data model that withstands the test, the toolsets allow end-users to deal with the tidal wave themselves.  The down side is that this runs the risk of opening up some sort of Pandora’s box…..badly written reports, measures calculated differently by different people, a back catalog of reports written but no longer used, inefficiency, inefficiency, inefficiency…  The whole things just needs managing.

And this is where the tricky balancing act arises.  For me, it is essential that the BICC takes responsibility for implementing a set of policies, processes and roles and responsibilities, all moulded into a Governance Framework, that lays out the ‘rules of engagement’.  Policy should cover off the main building blocks which will feed into the solution design (such as the security and access, data retention, SLA’s, metric ownership, report accreditation etc.).

The processes required to underpin your BI usage will naturally fall into three main buckets:  those that are internal to the BICC; those that relate to end-users and their relationship with the BICC; and those that relate to how the BICC interacts with IT in the delivery of ongoing change;  Again, the processes can’t really be prescribed but should be drawn up based on what is going to work and be adopted (factoring in BICC scope, resource profiles, organisational culture and so on).  However, the types of area that might be considered may be;  In the first bucket: how data quality is monitored; usage tracking; catalog housekeeping;  In the second bucket: report creation; change request; 1st line support; master data correction;  In the third bucket:  Work Take-On; Operational Schedule Management; 2nd/3rd Line Support; and so on.

Further to this, it is essential that the users of the system know what they are able to do and how they should be doing things…..in other words, they need to be trained and understand best practice!  And all I’ll say is that training needs to be seen as something more than a one-time session, scheduled at some point just before the system is implemented.  To my mind, the single most important difference between BI solutions and other transactional systems is that BI solutions are organic – they grow and change shape over time and along with the business.  For this reason, when the BICC thinks about its training strategy, it should be thinking about how it communicates out change and best practice in order to maintain effectiveness.  And whilst we’re on the subject of communication….

Getting Passionate

I believe that the best BI solutions (again technology set to one side) are where the business has a level of enthusiasm, whether natural or developed, for information and what can be done with it.

Therefore, and in addition to all the other strands of activity that might fall under your BICC, it is incumbent on the BICC to evangelise.  To advocate the usage of information for the benefit of the business.  And to fight the corner for investment in BI (regardless of whether the BICC actually owns a budget or not).  If the BI solution is well defined, designed and delivered, it will start its life being highly thought of by the business.  If it is not then protected, guided and grown, it runs a risk of slowly becoming delinquent and returning a reducing benefit.  At the outset, the BI solution will be used heavily.  If it is not then grown in the right way, it will become less useful, less useable and return a reducing benefit.

So, one of the key roles of the BICC has to be to communicate.  It has to develop methods of communicating out strategy, policy, process, change and best practice.  It has to develop methods of capturing feedback, to understand what works and what doesn’t, what is being used and what isn’t, what new data is needed and what existing data isn’t.  It has to develop methods of communication which ensure that change is defined, designed, delivered and managed efficiently and within the timeframes that the business needs.  Finally the BICC needs to develop methods of communicating with the outside world.  With solution vendors, so that the technology roadmaps can continually be compared and re-evaluated.  With 3rd Party BI partners, who might be engaged for implementation, support or ongoing consultancy.  With other, like-minded user organisations, in order to support innovation.  And, if the strategy is such, with suppliers, customers and others in the value chain.

On the bright side, there are usually people scattered around your organisation who are natural ‘data’ people.  It’s always a good idea to engage these people in what you are doing, not least as they may feel the most threatened by your BICC organisation.  They should be seen as potential ‘BI champions’ and future advocates of BI within their given departments and may prove a useful vehicle for communicating out the BI message.

Back To The Top

To summarise, BI technology comes in all sorts of shapes and sizes, but the organisations using them come is infinitely more shapes and sizes.  I would argue that, regardless of shape or size, to make best use of BI technology, a mechanism is required to manage its usage and to coordinate any associated efforts and decisions.  However, it is important to recognise that there is a continuum between a full-blown BICC on one hand, and nothing on the other.  For any BICC-like initiative to be successful, the specifics need to be guided by the size, shape, culture and objectives of your organisation.  If you are competing in a fast moving environment, where information is the life-blood of your business, then your BI is likely to need greater coordination and may, therefore, warrant a heavy BICC investment.  If your organisation see’s reporting as a necessity rather than a differentiator, then the management and coordination will be completely different.

The historical challenges for the acceptance of the ‘BICC ideal’ revolved largely around justification of the costs, especially where the benefits tend to be seen as indirect and somewhat intangible.  However, an ever increasing number of organisations are looking towards their information asset as a source of competitive advantage and, as a result, are seeing the value not just in BI per se, but also in the coordination of its BI efforts.  

So, if you are embarking on a strategic BI initiative, then it’s important that you look beyond the technology and the development lifecycle.  Start thinking in terms of how your BI is going to be used and how it is going to be managed as it evolves over time.  And, if you have implemented BI but seem to be seeing a reducing benefit over time, then it might be worth looking at whether your BI is becoming delinquent.  If it is, it might be time to do something about it.

If you have already implemented, are in the middle of implementing, or planning to implement a BICC-style organisation, I’d love to hear from you to understand some of your experiences.

BI Strategy

The starting point for any non-tactical BI initiative should be the development of a BI strategy map.  And it’s worth noting that this statement holds, regardless of the circumstance…whether a greenfield project, a migration to a new, pre-selected technology stack or where heritage applications are remaining in place.  That’s to say – the BI Strategy should be technology agnostic, not technology driven.  The form that your strategy takes is likely to be heavily influenced by the type of BI organisation that is planned or already in place.  Where a full-blown BICC is being built, then it is likely that the strategy will be a more formal mission statement.  Where something more subtle is being planned (say, a sub-team within an existing department), then we may be talking about some slide-ware which is used to disseminate the message. 

Regardless of form, the BI Strategy should be all about laying down the foundations for:

➡  How information will be utilised by the business to drive competitive advantage

➡  How the business is going to make best use of its information and technology resources

➡  What ‘genetic makeup’ the BICC will have (i.e. will it be business driven, IT driven or, most realistically, blended?)

➡  How the BICC is going to enable the business to leverage maximum value from the investments made in BI

➡  How strategic delivery will be resourced (internally, externally or blended)

Clearly, the BI Strategy must be aligned with the organisations wider strategic objectives and it will often help in communicating the message, if a) a high level vision of the delivery roadmap and b) the BICC’s roles and responsibilities can also be developed at this stage.

 Information Delivery

The second thing to be recognised is the importance of delivering information in a consistent, coherent and accurate way.  To me, this is central in building the business’ confidence in the BI they use.  Confidence will increase usage which will result in reliance and then dependency.  So, ultimately, ‘information delivery’ will factor heavily in the success or failure of the BICC itself.  Unfortunately, it is also probably the trickiest thing for the BICC to achieve.  And here’s why.

Under the ‘Information Delivery’ umbrella I would group the following activities:  Solution & Data Architecture; Data Quality Management; Data Stewardship; Metadata Management; Presentation & User Experience.  In a nutshell, then, this is where the BICC gets its most technical and so (if predominantly a business lead team) it is where the BICC will be least knowledgeable and least self-sufficient.  This is why the ‘genetic makeup’ of the BICC, as I have described it previously, is so important.  It also seems to be the area of most debate….should the BICC be 100% business focussed?  Should it be an extension of the IT department?  Should it be a blend and, if so, what is the balance of power?  The answer will inevitably vary from organisation to organisation (and, again, there is probably no “one size fits all” answer), but my view aligns with the consensus – that a BICC should be a blended team, but business lead.  As a minimum, senior people to represent solution and data architecture should be incorporated (unless you are lucky enough to have someone who can do both!). Such a composition should help to ‘sell’ the BICC at senior levels, smooth its inception and also ensure that the right architectural decisions are made along the way.  As vendor solutions and product portfolio’s continue to grow, the number of ways of ‘skinning the cat’ can increase exponentially.  A bad ‘technology’ decision could constrain future opportunities, whilst a decision based purely on the commercials will likely result in the same outcome.  Decisions in BI, therefore need to be balanced and require the representation of both business and technology expertise.

Data architecture is essential in the information delivery equation.  A data warehouse with missing data will not be capable of supporting user requirements.  Even where there is built in latency to allow for those unexpected requests, if the data warehouse is poorly modeled your users will be equally constrained and frustrated.  The same is true of your metadata design – build your semantic layer to support the reporting requirements alone and your users will not be able to easily fulfill their ad-hoc analyses.  DW and Metadata design need to go hand in hand and, if not, will result in an increased overhead for IT and BICC respectively.  Data quality is important in protecting the integrity of the information being consumed and user experience will have a large say in how comfortable the user community feel about actually using the BI.

Putting all this together makes ‘Information Delivery’ certainly the most tangible and probably the most complex aspect of the BICC’s remit.

Next time, I’ll look at some of the less tangible aspects, maintaining some level of control and generating passion for information.

In this series of blog posts (my first since joining Rittman Mead in October), I’m going to take a look at some of the ways in which BI can be managed effectively, by considering the place of the Business Intelligence Competency Centre (BICC) in the real world.

Much has been written about BICC’s over recent years and, although there are one or two variations on the theme, the general consensus as to the purpose of a BICC and the reasons why enterprises would implement a BICC are pretty well established.  I’m not going to pick over the debate, suffice to say that my crystalisation would be this: 

A BICC, or it’s equivalent, should exist to achieve four key fundamentals:

➡  To architect and then champion the enterprise BI Strategy

➡  To ensure that information is delivered to the business consistently, coherently and accurately

➡  To manage the tricky balancing act between end-user self-sufficiency, freedom and flexibility on the one hand and business control and alignment of efforts on the other

➡  To generate and maintain a level of passion for information within the business…..or (as I appreciate that this may be a tall order!) at least a level of awareness and interest. 

Beneath each of these pillars sits a number of key elements and core activities, which I’ll address in subsequent posting’s over the next few days.  But before diving in, there’s an important question that I think needs addressing……

So, Where Are All These BICC’s, Anyway?

It occurs to me that despite all of the hype and the discussion underpinning BICC theory, the anecdotal evidence suggests that the uptake in and establishment of BICC’s has, at very best, been patchy.  There is certainly no obvious correlation between the uptake of enterprise BI and the implementation of BICC’s…..the “every good BI project must have a BICC” maxim definitely doesn’t hold true.  And talking to people on the subject, it becomes obvious that a fair number of people are aware of the BICC concept, but know of few who have implemented (or at least attempted to implement) one. 

But I wonder whether this view is valid or not.  Based on my experiences, I’d suggest that there is already a significant amount of BICC-like activity happening in organisations….it just isn’t being driven under a ‘BICC banner’ and invariably does not adhere strictly to the theory.  In fact, I suspect that a lot of this activity is happening without consciousness of the BICC phrase – it’s just being done because it makes sense.  Indeed, I have been in organisations where the suggestion of something called a ‘Business Intelligence Competency Centre‘ would have gone down like a lead balloon.  However, talk to people about the principles, imperatives and benefits behind the BICC and I would receive near universal agreement.  And this is principally down to the fact that, for any business serious about using information to drive competitive advantage, the benefits case for a BICC is, in the main, very compelling.

 I suspect that the biggest challenge facing most historical BICC initiatives has been in the quantification of its value proposition.  Aside from efficiency gains arising from the delivery of BI systems, the benefits are largely intangible (how you put a figure against a benefits line of “Making Better Decisions” may have to wait for a future blog!).  And when compared to the costs associated with setting up and then running a BICC, in most cases, the equation just couldn’t be balanced.  It invariably depends on the business understanding the value that can be unlocked through its information assets.

Importantly, in my example above, the organisation had a near unstoppable thirst for information and also wore the psychological scars from a history of loosely controlled information (you know the story…..a proliferation of MS Access, departmental reports which were irreconcilable with each other, let alone back to the core systems, board meeting disputes over whose sales figure was correct etc. etc.)

So, if businesses are undertaking BICC-like activities (whether structured or not, consciously or not, whether called a ‘BICC’, ‘MIS Team’, ‘Group Reporting function’ or whatever), what are the things that they are trying to achieve?  And if you are about to embark on a strategic BI initiative, what are the non-technology things that you really should be thinking about? 

In my next post, I’ll return to look at the four fundamental areas in more depth, starting with BI Strategy and Information Delivery…

“What about iterations within the Delphi 18 stages? Do you have a structure for incorporating feedback from later stages into earlier stages i.e. learnings from the build phase being incorporated into and refining the design?”

What Damien was describing is really what governance is all about. Governance is how you go about improving your methodology.

Before you can get to that stage, you have to have a documented methodology and you have to use it. Sadly, not every project and/or IT shop can attest to talking the talk AND walking the walk.

Generally speaking, when an IT shop is operating without a well defined methodology, and it completes a project on time, it is due to heroic efforts and long hours on the part of the development team.

In this kind of environment, some of your best developers will get fed up leave you to work in better organized IT shops. Over time your development pool will degrade in quality and you will be less and less likely to bring projects in on time.

Sometimes, the business gets so fed up they outsource development and that’s not going to fix anything, if the team they outsource to doesn’t use a methodology either.

But let’s assume that you have a methodology and it is standard business process to use it on all your data warehouse projects.

The next step is for you to take measurements. This gets a bit tricky for us IT folks. It’s pretty easy to figure out that a widget factory should count how many widgets are produced/hour/day, but what should a data warehouse team be measured on?

Lines of code written/hour? Mappings deployed/week?

For data quality measurements – perhaps the number of reference data rows rejected/batch?

I was on a project once, where under pressure from the business a decision was made to skip the data profiling phase to “save time”. They ended up spending 18 months in UAT. There’s a new phrase for what happens to your project plan when you build a data warehouse without performing data profiling – they call it “Extract, Load and Explode”.

If the source data is really bad, IMHO it’s better for the company to spend this year’s budget fixing the data at source in the OLTP, and re-call the data warehouse team next year when the data is ready for prime time, the data stewards are empowered and a data governance policy is in place – but I digress.

Here are some industry standard warehouse measurements, but they are at a high level outside of the actual development processes:

• Do your data warehousing initiatives succeed more often than they fail?
• Is there a successful data warehouse implementation within your company?
• Is the successful data warehouse sustainable?
• Does your company know how much it is spending on data warehousing?
• Does your company have centralized IT groups?
• Does your company have an enterprise-wide meta data repository that supports the data warehouse and the operational systems?
• Has the quality of the data within your data warehouse improved over time?
• Do your programmers understand their role in helping the company reach its goals?

I think that a good DW development measurement might be hours in UAT per deliverable. If the design was really good, then theoretically the developers would build exactly the right thing, error free, the first time and the code and data should pass UAT and move into deployment quickly.

If the code is kicked back (fails UAT) that’s not a good thing. So the number of mappings that failed UAT or deployment would be good measurements as well.

Another measurement might be project plan accuracy. We all know how incredibly difficult it is to estimate how long it’s going to take to build these data warehouses and data marts. Good methodologies include processes to aid the project leader during the planning phase, but I have to say, having been in this position myself, if I have worked with the team previously, my plan is going to be much more accurate than if I am new to the team.

So now let’s assume you have finished your project, the code is deployed, the batch runs so flawlessly at night that the support & maintenance team is bored, your well trained business users are happily referring to their data mart documentation and you have gathered some meaningful measurements.

Additionally, you find yourself with time and resources available to support the governance phase.

The first step is to gather the lessons learned from the project. You need to document what went well and what didn’t and get everyone to agree. Next the team should brain storm what changes should be made to the methodology to improve things for next time.

These changes to the methodology have to be well documented and introduced carefully so that everyone understands the differences between the old way and the new way and why it changed. Sometimes the new way involves new tasks that aren’t much fun like going to meetings to gain consensus, documentation, change control procedures, coding standards, loosing access to production, business prioritization of tasks, issue log maintenance and time tracking. So your team has to buy into them.

Finally, you have to wait for the next project to complete with your changes in place and compare that project’s measurements to the first project’s to see if your methodology changes were effective.

Whew! Governance is hard. . .

That’s why so few IT shops are at that level of maturity. It is estimated that less than 2% of IT projects include a governance or methodology improvement phase.

To improve a business process such as widget production you can change the process and see results in hours or days. But changing IT methodology is harder. Each project takes longer, the team members are always changing and we never build the same thing twice.

Ideally, to be sure our methodology improvements worked, we should put a data warehouse team on a deserted island (well, somewhere isolated with really good connectivity) and make the team build a data mart. Then you’d have to improve the methodology, and have a similar team come along and build the exact same data mart. When they were finished you’d have to compare the measurement results.

That would be the only way to ensure that the process had indeed been improved, but even then I’m sure you’re thinking of how many parameters wouldn’t have been properly controlled the second time around.

Difficulties in measuring IT development efforts notwithstanding, for about 40 years now, businesses have wanted a way to determine if we IT folks are any good and figure out how to make us faster and cheaper. (I understand that this desire developed a few weeks after the first computer was installed.)

Back in 1989 Watts Humphrey first wrote about measuring how well IT projects were run in order to be able to assess the ability of IT government contractors. He called it the Process Maturity Framework. Then in 1995 Mark Paulk, Charles Weber, Bill Curtis, and Mary Beth Chrissis published their book on the Capability Maturity Model (CMM) which is based upon Humphrey’s work.

The book outlines exactly how to determine what level of maturity your IT organization is at, and how to improve things and move IT up from one level to the next.

There are five levels of maturity defined:

1. Initial (chaotic, ad hoc, heroic) the starting point for use of a new process.
2. Repeatable (project management, process discipline) the process is used repeatedly.
3. Defined (institutionalized) the process is defined/confirmed as a standard business process.
4. Managed (quantified) process management and measurement takes place.
5. Optimizing (process improvement) process management includes deliberate process optimization/improvement.

Jennifer’s easier to remember CMM definitions:

1. Dilbert cartoons are posted everywhere – (keep your CV/resume dusted off)
2. Talking the talk – (the methodology is written down)
3. Everyone is walking the walk – (using it)
4. Managers are measuring how far you can walk
5. Everyone’s getting predictably better at walking

The good news is that if you are at level 3, (everybody is walking the walk) you are working for a world class IT shop!

Figure 1 – Applying CMM Levels to Data Warehousing, David Marco

So to answer Damien’s comment, yes we have a structure for incorporating feedback from later stages into earlier stages. The last tasks on our project plan template are all about gathering the lessons learned and modifying the methodology for next time.

Sadly, so far I have always been asked to remove these tasks from my final signed off versions of the project plan in order to save money. – irony

I was lucky enough to join Rittman Mead last autumn as a Principal Consultant, and one of my responsibilities is to refine our approach and methodology. I’ll be writing about BI and data warehousing methodologies over the next few months and I’d like to talk about Delphi, a methodology I’ve championed within RMC for those projects when we take responsibility for the delivery, or those when we support members of the client team who are responsible for project leadership.

A solid methodology provides a standardized step-by-step plan for approaching projects consistently, clearly defined roles and responsibilities and a framework that enables client and project leadership to maintain control. But at the same time it shouldn’t be so detailed and overly encumbered with so many checks and balances that it slows delivery down or even brings it to a stand still.

I look at methodology as a way of reducing the frustration that everyone feels during IT projects:

•    The business users hate it when we build something they didn’t ask for
•    The developers hate it when they have to re-do things over and over again because the instruction they got weren’t clear or detailed enough
•    The project leadership hates it when the business keep asking for more and more “new features” to be thrown in at the last minute
•    The business management hate wondering what we are up to and hate weeks or months of no news
•    The business sponsors don’t like worrying that they may have wasted money
•    The support team hate having business critical applications thrown at them with little or no documentation and inadequate training

Many of you will have seen the famous “Tire and Tree” cartoon (apparently it has been in circulation since the mid ‘60s!), but it’s well worth including here because it so well exemplifies the dangers of poor communication – which IMHO characterizes all verbal communication:

Often when we work on contract engagements we are asked to use our client’s methodology. In this case, I always look for ideas and suggestions that I can recommend to them to improve their methodology and this advice has always been well received.

Other times the client either has no methodology, or does not practice using one effectively. In these projects it is really important that you try to at least apply a bare bones minimum set of methodology deliverables, not only for them but also for your own protection.

If you can at least minute important design meetings and send confirmation emails of all design changes as you go along, at least you’ll have something to fall back on if all £$%& breaks loose. If I had to choose one deliverable only, it would be the functional or design specification. This document not only encourages discussion about what is going to be developed, but by its very nature it helps to set scope.

If I had to choose one communication device it would be a weekly status update email that copies in everyone who might be wondering what you are up to and why it’s taking so long.

The RMC Delphi Methodology enables project leaders to pick and choose what deliverables and level of status reporting is applicable for each engagement. It’s important to tailor the scope of deliverables and communication for each project.

To aid in a smooth progression through project phases, the Delphi Methodology incorporates exit reviews at the end of each phase. Prior to closing the phase on the project plan, the project manager is required to complete an exit review to ensure all required steps have been taken and that open items are addressed. The review includes development of plans for resolving open issues, as well as a review of project schedule and budget, including managerial approval of changes.

The Delphi Methodology defines standardized responsibilities of logical roles in the data warehouse development for both the Rittman Mead Consulting team and the client team. There are 18 phases within Delphi, which can be fine tuned and/or cut down in scope to fit even the smallest project. The 18 phases are:

1. Project Startup
2. BI requirements definition
3. Data analysis requirements definition
4. Data architecture
5. ETL architecture
6. BI architecture
7. Infrastructure architecture
8. Design data model
9. Design ETL
10. Design BI
11. Design Infrastructure & proof of concept
12. Deployment & Testing Preparation
13. Build ETL
14. Build BI
15. System Test
16. User Acceptance Test
17. Train and deploy
18. Maintenance & Support

It’s important to go through each of the phases, even if vastly reduced in scope, to ensure that you deliver a quality product every time. Whether you are working on a two-year enterprise-wide data warehouse or just adding a new level of aggregation to an existing data mart.

Each phase of the methodology contains a list of deliverables and working documents with links to document templates, such as project plans, statements of work, issues logs, and checklists.

Using templates ensures quality documentation across projects and faster, easier development of documentation based on quality documents from past successful projects. As we complete each project, we try to take the time to update our templates with any improvements we have discovered. Although, I admit this non-funded phase of project work is the most likely to be neglected.

In conclusion, I like to sum up by saying that the key to a successful data warehouse project is a comprehensive methodology that applies best practices and proven experience to guide a data warehouse project team from project launch through to deployment.

I’d like to also say to all the die-hard brilliant developers out there who’d rather walk over hot coals than spend a week doing documentation, try to think of methodology as way to ensure that no one gets frustrated – especially you!

I intend to come back to this over the next few weeks and months and add detail on to the 18 Phases I’ve listed. I’m also keen to get feedback from you as your comments will help me to improve the Delphi Methodology.

In the interim, the customer can work on data quality; they know what they need to report on (it’s in the project mandate!) and they know the sources of information (their operational systems) so they can start to verify that all of the required facts can be found in the source systems and more importantly look at the data content and assess “fitness for purpose”. If data defects are found then it may be possible to get them fixed before the serious construction of the ETL layer starts. Besides, the knowledge of source and target gives a good head start in the specification of ETL interfaces.

One particular issue they might meet, and one that is sadly far too common across many business sectors, is the use of operational systems that do not enforce data integrity. For whatever reasons there is just too much freedom in data entry and although it may not affect the operational system much it really can cause problems when you try to aggregate information on the BI system.

But how do we deal with this? Recently I joined in on a thread on one of LinkedIn BI groups where it was proposed that a “receive garbage, store garbage strategy was adopted” – in my opinion this might be OK for a mature BI system where users can understand that the reporting accurately reflects the source, but for a new venture into BI? To me, this seems to be too much a risk; it might be that the new BI users do not have sufficient exposure to the source systems to realise that the data is at fault on the source. We could prevent data that fails a quality threshold from loading on the BI system, but then we would show incomplete results which although correctly aggregated are misleading because of omission; at the end of the day load policy is a business choice. If we go with the “reject poor data” route we should seriously  think about providing a data quality dashboard on the reporting system to indicate the numbers of records that failed to be loaded and drill-down to the reasons why they failed.

So what do we do with data that fails the quality standard? Ideally, we should get it fixed at source. Auto-fixing on load is possible, but then we need to think about data governance and the possible ‘trust’ problems of the data being not aligned with the source. Maybe you could ‘standardise’ country names and other columns on loading; I’ve seen systems with ‘USA’, ‘U.S.A’, ‘U S A’, ‘US of A’, ‘America’,  and ‘US’ in the country data feed and that’s before we get to the mis-keying of ‘United’ to get ‘Untied’!  But maybe that sort of improvement in quality should also be available to operational systems users.

For this customer, I have suggested that they construct a source to BI target matrix and include some basic traffic light measures on the source data:

• How good is it?
• What sort of errors are present; missing items, typographical errors, missing or incorrect parents, inconsistent use of names, even data entered in the wrong fields.
• How important is it to be correct in the BI system; for example street address can not be aggregated in reporting and we may not be going to use BI to create mailing lists, but postal code (or a sub string of it) can be used to aggregate people by location areas.
• How important is it to be correct on the operational source – do we need to apply the corrections at source to improve the operational use of the system

But this type of quality review may not tackle the data problem that is probably hardest to deal with what is a correct fact? How do I know if house value of £20,000 is reasonable (it  could be in a shared ownership scheme) or £2,0000,000 or £20,000,000? We could set a validation range, but where is there that point that one penny more is obviously wrong, but the current value OK?