“What about iterations within the Delphi 18 stages? Do you have a structure for incorporating feedback from later stages into earlier stages i.e. learnings from the build phase being incorporated into and refining the design?”

What Damien was describing is really what governance is all about. Governance is how you go about improving your methodology.

Before you can get to that stage, you have to have a documented methodology and you have to use it. Sadly, not every project and/or IT shop can attest to talking the talk AND walking the walk.

Generally speaking, when an IT shop is operating without a well defined methodology, and it completes a project on time, it is due to heroic efforts and long hours on the part of the development team.

In this kind of environment, some of your best developers will get fed up leave you to work in better organized IT shops. Over time your development pool will degrade in quality and you will be less and less likely to bring projects in on time.

Sometimes, the business gets so fed up they outsource development and that’s not going to fix anything, if the team they outsource to doesn’t use a methodology either.

But let’s assume that you have a methodology and it is standard business process to use it on all your data warehouse projects.

The next step is for you to take measurements. This gets a bit tricky for us IT folks. It’s pretty easy to figure out that a widget factory should count how many widgets are produced/hour/day, but what should a data warehouse team be measured on?

Lines of code written/hour? Mappings deployed/week?

For data quality measurements – perhaps the number of reference data rows rejected/batch?

I was on a project once, where under pressure from the business a decision was made to skip the data profiling phase to “save time”. They ended up spending 18 months in UAT. There’s a new phrase for what happens to your project plan when you build a data warehouse without performing data profiling – they call it “Extract, Load and Explode”.

If the source data is really bad, IMHO it’s better for the company to spend this year’s budget fixing the data at source in the OLTP, and re-call the data warehouse team next year when the data is ready for prime time, the data stewards are empowered and a data governance policy is in place – but I digress.

Here are some industry standard warehouse measurements, but they are at a high level outside of the actual development processes:

• Do your data warehousing initiatives succeed more often than they fail?
• Is there a successful data warehouse implementation within your company?
• Is the successful data warehouse sustainable?
• Does your company know how much it is spending on data warehousing?
• Does your company have centralized IT groups?
• Does your company have an enterprise-wide meta data repository that supports the data warehouse and the operational systems?
• Has the quality of the data within your data warehouse improved over time?
• Do your programmers understand their role in helping the company reach its goals?

I think that a good DW development measurement might be hours in UAT per deliverable. If the design was really good, then theoretically the developers would build exactly the right thing, error free, the first time and the code and data should pass UAT and move into deployment quickly.

If the code is kicked back (fails UAT) that’s not a good thing. So the number of mappings that failed UAT or deployment would be good measurements as well.

Another measurement might be project plan accuracy. We all know how incredibly difficult it is to estimate how long it’s going to take to build these data warehouses and data marts. Good methodologies include processes to aid the project leader during the planning phase, but I have to say, having been in this position myself, if I have worked with the team previously, my plan is going to be much more accurate than if I am new to the team.

So now let’s assume you have finished your project, the code is deployed, the batch runs so flawlessly at night that the support & maintenance team is bored, your well trained business users are happily referring to their data mart documentation and you have gathered some meaningful measurements.

Additionally, you find yourself with time and resources available to support the governance phase.

The first step is to gather the lessons learned from the project. You need to document what went well and what didn’t and get everyone to agree. Next the team should brain storm what changes should be made to the methodology to improve things for next time.

These changes to the methodology have to be well documented and introduced carefully so that everyone understands the differences between the old way and the new way and why it changed. Sometimes the new way involves new tasks that aren’t much fun like going to meetings to gain consensus, documentation, change control procedures, coding standards, loosing access to production, business prioritization of tasks, issue log maintenance and time tracking. So your team has to buy into them.

Finally, you have to wait for the next project to complete with your changes in place and compare that project’s measurements to the first project’s to see if your methodology changes were effective.

Whew! Governance is hard. . .

That’s why so few IT shops are at that level of maturity. It is estimated that less than 2% of IT projects include a governance or methodology improvement phase.

To improve a business process such as widget production you can change the process and see results in hours or days. But changing IT methodology is harder. Each project takes longer, the team members are always changing and we never build the same thing twice.

Ideally, to be sure our methodology improvements worked, we should put a data warehouse team on a deserted island (well, somewhere isolated with really good connectivity) and make the team build a data mart. Then you’d have to improve the methodology, and have a similar team come along and build the exact same data mart. When they were finished you’d have to compare the measurement results.

That would be the only way to ensure that the process had indeed been improved, but even then I’m sure you’re thinking of how many parameters wouldn’t have been properly controlled the second time around.

Difficulties in measuring IT development efforts notwithstanding, for about 40 years now, businesses have wanted a way to determine if we IT folks are any good and figure out how to make us faster and cheaper. (I understand that this desire developed a few weeks after the first computer was installed.)

Back in 1989 Watts Humphrey first wrote about measuring how well IT projects were run in order to be able to assess the ability of IT government contractors. He called it the Process Maturity Framework. Then in 1995 Mark Paulk, Charles Weber, Bill Curtis, and Mary Beth Chrissis published their book on the Capability Maturity Model (CMM) which is based upon Humphrey’s work.

The book outlines exactly how to determine what level of maturity your IT organization is at, and how to improve things and move IT up from one level to the next.

There are five levels of maturity defined:

1. Initial (chaotic, ad hoc, heroic) the starting point for use of a new process.
2. Repeatable (project management, process discipline) the process is used repeatedly.
3. Defined (institutionalized) the process is defined/confirmed as a standard business process.
4. Managed (quantified) process management and measurement takes place.
5. Optimizing (process improvement) process management includes deliberate process optimization/improvement.

Jennifer’s easier to remember CMM definitions:

1. Dilbert cartoons are posted everywhere – (keep your CV/resume dusted off)
2. Talking the talk – (the methodology is written down)
3. Everyone is walking the walk – (using it)
4. Managers are measuring how far you can walk
5. Everyone’s getting predictably better at walking

The good news is that if you are at level 3, (everybody is walking the walk) you are working for a world class IT shop!

Figure 1 – Applying CMM Levels to Data Warehousing, David Marco

So to answer Damien’s comment, yes we have a structure for incorporating feedback from later stages into earlier stages. The last tasks on our project plan template are all about gathering the lessons learned and modifying the methodology for next time.

Sadly, so far I have always been asked to remove these tasks from my final signed off versions of the project plan in order to save money. – irony

I was lucky enough to join Rittman Mead last autumn as a Principal Consultant, and one of my responsibilities is to refine our approach and methodology. I’ll be writing about BI and data warehousing methodologies over the next few months and I’d like to talk about Delphi, a methodology I’ve championed within RMC for those projects when we take responsibility for the delivery, or those when we support members of the client team who are responsible for project leadership.

A solid methodology provides a standardized step-by-step plan for approaching projects consistently, clearly defined roles and responsibilities and a framework that enables client and project leadership to maintain control. But at the same time it shouldn’t be so detailed and overly encumbered with so many checks and balances that it slows delivery down or even brings it to a stand still.

I look at methodology as a way of reducing the frustration that everyone feels during IT projects:

•    The business users hate it when we build something they didn’t ask for
•    The developers hate it when they have to re-do things over and over again because the instruction they got weren’t clear or detailed enough
•    The project leadership hates it when the business keep asking for more and more “new features” to be thrown in at the last minute
•    The business management hate wondering what we are up to and hate weeks or months of no news
•    The business sponsors don’t like worrying that they may have wasted money
•    The support team hate having business critical applications thrown at them with little or no documentation and inadequate training

Many of you will have seen the famous “Tire and Tree” cartoon (apparently it has been in circulation since the mid ‘60s!), but it’s well worth including here because it so well exemplifies the dangers of poor communication – which IMHO characterizes all verbal communication:

Often when we work on contract engagements we are asked to use our client’s methodology. In this case, I always look for ideas and suggestions that I can recommend to them to improve their methodology and this advice has always been well received.

Other times the client either has no methodology, or does not practice using one effectively. In these projects it is really important that you try to at least apply a bare bones minimum set of methodology deliverables, not only for them but also for your own protection.

If you can at least minute important design meetings and send confirmation emails of all design changes as you go along, at least you’ll have something to fall back on if all £$%& breaks loose. If I had to choose one deliverable only, it would be the functional or design specification. This document not only encourages discussion about what is going to be developed, but by its very nature it helps to set scope.

If I had to choose one communication device it would be a weekly status update email that copies in everyone who might be wondering what you are up to and why it’s taking so long.

The RMC Delphi Methodology enables project leaders to pick and choose what deliverables and level of status reporting is applicable for each engagement. It’s important to tailor the scope of deliverables and communication for each project.

To aid in a smooth progression through project phases, the Delphi Methodology incorporates exit reviews at the end of each phase. Prior to closing the phase on the project plan, the project manager is required to complete an exit review to ensure all required steps have been taken and that open items are addressed. The review includes development of plans for resolving open issues, as well as a review of project schedule and budget, including managerial approval of changes.

The Delphi Methodology defines standardized responsibilities of logical roles in the data warehouse development for both the Rittman Mead Consulting team and the client team. There are 18 phases within Delphi, which can be fine tuned and/or cut down in scope to fit even the smallest project. The 18 phases are:

1. Project Startup
2. BI requirements definition
3. Data analysis requirements definition
4. Data architecture
5. ETL architecture
6. BI architecture
7. Infrastructure architecture
8. Design data model
9. Design ETL
10. Design BI
11. Design Infrastructure & proof of concept
12. Deployment & Testing Preparation
13. Build ETL
14. Build BI
15. System Test
16. User Acceptance Test
17. Train and deploy
18. Maintenance & Support

It’s important to go through each of the phases, even if vastly reduced in scope, to ensure that you deliver a quality product every time. Whether you are working on a two-year enterprise-wide data warehouse or just adding a new level of aggregation to an existing data mart.

Each phase of the methodology contains a list of deliverables and working documents with links to document templates, such as project plans, statements of work, issues logs, and checklists.

Using templates ensures quality documentation across projects and faster, easier development of documentation based on quality documents from past successful projects. As we complete each project, we try to take the time to update our templates with any improvements we have discovered. Although, I admit this non-funded phase of project work is the most likely to be neglected.

In conclusion, I like to sum up by saying that the key to a successful data warehouse project is a comprehensive methodology that applies best practices and proven experience to guide a data warehouse project team from project launch through to deployment.

I’d like to also say to all the die-hard brilliant developers out there who’d rather walk over hot coals than spend a week doing documentation, try to think of methodology as way to ensure that no one gets frustrated – especially you!

I intend to come back to this over the next few weeks and months and add detail on to the 18 Phases I’ve listed. I’m also keen to get feedback from you as your comments will help me to improve the Delphi Methodology.