Now in 11g, you can add a PREPROCESSOR clause to your external table DDL to point to an executable that will preprocess and pipe your data, allowing the external table to read from the file via the uncompression utility without having to uncompress it all first. It’s one of those things that make you think “why wasn’t that there before?”, as Greg says though it’s now in 11.1.0.7 albeit sketchily documented, so if you’re planning to move large files around as part of your ETL proces and you’re on the latest version of the database, it’ll be worth taking a look at Greg’s article.

This is of course pretty significant, as Oracle have been more or less the only major OLAP vendor to not support MDX as a query language, at least in their in-database OLAP product (Essbase of course supports MDX for Aggregate Storage Option cubes). The rumour about Oracle extending MDX support to Oracle OLAP has been circulating the various conferences for the past year or so, I’d heard that it might be through a third-party spreadsheet add-in rather than direct support in the database and obviously this is it.

Of course this falls short of direct support for MDX, I guess with Oracle’s clear backing of SQL as their preferred OLAP query language that would have been a step too far, but this does give customers who prefer Excel as their OLAP query tool (i.e. most of them) and want to create calculations and selections in MDX the ability to choose Oracle OLAP, with all it’s clear security, scalability and managability benefits, rather than having to use a separate OLAP server. I don’t know what pricing is like, and how well the bridge performs, but it’s certainly interesting and gives Oracle OLAP 11g customers a means to analyze their cubes in a proper multi-dimensional environment, at least until Answers+ arrives.

More details at the Vlamis blog and on the Simba Technologies website.

The basic premise with the BI Management Pack is that it comes with Oracle Grid Control 10.2.0.4 and allows you to manage your BI infrastructure as well. Price-wise I think it adds about $11k to the standard $260k (list price) for OBIEE and it’s already pre-installed if you patch Grid Control up to 10.2.0.4, from the standard 10.2.0.1 version that’s on OTN. Once you apply the patch you then have to make sure the Usage Tracking and Scheduler tables are installed and configured for OBIEE, there’s an additional JMX agent to get running so that OEM can check all the OBIEE counters and so on, and then you can go through the “discovery” process detailed in the docs to register your OBIEE and DAC components.

This went fairly straightforward for me, the only issue I had was that the discovery process couldn’t initially “log on” to my Windows XP host until I made sure the account I was using had the “log on as a batch job” privilege. Once I did that, I was able to add my OBIEE BI Server, Presentation Server and Scheduler to the Grid Control list of targets, along with the DAC (Data Warehouse Administration Console, used for controlling BI Apps ETL jobs) repository. Once you’ve done this, and you’ve added the various Grid Control, Oracle Management Agent and Oracle Management Service elements to your OBIEE setup, your architectural diagram now looks like this:

So once you’ve set this all up, what do you get?

Once you’ve set everything up, you access the various OBIEE components by initially selecting the host they’re on, and then locating the “targets” in the host’s list, like this:

The little “up arrow” next to the target shows whether, for example, the BI Server is currently running, the Scheduler is up and so on. Notice the DAC Server bit in the middle? That’s reporting on whether the DAC Server is up and running and able to service Execution Plan requests from the DAC Console.

The overview for the BI Server shows the general CPU usage, memory usage and so on of the BI Server process. You can also use this page to get a general overview of how “busy” the BI Server is, either currently in minute or so intervals, or over the past day, seven days or so on.

The Dashboard Reports tab shows you data from the Usage Tracking tables, so that you can see which dashboards (note, not requests/reports) are the most active, which users are making the most use of dashboards and so on.

In my case, the OMMgr and admin users have been making the most use of the system.

You can also see to what degree the cache has been used, the cache hit ratio (!) and so on - note that this is the BI Server cache, not the database buffer cache, before anyone gets too excited…

The Presentation Server pages give you a bit more information about how active the front-end of the application has been, how much activity the charting engine has undertaken and so on.

Here’s the overview page, not much to see here though apart from uptime and load.

Now something I did find interesting was integration with the DAC Server and Repository. The BI Management Pack links in to the DAC Repository and shows you graphs on ETL runs, for example:

Here’s a bit more detail on the specific ETL runs that have happened.

So, it looks quite interesting, particularly if you’re already a user of Grid Control 10g and you’d like to bring your OBIEE targets into the fold. For me, what’s particularly interesting isn’t what’s here at the moment; this is obviously a pretty early version of what’s possible, the data that Grid Control is getting from the various OBIEE servers is pretty basic and nothing really more than what you can get from the Usage Tracking tables and reports combined with operating system CPU, RAM and I/O reports; for me though, the interesting thing is what’s possible going into the future.

If you think about your complete Oracle BI stack - going from storage, through to the database, application server and then the various OBIEE servers, then you can see how a tool like this can help you manage performance and your architecture from top to bottom. If you get the classic request from users - “the system is running slow, can you take a look and find out what’s up?” - well with a system like this, if Grid Control in the future links together the reports that are running, with the application server session that hosted them, with the database query that provided the data, combined with the self-tuning (ASH, AWR etc) bits of the database, plus the diagnostics and so on that are in ASM and the general storage layer, well you can really imagine being able to trace a query through from report through to the underlying server system and understand just exactly where the time is going. I don’t think we’re quite there yet, the reports that the BI Management Pack produces are a bit simple and somewhat disjointed, but if the product management team start adding in the sort of advisors that we get from the database side - perhaps an “Aggregate Storage Advisor” that recommends aggregate tables or even Essbase cubes, then passes the details through to the Aggregate Persistence Wizard to create the required summaries; maybe a cache advisor that recommends caching, maybe even a request performance wizard that recommends changes to the underlying data structures including the addition of indexes or materialized views, well, that would be very interesting.

Of course this is all potentially quite a while off, so my next task is to come up with some sort of approach where we can use what’s currently in the BI Management Pack, along with the various database advisors and application server reports to try and diagnose and improve the performance of queries today. I’m actually writing this up as an article for OTN, so keep an eye out after Christmas and hopefully I’ll come up with some good guidelines and a simple methodology so that you can start using this interesting new addition to Grid Control 10gR4.

In terms of presentations, our schedule is as follows:

• “Extending & Customizing the Oracle BI Apps Data Warehouse” - Mark Rittman, Tuesday 12:25 - 13:10
• “A Better BI Methodology” - Jon Mead, Tuesday 15:15 - 16:00
• “Oracle BI EE Performance Optimization” - Mark Rittman, Wednesday 14:35 - 15:20
• “High Availability in Oracle BI EE” - Jon Mead, Wednesday 15:40 - 16:25
• “Deploying a Dimensional Model on the Oracle Database” - Stewart Bryson, Thursday 13:00 - 13:45
• “Oracle 11g Data Warehousing Masterclass” - Mark Rittman, Friday 09:30 - 11:30

As well as doing the presentations and manning the stand, we’ll also be giving away t-shirts and chilled bottles of an exclusive Rittman Mead beer that Borkur’s sourced from Belgium. Hopefully we’ll see some of you there, if you make it to the exhibition keep a look out for stand 90 and come and say hello.

Jennifer comes to us from a major IT services provider and specialises in Informatica, Warehouse Builder, data warehouse design and project management, whilst Ragnar comes from another big IT services provider and specialises in OWB, OBIEE, Discoverer and database development. Continuing our international jet-setting theme, Ragnar is Dutch, worked in the past with Borkur in Iceland, and was up until recently living and working in Canada, whilst Jen is originally from Canada but has lived and worked in the UK for the past few years, and is now working with Borkur and Ragnar on a project in Brussels. Given the fact that three of the team are over in Belgium, Pete’s in Athens whilst Jon and I are working in London, Ireland and Oslo over the next couple of weeks, working out where to hold our team Christmas party is turning into a tricky decision (On the Eurostar, perhaps, or at Schiphol Airport?)

Anyway welcome, Jen and Ragnar, and we look forward to working with you (and reading your blog posts) over the next few months!

One point to keep in mind about fixing bad data on the source; it is just that, fixing DATA. We are not fixing bad applications or bad processes. Often fixing source applications is just not going to happen, especially with the case of legacy and third-party packaged applications. Likewise process can be hard to fix; if we rely on humans to key information in and the application is incapable of enforcing data rules then data entry problems will occur. For example take a university admissions system; here data entry is highly seasonal, first off a lot of prospective candidates (say about 10x the number who will start as students in the new year) then a reduced number of place offers which get further reduced by the candidates choosing another university or not reaching the entrance grades. This work flow often requires extra temporary staff just to key in details, they are not full-time employees, they do not know the significance of the data, they do not know about the reporting systems that give insight to the data, they are just there to get the data in as quickly as possible and with enough accuracy to let the process as whole work. It really does not matter to the process if the country name is not standardised providing the funding code is correct; even the course code may not matter that much providing the faculty is correct, especially if all faculty members follow a common scheme of study in the first year.
So, fixing data at source is most unlikely to be a one off exercise. True, a one-off data fix will improve the quality of our historic reporting, but what we are not doing is preventing data errors from occurring in the future. We must build into our ETL processes methods to continually monitor source data quality and build processes so that any errors detected can be corrected at source.

Of course bitmap indexes, and indexes in general, are a subject that people like Richard Foote and Jonathan Lewis have covered very well in the past and there are number of good article on the internet that explain how the feature works. In particular three articles by Jonathan, on bitmap indexes, bitmap star transformations and bitmap join indexes, are about the closest I’ve seen to a definitive introduction to the subject, whereas this article by Vivek Sharma compares bitmap indexes to b- tree indexes in the context of decision support with some useful examples of test cases to illustrate the differences. There’s also a good article on Wikipedia that sets out a generic background to bitmap indexes, which of course are used across the database industry and aren’t just an Oracle feature. Looking through these articles was certainly useful in filling in some of the gaps in my knowledge, and in particular confirmed the basic understanding that I had around the subject.

Bitmap indexes differ from the more common b-tree indexes in that they use bit arrays (also known as “bitmaps”) to indicate whether a particular row in a table contains a particular column value. To take an example, you might have a table of customer details, and you have a column called “hair colour”. A bitmap index on this column would logically store the index data in the following way:

So what you have there is a string of ones and zeros (the “bit array” mentioned above”) that record whether a particular row has that particular column value. Now the clever bit comes when you want to filter based on more than one bitmap-indexed value; in this case, the bit arrays are logically ANDed, OR’d or XOR’d so that you can quickly find out, for example, which customers have red hair, big feet, snore and have bad breath (so that presumably you can avoid sending them an invite to next week’s customer reception):

So if that’s how they logically work, the next question I had was “how are these details stored internally?” Jonathan’s first article, on bitmap indexes, does a good job on explaining this (it’s written for Oracle 9i so I’m not sure if the details have changed) and includes a symbolic dump that shows bitmap index leaf blocks being made up of five components:

1) A set of flags
2) A lock byte
3) The indexed value (”Red Hair”, for example, or “Big Feet”)
4) A pair of row IDs, and
5) A stream of bits

The lock byte is used to determine whether this particular part of the index is locked, because of some DML happening to the underlying table. The pair of row IDs designates the portion of the table that this index leaf block covers (back in the 9i days, Jonathan reckons this range can cover up to 24,000 rows, this may have changed with more recent releases), whilst the stream of bits tells Oracle whether a particular row ID contains the indexed value in question. Presumably then some sort of compression algorithm is applied and, depending on the number of distinct values that are being indexed and the distribution of the data, the index is then stored taking up potentially far less space than a regular b- tree index.

So the general thinking here is that bitmap indexes are particularly suited to decision support applications as they

(a) can be easily combined using bit-wise operations to filter data based on multiple criteria
(b) their high compression ratios mean that they can take up less space than comparable b- tree indexes
(c) they can be quick to create, meaning you can create more of them in a given ETL window or your ETL process can take less time, and
(d) if you are using Oracle, you can use additional features such as bitmap star transformations to rapidly return data from your data warehouse.

Now there are a lot of claims, myths and ideas out there about bitmap indexes which are partly explained by the fact that it’s often it’s difficult to test them under the right conditions (multiple concurrent users, for example, or with big enough data sets) and the characteristics of the feature have changed over time (the effect of multiple updates onto bitmap-indexed columns has become less pronounced since the advent of 10g, with these two blog postings (here, and here) being good examples of where people can end up drawing the wrong conclusions.

That said though, looking back at the session I ran, in hindsight what it lacked were some basic examples of bitmap index creation, with some tests afterwards to compare their build time, size and query performance compared to regular b-tree indexes. I’m also curious to see what extra benefit is gained by using bitmap join indexes both in comparison to regular bitmap indexes and compared to what seems to me to be a similar thing - creating join-only materialized views (though I could be wrong here). Given this, I took the SH Sales History data set and started to knock up some examples. Now bear with me here as this is still only a very limited example, I’m also happy for anyone who knows more about this (Pete? Richard? Stuart?) to chip in and point out if I’ve drawn the wrong conclusion.

To start off then, I’m going to take two copies of the SH.CUSTOMERS table, and load the table data back into the tables a couple of times to increase the table row counts, otherwise most of the tests I do will return data too quickly. The version I’m using of Oracle is 11.1.0.6 on Windows XP SP2, running on a VMWare Fusion virtual machine with 1.25GB of RAM and 1 processor enabled.

SQL> select count(*) from customers_copy_btree;

COUNT(*)
----------
1776000                                                                                                                                                                                          

SQL> select count(*) from customers_copy_bitmap;

COUNT(*)
----------
1776000

OK, so both CUSTOMER table copies have now got around 1.7M rows within them. The SH schema from which they are sourced is a star schema, so the CUSTOMERS table has lots of “attribute” columns that I can index and then filter on.

SQL> select count(distinct cust_gender) from customers_copy_bitmap;

COUNT(DISTINCTCUST_GENDER)
--------------------------
2                                                                                                                                                                          

SQL> select count(distinct cust_year_of_birth) from customers_copy_bitmap;

COUNT(DISTINCTCUST_YEAR_OF_BIRTH)
---------------------------------
75                                                                                                                                                                   

SQL> select count(distinct cust_last_name) from customers_copy_bitmap;

COUNT(DISTINCTCUST_LAST_NAME)
-----------------------------
908                                                                                                                                                                       

SQL> select count(distinct cust_street_address) from customers_copy_bitmap;

COUNT(DISTINCTCUST_STREET_ADDRESS)
----------------------------------
50945

So looking at those distinct row counts, the table’s attributes range from 2 distinct values (gender) through 75 (year of birth) through to 50,945 for street addresses. That should give us a range of different column cardinalities to test.

SQL> set timing on

SQL> create index cus_gender_btree_idx on customers_copy_btree(cust_gender);

Index created.

Elapsed: 00:00:34.01

SQL> create index cus_yob_btree_idx on customers_copy_btree(cust_year_of_birth);

Index created.

Elapsed: 00:00:37.12

SQL> create index cus_lname_btree_idx on customers_copy_btree(cust_last_name);

Index created.

Elapsed: 00:00:29.25

SQL> create index cus_adr_btree_idx on customers_copy_btree(cust_street_address);

Index created.

Elapsed: 00:00:46.43

Creating b-tree indexes on a selection of columns comes in at a total of 34+37+29+46 = 2 mins 26 seconds. How long to comparable bitmap indexes take to create?

SQL> create bitmap index cus_gender_bix on customers_copy_bitmap(cust_gender);

Index created.

Elapsed: 00:00:36.11

SQL> create bitmap index cus_yob_bix on customers_copy_bitmap(cust_year_of_birth);

Index created.

Elapsed: 00:00:10.57

SQL> create bitmap index cus_lname_bix on customers_copy_bitmap(cust_last_name);

Index created.

Elapsed: 00:00:13.92

SQL> create bitmap index cus_adr_bix on customers_copy_bitmap(cust_street_address);

Index created.

Elapsed: 00:00:10.87

So to create the bitmap indexes, the total time was 36+10+14+10 = 1 min 10 seconds, about half the time of the b-tree indexes. I then gather statistics on the tables and indexes and then run a query to determine the table and index sizes.

SQL> select segment_name
2  ,	    bytes/1024/1024 "Size in MB"
3  from   user_segments
4  where  segment_name like 'CUSTOMERS_COPY%';

SEGMENT_NAME                             Size in MB
---------------------------------------- ----------
CUSTOMERS_COPY_BITMAP                           368
CUSTOMERS_COPY_BTREE                            368                                                                                                                                                 

SQL> select segment_name
2  ,	    bytes/1024/1024 "Size in MB"
3  from   user_segments
4  where  segment_name like '%_IDX';

SEGMENT_NAME                             Size in MB
---------------------------------------- ----------
CUS_ADR_BTREE_IDX                                72
CUS_GENDER_BTREE_IDX                             26
CUS_LNAME_BTREE_IDX                              36
CUS_YOB_BTREE_IDX                                30                                                                                                                                                 

SQL> select segment_name
2  ,	    bytes/1024/1024 "Size in MB"
3  from   user_segments
4  where  segment_name like '%_BIX';

SEGMENT_NAME                             Size in MB
---------------------------------------- ----------
CUS_ADR_BIX                                      11
CUS_GENDER_BIX                                 .625
CUS_LNAME_BIX                                     3
CUS_YOB_BIX                                       5

The tables we’re indexing are 368MB in size, and the b-tree indexes range from 72MB for the high-cardinality address index, through to 30MB for the low-cardinality gender index. The bitmap indexes, though, are around six times smaller, going from 11MB for the address index through to just over half a MB for the gender index. So they’re certainly smaller, at least for this dataset’s data distribution, and as we saw before, they take less time to prepare.

How about some queries then, how do the explain plans look for simple queries using these columns as filters? Let’s try filtering on year of birth, which had 75 different values in our test data.

SQL> set lines 200
SQL> set autotrace traceonly
SQL> select *
  2  from   customers_copy_btree
  3  where  cust_year_of_birth = 1967;

30592 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 2533345702
------------------------------------------------------------------------------------------
| Id  | Operation         | Name                 | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |                      | 30592 |  5526K| 12728   (1)| 00:02:33 |
|*  1 |  TABLE ACCESS FULL| CUSTOMERS_COPY_BTREE | 30592 |  5526K| 12728   (1)| 00:02:33 |
------------------------------------------------------------------------------------------                                                                                                              

Predicate Information (identified by operation id):
---------------------------------------------------                                                                                                                                                     

   1 - filter("CUST_YEAR_OF_BIRTH"=1967)                                                                                                                                                                

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
      48332  consistent gets
      46337  physical reads
          0  redo size
    4711189  bytes sent via SQL*Net to client
      22845  bytes received via SQL*Net from client
       2041  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      30592  rows processed                                                                                                                                                                             

SQL> select *
  2  from   customers_copy_bitmap
  3  where  cust_year_of_birth = 1967;

30592 rows selected.

Execution Plan
----------------------------------------------------------
Plan hash value: 3003431057                                                                                                                                                                             

------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name                  | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                       | 30592 |  5526K|  5652   (1)| 00:01:08 |
|   1 |  TABLE ACCESS BY INDEX ROWID | CUSTOMERS_COPY_BITMAP | 30592 |  5526K|  5652   (1)| 00:01:08 |
|   2 |   BITMAP CONVERSION TO ROWIDS|                       |       |       |            |          |
|*  3 |    BITMAP INDEX SINGLE VALUE | CUS_YOB_BIX           |       |       |            |          |
------------------------------------------------------------------------------------------------------                                                                                                  

Predicate Information (identified by operation id):
---------------------------------------------------                                                                                                                                                     

   3 - access("CUST_YEAR_OF_BIRTH"=1967)                                                                                                                                                                

Statistics
----------------------------------------------------------
        170  recursive calls
          0  db block gets
      24427  consistent gets
      16164  physical reads
          0  redo size
    5975063  bytes sent via SQL*Net to client
      22845  bytes received via SQL*Net from client
       2041  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      30592  rows processed

So in that example, Oracle actually used a full table scan for the b-tree indexed table, whilst the bitmap indexed one used an index lookup. The cost of the execution plan against the bitmap indexed table was the lowest out of the two. Now what about if we filter on several columns, this is where bitmap indexes surely come into their own?

SQL> select *
  2  from   customers_copy_bitmap
  3  where  cust_gender = 'M'
  4  and    cust_year_of_birth = 1980
  5  and    cust_last_name = 'SMITH';

no rows selected

Execution Plan
----------------------------------------------------------
Plan hash value: 2767299321                                                                                                                                                                             

------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name                  | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                       |    21 |  3885 |     8   (0)| 00:00:01 |
|   1 |  TABLE ACCESS BY INDEX ROWID | CUSTOMERS_COPY_BITMAP |    21 |  3885 |     8   (0)| 00:00:01 |
|   2 |   BITMAP CONVERSION TO ROWIDS|                       |       |       |            |          |
|   3 |    BITMAP AND                |                       |       |       |            |          |
|*  4 |     BITMAP INDEX SINGLE VALUE| CUS_LNAME_BIX         |       |       |            |          |
|*  5 |     BITMAP INDEX SINGLE VALUE| CUS_YOB_BIX           |       |       |            |          |
|*  6 |     BITMAP INDEX SINGLE VALUE| CUS_GENDER_BIX        |       |       |            |          |
------------------------------------------------------------------------------------------------------                                                                                                  

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access("CUST_LAST_NAME"='SMITH')
   5 - access("CUST_YEAR_OF_BIRTH"=1980)
   6 - access("CUST_GENDER"='M')                                                                                                                                                                        

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          2  consistent gets
          2  physical reads
          0  redo size
       1701  bytes sent via SQL*Net to client
        405  bytes received via SQL*Net from client
          1  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          0  rows processed                                                                                                                                                                             

SQL> select *
  2  from   customers_copy_btree
  3  where  cust_gender = 'M'
  4  and    cust_year_of_birth = 1980
  5  and    cust_last_name = 'SMITH';

no rows selected

Execution Plan

----------------------------------------------------------
Plan hash value: 2580896903                                                                                                                                                                             

---------------------------------------------------------------------------------------------------------
| Id  | Operation                        | Name                 | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                 |                      |    21 |  3885 |    77   (2)| 00:00:01 |
|*  1 |  TABLE ACCESS BY INDEX ROWID     | CUSTOMERS_COPY_BTREE |    21 |  3885 |    77   (2)| 00:00:01 |
|   2 |   BITMAP CONVERSION TO ROWIDS    |                      |       |       |            |          |
|   3 |    BITMAP AND                    |                      |       |       |            |          |
|   4 |     BITMAP CONVERSION FROM ROWIDS|                      |       |       |            |          |
|*  5 |      INDEX RANGE SCAN            | CUS_LNAME_BTREE_IDX  |  1956 |       |     7   (0)| 00:00:01 |
|   6 |     BITMAP CONVERSION FROM ROWIDS|                      |       |       |            |          |
|*  7 |      INDEX RANGE SCAN            | CUS_YOB_BTREE_IDX    |  1956 |       |    62   (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------------                                                                                               

Predicate Information (identified by operation id):
---------------------------------------------------
   1 - filter("CUST_GENDER"='M')
   5 - access("CUST_LAST_NAME"='SMITH')
   7 - access("CUST_YEAR_OF_BIRTH"=1980)                                                                                                                                                                

Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
          3  consistent gets
          2  physical reads
          0  redo size
       1701  bytes sent via SQL*Net to client
        405  bytes received via SQL*Net from client
          1  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          0  rows processed

Interestingly, Oracle has converted the b-tree indexes into bitmap ones, when querying the b-tree indexed table, so that it can do a bitmap AND on the filter results, which I presume brings the performance of data warehouse b-tree queries closer to “native” bitmap ones (this, I think, has been a feature of the database since the 10g version at least). Even so, the cost of the bitmap query appears to be a lot less than the b-tree index one, and so far, the results are as you’d expect.

So what about the famous issue with bitmap indexes, the one where updates to the column that’s indexed cause the index to grow in size, because the index loses compression? The fact this happens seems to be received wisdom, especially when updates are in the form of lots of little updates rather than a small amount of large updates, however people have mentioned that in recent versions, the effect is less pronounced. The other commonly held issue with updates against bitmap indexes is that they are slow, in comparison to tables with regular b-tree indexes, but this seems to be more of an issue when the updates are applied concurrently, due to portions of the indexed table (or perhaps portions of the indexed column) getting locked whilst the associated bitmap index is reconfigured. The concurrency test is tricky to perform when you’ve just got a single-user laptop, but what about the size issue, what if I applied a couple of hundred single-row updates to a bitmap indexed table and column using Oracle 11.1.0.6, would this increase the size of the index now?

SQL> create index test_cust_id_1_idx on customers_copy_btree (cust_id);

Index created.

SQL> create index test_cust_id_2_idx on customers_copy_bitmap (cust_id);

Index created.

SQL> set timing on

SQL> declare
  2  upd_cust_id number(5);
  3  cust_yob_value number(4);
  4  begin
  5    for i in 1 .. 500 loop
  6  	 upd_cust_id := dbms_random.value(1,55000);
  7  	 cust_yob_value := dbms_random.value(1900,2000);
  8  	 update customers_copy_btree
  9  	   set cust_year_of_birth = cust_yob_value
 10  	   where cust_id = upd_cust_id;
 11  	 commit;
 12    end loop;
 13  end;
 14  /

PL/SQL procedure successfully completed.

Elapsed: 00:03:15.67

SQL> declare
  2  upd_cust_id number(5);
  3  cust_yob_value number(4);
  4  begin
  5    for i in 1 .. 500 loop
  6  	 upd_cust_id := dbms_random.value(1,55000);
  7  	 cust_yob_value := dbms_random.value(1900,2000);
  8  	 update customers_copy_bitmap
  9  	   set cust_year_of_birth = cust_yob_value
 10  	   where cust_id = upd_cust_id;
 11  	 commit;
 12    end loop;
 13  end;
 14  /

PL/SQL procedure successfully completed.

Elapsed: 00:05:46.21

SQL> set timing off

SQL> select segment_name
  2  ,	    bytes/1024/1024 "Size in MB"
  3  from   user_segments
  4  where  segment_name in ('CUS_YOB_BTREE_IDX','CUS_YOB_BIX');

SEGMENT_NAME                             Size in MB
---------------------------------------- ----------
CUS_YOB_BIX                                       5
CUS_YOB_BTREE_IDX                                30

So the update to the bitmap indexed column took a fair bit longer than the b-tree indexed column, although as I said this should really be tested with lots of concurrent updates as this is where, I believe, the effect is most pronounced. When it comes to the size of the indexes though, they’re more or less the same as before the updates took place, which seems to indicate that in this release, bitmap indexes don’t grow is size so much when updates are applied. YMMV though.

Ok, so how about bitmap join indexes though? If we’ve got a fact table that we want to join to the dimension table, how much better performance do we get if we create bitmap join indexes on the fact table that join to the dimension key and dimension attribute values, compared to just standard bitmap indexes? To test this I took a copy of the SH.SALES and SH.CUSTOMER tables and created some example indexes.

SQL> select count(*) from customers_bijx_test_bitmap;

  COUNT(*)
----------
     55500                                                                                                                                                                                              

SQL> select count(*) from sales_bijx_test_bitmap;

  COUNT(*)
----------
  14701488                 

SQL> alter table customers_bijx_test_bitmap add constraint cust_bijx_test_bitmap_pk primary key (cust_id);

Table altered.

SQL> select count(*) from customers_bijx_test_bitjoin;

  COUNT(*)
----------
     55500                                                                                                                                                                                              

SQL> select count(*) from sales_bijx_test_bitjoin;

  COUNT(*)
----------
  14701488             

SQL> alter table customers_bijx_test_bitjoin add constraint cust_bijx_test_bitjoin_pk primary key (cust_id);

SQL> set timing on

SQL> create bitmap index sales_bijx_test_bitmap_bix1 on sales_bijx_test_bitmap(cust_id);

Index created.

Elapsed: 00:00:33.43

SQL> create bitmap index cust_bijx_test_bitmap_bix1 on customers_bijx_test_bitmap(cust_last_name);

Index created.

Elapsed: 00:00:00.73

SQL> create bitmap index sales_bijx_test_bitjoin_bjx1 on sales_bijx_test_bitjoin(customers_bijx_test_bitjoin.cust_id)
  2  from sales_bijx_test_bitjoin, customers_bijx_test_bitjoin
  3  where sales_bijx_test_bitjoin.cust_id = customers_bijx_test_bitjoin.cust_id;

Index created.

Elapsed: 00:02:38.64

SQL> create bitmap index sales_bijx_test_bitjoin_bjx2 on sales_bijx_test_bitjoin(customers_bijx_test_bitjoin.cust_last_name)
  2  from sales_bijx_test_bitjoin, customers_bijx_test_bitjoin
  3  where sales_bijx_test_bitjoin.cust_id = customers_bijx_test_bitjoin.cust_id;

Index created.

Elapsed: 00:02:46.70

SQL> set timing off

That’s the bitmap index and bitmap join indexes created then, with the bitmap join indexes obviously taking a lot more time to create, as the have to perform the join between the fact and dimension tables. As far as I can see I’ve now got the same sets of columns indexed, so I then gather statistics for the tables and their indexes (”analyze table … compute statistics for table for all indexes for all indexed columns”) and check the sizes of the indexes.

SQL> col segment_name for a30

SQL> select segment_name
  2  ,	    bytes/1024/1024 "Size in MB"
  3  from   user_segments
  4  where  segment_name in ('SALES_BIJX_TEST_BITMAP_BIX1','CUST_BIJX_TEST_BITMAP_BIX1',
  5  'SALES_BIJX_TEST_BITJOIN_BJX1','SALES_BIJX_TEST_BITJOIN_BJX2');

SEGMENT_NAME                   Size in MB
------------------------------ ----------
CUST_BIJX_TEST_BITMAP_BIX1           .125
SALES_BIJX_TEST_BITJOIN_BJX1           48
SALES_BIJX_TEST_BITJOIN_BJX2           37
SALES_BIJX_TEST_BITMAP_BIX1            48 

So the two indexes (SALES_BIJX_TEST_BITJOIN_BJX1 and SALES_BIJX_TEST_BITMAP_BIX1) that index the CUST_ID column on the fact able are the same size, even though one of them includes the join through to the dimension table. The bitmap index just on the customer name column on the dimension table is very small, whereas the bitmap join index on the CUST_ID fact table column joined to the customer name dimension column is a bit smaller than the ones that just index the CUST_ID column. How do things look when we run some queries than, and generate some execution plans?

SQL> set lines 200

SQL> set autotrace traceonly

SQL> select sum(s.amount_sold)
  2  from   sales_bijx_test_bitmap s
  3  ,	    customers_bijx_test_bitmap c
  4  where  s.cust_id = c.cust_id
  5  and    c.cust_last_name = 'SMITH';

Execution Plan
----------------------------------------------------------
Plan hash value: 3492243084                                                                                                                                                                             

-------------------------------------------------------------------------------------------------------------
| Id  | Operation                      | Name                       | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT               |                            |     1 |    28 | 19594   (2)| 00:03:56 |
|   1 |  SORT AGGREGATE                |                            |     1 |    28 |            |          |
|*  2 |   HASH JOIN                    |                            |   127K|  3480K| 19594   (2)| 00:03:56 |
|   3 |    TABLE ACCESS BY INDEX ROWID | CUSTOMERS_BIJX_TEST_BITMAP |    61 |   671 |    15   (0)| 00:00:01 |
|   4 |     BITMAP CONVERSION TO ROWIDS|                            |       |       |            |          |
|*  5 |      BITMAP INDEX SINGLE VALUE | CUST_BIJX_TEST_BITMAP_BIX1 |       |       |            |          |
|   6 |    TABLE ACCESS FULL           | SALES_BIJX_TEST_BITMAP     |    14M|   238M| 19512   (2)| 00:03:55 |
-------------------------------------------------------------------------------------------------------------                                                                                           

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("S"."CUST_ID"="C"."CUST_ID")
   5 - access("C"."CUST_LAST_NAME"='SMITH')                                                                                                                                                             

Statistics
----------------------------------------------------------
         24  recursive calls
          0  db block gets
          4  consistent gets
          2  physical reads
          0  redo size
        426  bytes sent via SQL*Net to client
        416  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed                                                                                                                                                                             

SQL> select sum(s.amount_sold)
  2  from   sales_bijx_test_bitjoin s
  3  ,	    customers_bijx_test_bitjoin c
  4  where  s.cust_id = c.cust_id
  5  and    c.cust_last_name = 'SMITH';

Execution Plan
----------------------------------------------------------
Plan hash value: 4184703125                                                                                                                                                                             

--------------------------------------------------------------------------------------------------------------
| Id  | Operation                     | Name                         | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |                              |     1 |    26 |  3352   (1)| 00:00:41 |
|   1 |  SORT AGGREGATE               |                              |     1 |    26 |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID | SALES_BIJX_TEST_BITJOIN      | 18240 |   463K|  3352   (1)| 00:00:41 |
|   3 |    BITMAP CONVERSION TO ROWIDS|                              |       |       |            |          |
|*  4 |     BITMAP INDEX SINGLE VALUE | SALES_BIJX_TEST_BITJOIN_BJX2 |       |       |            |          |
--------------------------------------------------------------------------------------------------------------                                                                                          

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access("S"."SYS_NC00009$"='SMITH')                                                                                                                                                               

Statistics
----------------------------------------------------------
         24  recursive calls
          0  db block gets
          5  consistent gets
          2  physical reads
          0  redo size
        426  bytes sent via SQL*Net to client
        416  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed                                                                                                                                                                             

SQL> set autotrace off

Looking at the execution plan for the table with just a regular bitmap index, you can see both tables being accessed, though you can also see a full table scan on the sales table, which isn’t what I’d expected as the CUST_ID column is indexed. The execution plan for the query against the table with a bitmap join index clearly shows just the one table (the sales table) being accessed, as Oracle can get the dimension information it needs from the bitmap join index. So I’m not sure about the first test, but the second shows the bitmap join index working and the dimension table not needing to be accessed on this occaision. Again, I’d like to test how much extra time updates take to a table with bitmap join indexes on it, but again I think the effect will be most noticeable with lots of concurrent updates and so a simple single-user test won’t really show this properly, suffice to say, my impression is that if bitmap indexes cause contention issues with lots of concurrent updates, bitmap join indexes will be even worse.

Now all this thinking about “pre-joining” tables through the use of bitmap join indexes got me thinking, “this sounds a lot like the pre-joining you can do through join-only materialized views”. Join-only materialized views differ from your common or garden materialized views in that they don’t aggregate or calculate any data, they just store the join in advance so that subsequent queries don’t need to incur the join cost on every query execution. Surely this sounds a lot like bitmap join indexes? Let’s create one and compare it to the bitmap join index we created earlier, against another copy of the SH.SALES and SH.CUSTOMERS tables.

SQL> create materialized view log on sales_mvjoin_test with rowid;

Materialized view log created.

SQL> create materialized view log on cust_mvjoin_test with rowid;

Materialized view log created.

SQL> create materialized view sales_cust_join_mv
2  build immediate
3  refresh fast
4  enable query rewrite as
5  select s.rowid sales_rid, c.rowid cust_rid,
6  	    s.amount_sold, c.cust_id c_cust_id, s.cust_id s_cust_id, c.cust_last_name
7  from   sales_mvjoin_test s, cust_mvjoin_test c
8  where  s.cust_id = c.cust_id;

Materialized view created.

SQL> create bitmap index sales_cust_join_mv_bix1 on sales_cust_join_mv(cust_rid);

Index created.

SQL> create bitmap index sales_cust_join_mv_bix2 on sales_cust_join_mv(sales_rid);

Index created.

SQL> create bitmap index sales_cust_join_mv_bix3 on sales_cust_join_mv(s_cust_id);

Index created.

SQL> create bitmap index sales_cust_join_mv_bix4 on sales_cust_join_mv(c_cust_id);

Index created.

SQL> create bitmap index sales_cust_join_mv_bix5 on sales_cust_join_mv(cust_last_name);

Index created.

SQL> analyze table sales_cust_join_mv
2  compute statistics for table for all indexes
3  for all indexed columns;

Table analyzed.

SQL> alter session set query_rewrite_enabled=true;

Session altered.

Notice that I had to create materialized view logs on the underlying tables so that my materialized view would “fast refresh” (this is the closest comparison to how indexes behave), and I had to subsequently index and then analyze the materialized view, which obviously adds to the preparation time and space taken up, making the creation of the materialized view a fair bit more involved than the creation of the bitmap join index. So how do queries against the bitmap-indexed table, bitmap join indexed-table and tables with a materialized view containing joins compare?

SQL> set lines 200

SQL> set autotrace traceonly

SQL> 

SQL> select sum(s.amount_sold)
  2  from   sales_bijx_test_bitmap s
  3  ,	    customers_bijx_test_bitmap c
  4  where  s.cust_id = c.cust_id
  5  and    c.cust_last_name = 'SMITH';

Execution Plan
----------------------------------------------------------
Plan hash value: 3492243084                                                                                                                                                                             

-------------------------------------------------------------------------------------------------------------
| Id  | Operation                      | Name                       | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT               |                            |     1 |    28 | 19594   (2)| 00:03:56 |
|   1 |  SORT AGGREGATE                |                            |     1 |    28 |            |          |
|*  2 |   HASH JOIN                    |                            |   127K|  3480K| 19594   (2)| 00:03:56 |
|   3 |    TABLE ACCESS BY INDEX ROWID | CUSTOMERS_BIJX_TEST_BITMAP |    61 |   671 |    15   (0)| 00:00:01 |
|   4 |     BITMAP CONVERSION TO ROWIDS|                            |       |       |            |          |
|*  5 |      BITMAP INDEX SINGLE VALUE | CUST_BIJX_TEST_BITMAP_BIX1 |       |       |            |          |
|   6 |    TABLE ACCESS FULL           | SALES_BIJX_TEST_BITMAP     |    14M|   238M| 19512   (2)| 00:03:55 |
-------------------------------------------------------------------------------------------------------------                                                                                           

Predicate Information (identified by operation id):
---------------------------------------------------
   2 - access("S"."CUST_ID"="C"."CUST_ID")
   5 - access("C"."CUST_LAST_NAME"='SMITH')                                                                                                                                                             

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          2  consistent gets
          2  physical reads
          0  redo size
        426  bytes sent via SQL*Net to client
        416  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed                                                                                                                                                                             

SQL> select sum(s.amount_sold)
  2  from   sales_bijx_test_bitjoin s
  3  ,	    customers_bijx_test_bitjoin c
  4  where  s.cust_id = c.cust_id
  5  and    c.cust_last_name = 'SMITH';

Execution Plan
----------------------------------------------------------
Plan hash value: 4184703125                                                                                                                                                                             

--------------------------------------------------------------------------------------------------------------
| Id  | Operation                     | Name                         | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT              |                              |     1 |    26 |  3352   (1)| 00:00:41 |
|   1 |  SORT AGGREGATE               |                              |     1 |    26 |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID | SALES_BIJX_TEST_BITJOIN      | 18240 |   463K|  3352   (1)| 00:00:41 |
|   3 |    BITMAP CONVERSION TO ROWIDS|                              |       |       |            |          |
|*  4 |     BITMAP INDEX SINGLE VALUE | SALES_BIJX_TEST_BITJOIN_BJX2 |       |       |            |          |
--------------------------------------------------------------------------------------------------------------                                                                                          

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access("S"."SYS_NC00009$"='SMITH')                                                                                                                                                               

Statistics
----------------------------------------------------------
          0  recursive calls
          0  db block gets
          3  consistent gets
          2  physical reads
          0  redo size
        426  bytes sent via SQL*Net to client
        416  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed                                                                                                                                                                             

SQL> select sum(s.amount_sold)
  2  from   sales_mvjoin_test s
  3  ,	    cust_mvjoin_test c
  4  where  s.cust_id = c.cust_id
  5  and    c.cust_last_name = 'SMITH';

Execution Plan
----------------------------------------------------------
Plan hash value: 1046520199                                                                                                                                                                             

-------------------------------------------------------------------------------------------------------------------
| Id  | Operation                               | Name                    | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                        |                         |     1 |    20 |  3497   (1)| 00:00:42 |
|   1 |  SORT AGGREGATE                         |                         |     1 |    20 |            |          |
|   2 |   MAT_VIEW REWRITE ACCESS BY INDEX ROWID| SALES_CUST_JOIN_MV      | 18240 |   356K|  3497   (1)| 00:00:42 |
|   3 |    BITMAP CONVERSION TO ROWIDS          |                         |       |       |            |          |
|*  4 |     BITMAP INDEX SINGLE VALUE           | SALES_CUST_JOIN_MV_BIX5 |       |       |            |          |
-------------------------------------------------------------------------------------------------------------------                                                                                     

Predicate Information (identified by operation id):
---------------------------------------------------
   4 - access("SALES_CUST_JOIN_MV"."CUST_LAST_NAME"='SMITH')                                                                                                                                            

Statistics
----------------------------------------------------------
         81  recursive calls
          0  db block gets
        183  consistent gets
       2865  physical reads
          0  redo size
        426  bytes sent via SQL*Net to client
        416  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          2  sorts (memory)
          0  sorts (disk)
          1  rows processed                                                                                                                                                                             

SQL> set autotrace off

So there’s not a lot in it between the costs in the execution plans for the bitmap join indexed-table and the table with an associated materialized view containing joins only, but the consistent gets and physical reads when using the materialized view are far higher than the bitmap join index approach. Given that the materialized view took a fair bit more work to set up than the bitmap join index based on these figures I’d stick with the bitmap join index, but two things I know I’ve ignored are (a) I could just as easily make the materialized view contain the aggregate of sales rather than just detail, you can’t do this with indexes and (b) again one thing I can’t really test here is how well the both of these solutions stands up to concurrent inserts, updates and deletes. So far to me there’s not a lot in it, except for bitmap join indexes appear to be more efficient at least for querying comparing apples to apples, but I suspect there’s a bit more to it here - anyone else have any thoughts?

Anyway, to conclude. With the big caveats - “your mileage may vary” and “these were only simple tests, on a single user one-CPU system”, here’s my thoughts using 11.1.0.6

• As is generally accepted, bitmap indexes can be a good solution for DSS-style applications that filter data based on multiple columns
• In recent versions of Oracle though (10g+ I think) Oracle will however often convert b-tree indexes to bitmap ones on the fly, bringing b-tree indexes nearer to “native’ bitmap index performance
• Updates to bitmap indexes are still slow, compared to b-tree indexes, but the size increase issue is perhaps less pronounced now with recent (10g+ versions of Oracle)
• The main issue around bitmap index updates though is where updates are concurrent, and you can only really test this with a realistic workload. However this isn’t really the situation where you’d want to use bitmap indexes anyway (OLTP applications)
• Bitmap join indexes can give you even more of a DSS-query performance boost, by removing the need at query time to join to the dimension table. However my suspicion is that they are even more costly to update, especially when updates are highly concurrent
• Materialized Views using joins only are an alternative to bitmap join indexes, but probably have little additional value in their non-aggregated form, and they take a lot more work to set up. If you’re considering using materialized views, you might as well go the whole hog and include the aggregates in them as well.

Any thoughts from anyone on this?

Looking at the agenda, you’ve got sessions from Bob Stackowiak (who shared the stage with me at the recent PHLOUG event), Tim Gorman, Chris Claterbos, Ian Abramson (IOUG Chair), Bud Endress and Marty Gubar (Oracle OLAP), Joe Leva, Edward Roske (Essbase Oracle ACE Director, entering the lions den), Charlie Berger, Scott Rappoport and many other US-based and international speakers. The BIWA event is about as close as you can get to a worldwide Oracle BI&DW conference, and I’m pleased to see that ODTUG are taking part as one of the official sponsors. If you’re interested, I’d definately recommend the event and it’s a shame I can’t be there. Good luck to Shyam, Mark, Charlie and the other organizers and I hope it’ll be a great event.

ODTUG Kaleidoscope is different to all the other conferences in that it’s practical, technical, focused on developers and has a very strong BI and Essbase slant. Together with Kent Graziano I chair the ODTUG BI&DW SIG, and we’re keen to get papers this year that go beyond the basics and really get under the hood of tools like Oracle Warehouse Builder, Oracle BI Enterprise Edition, Oracle Discoverer and Oracle Essbase. The event runs over five days and has a relatively small, select list of attendees (compared to events like Collaborate or OOW) meaning that you get to meet everyone, the catering is superb and we can focus just on the needs of developers.

If you’re interested in speaking, submit an abstract, and if you’re thinking of attending and would like to suggest a topic or speaker, take a look at the Kaleidoscope Forum where you can make suggestions or discuss the conference. Whichever way, make a note in your diary and hopefully we’ll see you in Monterey next year!

In the interim, the customer can work on data quality; they know what they need to report on (it’s in the project mandate!) and they know the sources of information (their operational systems) so they can start to verify that all of the required facts can be found in the source systems and more importantly look at the data content and assess “fitness for purpose”. If data defects are found then it may be possible to get them fixed before the serious construction of the ETL layer starts. Besides, the knowledge of source and target gives a good head start in the specification of ETL interfaces.

One particular issue they might meet, and one that is sadly far too common across many business sectors, is the use of operational systems that do not enforce data integrity. For whatever reasons there is just too much freedom in data entry and although it may not affect the operational system much it really can cause problems when you try to aggregate information on the BI system.

But how do we deal with this? Recently I joined in on a thread on one of LinkedIn BI groups where it was proposed that a “receive garbage, store garbage strategy was adopted” - in my opinion this might be OK for a mature BI system where users can understand that the reporting accurately reflects the source, but for a new venture into BI? To me, this seems to be too much a risk; it might be that the new BI users do not have sufficient exposure to the source systems to realise that the data is at fault on the source. We could prevent data that fails a quality threshold from loading on the BI system, but then we would show incomplete results which although correctly aggregated are misleading because of omission; at the end of the day load policy is a business choice. If we go with the “reject poor data” route we should seriously  think about providing a data quality dashboard on the reporting system to indicate the numbers of records that failed to be loaded and drill-down to the reasons why they failed.

So what do we do with data that fails the quality standard? Ideally, we should get it fixed at source. Auto-fixing on load is possible, but then we need to think about data governance and the possible ‘trust’ problems of the data being not aligned with the source. Maybe you could ’standardise’ country names and other columns on loading; I’ve seen systems with ‘USA’, ‘U.S.A’, ‘U S A’, ‘US of A’, ‘America’,  and ‘US’ in the country data feed and that’s before we get to the mis-keying of ‘United’ to get ‘Untied’!  But maybe that sort of improvement in quality should also be available to operational systems users.

For this customer, I have suggested that they construct a source to BI target matrix and include some basic traffic light measures on the source data:

• How good is it?
• What sort of errors are present; missing items, typographical errors, missing or incorrect parents, inconsistent use of names, even data entered in the wrong fields.
• How important is it to be correct in the BI system; for example street address can not be aggregated in reporting and we may not be going to use BI to create mailing lists, but postal code (or a sub string of it) can be used to aggregate people by location areas.
• How important is it to be correct on the operational source - do we need to apply the corrections at source to improve the operational use of the system

But this type of quality review may not tackle the data problem that is probably hardest to deal with what is a correct fact? How do I know if house value of £20,000 is reasonable (it  could be in a shared ownership scheme) or £2,0000,000 or £20,000,000? We could set a validation range, but where is there that point that one penny more is obviously wrong, but the current value OK?