Anyway, take a look if you get a chance, take a look also at the OLAP Option pages on the Oracle Wiki, and if you’re going to Collaborate’08 don’t forget that Peter Scott is doing a conference session on the Thursday on this very subject.

Before I start, I review the new data warehouse tables I’ve imported into my OBIEE physical model.

Notice all the alias tables in there – I’m going to use those later on when creating separate logical dimension tables for each of my fact table date hierarchies. Now, I use the physical database diagrammer to check that all of my new warehouse tables are joined together properly.

This is particularly important as I’ve imported the views that OWB creates over dimension tables in instead of the actual tables themselves; this is because OWB in versions 10gR2 upwards adds additional rows (with negative dimension keys) into dimension tables for levels above the bottom level, so that you can join to the dimension at any level. For what I’m doing though, these extra rows are just confusing for users and so I import the views in instead, which filter these rows out. The views though, of course don’t have FK relationships with the fact tables, and so I have to use the OBIEE diagrammer to create these for me.

Doing this is fairly easy as each dimension table/view that OWB creates has the same column name, “dimension key” as the primary key column, and I join these back to the fact tables on the relevant fact table columns. You don’t join on the surrogate key columns, you join on these dimension key columns instead, which is a bit confusing first of all but when you get the hang of it, it (sort of) makes sense.

Now that my physical model is set up correctly, it’s time to re-map the dimensions using new logical table sources. To take an example, my Jobs dimension logical table has a business key of Job ID and a number of attributes. I leave this business key and attributes in place, but add a new column called “dimension key” which will hold the new dimension/surrogate key that the warehouse dimension table will use.

Then, as the Job ID business key will no longer be the primary key for this logical column, I delete it and create a new primary key that maps to the new dimension key column.

Finally, I map the columns from the warehouse job dimension table onto the logical job dimension table, which adds this new warehouse table as the logical table source for this logical table.

This process is then repeated for the other dimension tables until they’re all swapped over to use the new data warehouse tables. Then, I repeat the process for the fact tables, starting with the Appointments Fact table, which initially has four business keys that link it through to its logical dimensions.

I edit the logical table definition and remove these business keys, as I now need the fact table to join using the dimension keys I created earlier.

I then create a new primary key for this fact table that uses the new dimension keys.

Finally, I map the warehouse physical columns on to these new logical columns, so that the logical fact table’s logical table source now points through to the warehouse fact table.

In most cases, I don’t need to re-join the logical fact tables to the logical dimension tables as I previously created these joins as complex (logical) joins, which don’t specify actual logical columns and instead use the joins found in the physical layer, which now use the dimension key joins that OWB created.

I repeat the process for the other fact tables, replacing business keys in the logical fact table with the new dimension keys that the logical dimension tables now use, like this:

Once this is all done, I review the final model prior to testing it all out.

Now, the final test: do the reports I created right at the start, which ran against my single unified logical model that took data from the three different source system, still work now I’ve re-wired the model so that it now points to the data warehouse? I start off by trying the simple tabular report.

Bingo. The report still runs and returns the same data as before. What about the crosstab report?

Nice one. It runs the same, with no changes needed to the report, as the only real changes I’ve made are to the table keys, with the logical model adjusting automatically itself to the new datatypes as I mapped in the new logical table sources.

So, what do I conclude from all this? Well, firstly, it was a lot easier to re-wire the OBIEE logical model than I expected, principally because I’d built it independently of the original source systems first of all, then mapped them in and then afterwards switched this to the warehouse later on. The main change that I needed each time was to bring in the new dimension key that OWB created for each dimension, but this didn’t affect any of the existing reports as I could still keep the business key for each table, and the reports themselves didn’t display any key values, they just used them to join to other tables.

This approach should also work for slowly-changing dimensions – a dimension where we tracked attribute value changes (a Type 2 dimension) would just end up with more than one dimension key value for a given business key, but it’d still join back the same way and values for a particular dimension member would be aggregated as long as the dimension key wasn’t included in the report.

So, I was fairly pleased with this. The most time-consuming part was putting the OWB data model and mappings together, if Oracle can automate the process of generating the data warehouse schema and mapping data to it, this’ll make the whole process much much simpler. As it is though, it’s eminently do-able which will give me a lot more confidence proposing it at my talk later in the week.

In todays posting, I’m going to take the data held in these three application schemas and create a data warehouse out of them using Oracle Warehouse Builder 10.2.0.3. Before I do this though, one thing I have done is extend the logical model in OBIEE to three fact tables and a few more dimensions, so that it becomes a complete dimensional model over all of my source data. When I come to create the data warehouse, it’ll have fact tables corresponding to the four fact tables in my logical model, and dimensions to match the total set of dimensions in the model.

Before I start the process, there’s a few issues that I can see cropping up with this migration. Firstly, OWB is going to create surrogate keys (or to be more precise, a “dimension key”) for each of the dimension tables, and it’s these rather than business keys such as CUST_ID, PROD_ID and so on that are going to join to the fact tables. The fact tables themselves will have surrogate/dimension keys rather than business keys in them, and together these issues are going to mean that the tables I map to in OBIEE are going to be keyed on different fields than before.

Another issue that immediately springs to mind is how OBIEE will handle slowly-changing dimensions. OWB has automatic support for slowly-changing dimensions in the tool, but it’s not clear (at least to me) how OBIEE will handle this. I’m actually going to cheat on this a bit and just use Type 1 dimensions in my example, but I’ll give some thought as I go on to how Type 2 dimensions, which can have multiple entries per dimension member, partitioned by time, will be handled.

The final issue is around all the extraneous columns and rows that OBIEE adds to the tables it creates, particularly the dimension tables. OWB creates a surrogate key for each dimension level, as well as a “dimension key” field that acts as table-wide surrogate key, which has the benefit of allowing slowly-changing dimensions and separation from changes to source system keys, but to the inexperienced it seems like an awful lot of keys. In addition, you get extra rows in your dimension tables for all the levels above the bottom level in your hierarchy, which in conjunction with the dimension key I mentioned earlier allows you to join to the dimension table at any level of aggregation, but which could confuse people if they just want to list out customers, or products for example. There are ways around this (OWB automatically creates views that at least remove the extra rows, and in fact the dimension key concept is quite simple once you work it out) but again it’ll need a bit of thought.

Anyway, the first thing I do is go into Oracle Warehouse Builder and create modules for the three source applications and the data warehouse that I’m going to create.

Then, I create the dimension and fact objects, together with their supporting tables, in the data warehouse module. In this instance, I create them as relational (ROLAP) objects as it’s a bit trickier, currently, to import MOLAP objects into OBIEE.

Once that’s done, I create the mappings to load data from the source tables into the dimension objects. OWB takes care of all the surrogate key handling for me, all I have to do is map in the source data.

Now I can deploy the dimension objects and tables to the Oracle database, and run the mappings to load data into them.

At this point, my dimension tables have data loaded in to them. The next thing to do is to map the transactional information from my source systems into the fact tables. Again, OWB takes care of converting business keys into surrogate keys, the only tricky thing is the daft way that OWB specifies the date business key as a number, in the format YYYYMMDD, which means I have to convert it from the date format used by the source into this numeric format, before I can load it into the fact table.

Finally, once all the data is loaded into the warehouse, I can take stock of the final OWB project, which looks like this:

and then load the resulting objects into my OBIEE repository, like this – note the views over the dimensions that I’ve brought in, I’ll come back to this in tomorrows’ posting.

So, that’s the data warehouse created from the source data I brought in to OBIEE yesterday. Although these steps seem all fairly simple, it was quite a time-consuming task to do this – it took me a good few hours to set up all the dimension and fact objects, make sure all the datatype lengths are correct, create the mappings, add all the constants and conversions that were needed and so on, so I’m looking forward to some sort of automation of this process in the future – automating the creation of a physical schema from the logical model to create the data warehouse schema would be an obvious easy win, and it should even be possible to automate the creation of the data mappings from the source systems to the warehouse tables.

Tomorrow though, I’ll be mapping the logical model in my OBIEE metadata layer to these new warehouse tables, and trying to see whether it’s possible to preserve all the reports and avoid changing the logical model around. It’ll be interesting to see how the table key changes are handled, and how easy it is to get rid of all the extra information OWB adds to the warehouse tables.

All well and good, but how well does this work in practice? Well, there’s no better way to find out than to give it a go, so what I’m going to do over the next three days is work through three stages of the type of project I’m suggesting:

1. Firstly, we’ll start off with three operational systems (Orders, CRM and HR), and create a single logical model that maps on top of them.
2. We’ll then create some reports against this data and save them for checking later.
3. Then, we’ll use OWB to extract this data from the source systems and create a data warehouse
4. We’ll then re-map the logical model I created earlier to point to this new data source
5. Then, we’ll check the same reports and see whether they still work ok.

Starting up SQL Developer and taking a look at the three application data sources, the Orders schema contains a number of tables in a “normalized” design. It’s got an orders table, an order details table, various product tables and so on.

The CRM schema is a bit simpler, and contains details on customer interactions, their reason and their outcome. The customers it refers to are in the Orders schema.

So what we have here is a set of normalized tables that contains details on order activity, refunds, staff appointments and customer interactions. Out of these tables I therefore want to create a dimensional model, with the fact tables corresponding to the activities I just mentioned (orders, appointments, customer calls and so on) and the dimensions corresponding to the customer, product, employee and other lookup tables in the source schemas.

Starting off with a new repository then, the first thing I do is start creating this logical model. I’ll start off by modeling the facts first, then I’ll build out the dimensions, and only once that’s complete will I start thinking about mapping them to the source data.

I start off them by creating my first fact table, which will model the orders that I receive.

At this initial stage, I just create the logical columns within the fact table; the datatypes for each column will be defined later once I map source columns to them. Then, I repeat the process for the dimension tables, again just creating the column names and leaving the definition of the datatypes until later on. The point here is that you are modeling what the logical model should be, later on we’ll worry about mapping these objects to the source data.

Now, I create logical(complex) joins between the dimensions and fact table that I’ve just created.

Then, it’s a case of defining the dimensions for my logical model and setting the aggregation method for the measures, and the logical model is complete.

So I’m now at the point where I have a complete first cut of my logical model. At the moment, I’ve only got a single fact table, later on I’ll add some more, but for now my next step is to map in the source tables. To do this, I use the Import … From Database feature in Administrator and read in the source table metadata.

One of the most important things you have to do when importing source data into OBIEE is to make sure all the primary and foreign keys are registered in the physical model. I therefore go and create a physical database diagram for each schema, making sure all the tables are properly joined together – this for example is the Orders schema.

Then, I go and create joins between the different schemas, joining for example the tables in the HR schema to the Order schema via the Orders table, as employee data is references in orders via the Salesperson ID.

By repeating this across all of the source schemas, I make it possible for OBIEE to join together all the source data into a single, “virtual” source database.

Now it’s time to map the source data onto the logical model I created earlier. I start off by dragging the Order ID field from the Order Details physical table and dropping it on top of the Orders Fact logical table, which adds this physical table as a logical table source for the fact table and maps the Order ID logical column to the Order ID physical column.

I then repeat the process for the Order Line ID column, and the rest of the columns in this source table that have corresponding entries in the logical fact table.

The Order Details physical table only contains a part of the order information I want to put into my logical fact table – the most granular part that details the individual line items in the order. I also want to map across some data from the actual Orders table as this contains the order “header” information, such as the date of the order and the customer who made the order. To do this, I first map the existing fact table logical table source to this additional table, which I can do because the Orders and Order Details tables are joined together in the physical model.

Now I can just map in the additional columns from the Orders table to finish off this logical table’s mapping.

Once this step is complete, I’ve mapped in all the source columns for the logical fact table, with the fact table’s logical table source pointing to two source tables, the Orders and Order Details tables.

I then repeat the process for the logical dimension tables, editing the logical table sources as needed to provide access to additional lookup tables.

Once this is complete, I then edit the logical tables to add the primary keys, and then check the state of the logical model at this stage.

Everything is now complete except that I haven’t yet provided source table mappings for the time dimension. There’s a reason for this though, in that the source systems don’t have a “Times” table, all dates are held as Date datatypes in the various transaction tables and it’ll therefore be up to me to make a separate table out of them, together with derived fields such as Month, Day of Week and so on.

In my Orders fact table, time actually crops up a couple of times; there’s an Order Date and a Ship Date, and to be able to analyze these dates independently, I’ll need to create two time dimension tables, one for order data and one for ship date. I rename the Time dimension table I created earlier on to Order Date instead, and then map in the Order Date from the Orders table on to the Day ID logical column within it.

I then edit the logical table source for the Order Date logical table and use the Expression Editor to derive the various date components from the Order Date physical column.

We’re now ready to go. I create a simple Presentation model out of my Logical Model, start up Oracle BI Answers and run a couple of test reports.

The tabular report comes back OK. What about one that involves dates and a bit more analysis, say a pivot table report?

Not bad, it looks OK. So, tomorrow we’ll look at the next step – taking this operational data spread over three applications and copying it into a data warehouse, which we’ll do using Oracle Warehouse Builder 10gR2.

I briefly discussed this last year on my blog and probably also over at Datageekgal’s blog (and congratulations Beth on the new job!) The indexing needs of the security services, medicine and a whole host of organisations that need to index patterns within a LOB type object will spawn some pretty clever indexing methods, and hopefully some of those will become accessible through database vendors products.

In a way there are some similarities with data mining, except that a LOB could (I think would) contain bit patterns for more than one index key value. We are probably talking about non-unique indexes, as for non-archival purposes researchers are usually concerned with finding similar records.

But one good thing about the need to index LOB contents is that they are usually non-volatile, a recorded conversation or a DNA sequence is historical fact and is not going to change so perhaps index updates are not going to be important. Most of the building blocks to do this type of indexing are already available (especially if we choose to create an index of the “index” by using some form of indirect table approach) the only bit to do is to write the domain specific code to identify the keys values in the LOB… hang on that’s the hard bit!

Recently, I have been set the challenge to grow the amount of data warehouse and data quality work I am involved in and perhaps at the same time help dispel the perception that Rittman Mead Consulting just do short engagements and training; in the three months or so I have been here I have spent just three days delivering training and 2.5 months on-site with a single customer developing a data warehouse, so that notion is a bit of a myth anyway. Maybe the new Rittman Mead web site will feature some of the longer term type of engagements we are involved in.

I suppose a good question to start with is “do people still develop data warehouses?” Mark brought up a similar idea the other week in his ‘future BI architecture’ piece which also attracted some good comments. Indirectly, I mentioned alternatives to DWs in my article on Real Time BI for the Evaluation Centre – I hope to either post the article here on the blog or perhaps on the Rittman Mead web site in the next few days. A lot of organisations have already invested in data warehouses, these will continue to need attention, either from the support perspective or for enhancement as new classes of information are added. Data warehouses still have a place in BI as way of facilitating the delivery of quality data to the reporting layer in an efficient way so I can’t quite see the death of the Data Warehouse just yet.

I could run out the argument that I work in a BI consultancy and none of us here at Rittman Mead are 100% aligned to a specific expertise area; for example I do OBIEE and OWB as well as data warehouse design and performance. And besides, in my past I did look after one of the largest Oracle Sales Analyzer systems in Europe (ROLAP reporting on a couple terabytes of relational data), so there is quite a bit of Oracle Express Server (and OLAP DML) lurking in my past. But in reality my talk is going to look at the use of OLAP Cube Organized Materialized Views in the context of summary management and that, of course, is right at the heart of what I do!

Summary management has always been one of those ‘art form’ areas of DW/BI – which summary tables to build and how many of them are needed to balance performance, maintenance time and space usage. Query rewrite was great step forward in Oracle 9.2 (OK, it was there before, but I tended not to use it) at last, I only needed to map a single fact table (the base table) into the query tool, and for some third-party tools that was a significant thing. And now with Oracle 11g, the ability to transparently rewrite a query against the base fact table to a OLAP cube is a significant advance, it means that I needn’t think about the materialized views I need create since the cube effectively contains them all! This is a bit of a simplification of course and I will say a lot more in the 50 minutes or so of speaking slot – so if you are Collaborate 08 – and that is just a month away, you could get to hear the whole talk.