It’s interesting being up here in Birmingham for a reason other than the UKOUG conference – I’ve been up here a few times since last December for various client visits, this time though I’m here for a few days and staying in Jury’s Inn, the hotel we all stayed in for the last conference. Outside it’s snowing though Broad Street looks as busy and manic as usual, I may even pop around to one of the bars around Brindley Place one evening just for a bit of UKOUG meetup nostalgia.

The other thing I’ve been working on is moving all my Oracle BI software on to a new virtual machine based around VMWare, as opposed to Parallels which I’ve been using ever since getting a Mac (VMWare is our company standard). One thing I would also like to get up and running in this new environment is the Oracle BI Apps, which like OBIEE you can download in full from edelivery.oracle.com; I’m keen to see how the BI Apps dimensional model fits together, how the DAC works and how Informatica gets data in to the warehouse, and whilst I could do this using the “BIC2G” demo environment that’s available to partners, installing and configuring it yourself always gives you a bit more insight into how it all works.

One slight issue that I think I’m going to have is that, of course, I’ll need an ERP system to extract data from, Oracle e-Business Suite is the obvious candidate but it’s no small install – last time I looked it was around 33Gb worth of software, which means I’ll probably have to create another VM to put this in and of course go through all the install steps, but at least I should then have the Vision demo instance to use as a data source (these blog postings seem to suggest it’s possible and not too tricky a task, once you’ve managed to download all the disks).

The other thing I considered was one of these “E-Business Suite in a box” setups that gives you 11i on a Mini-ITX server, this would probably be a whole lot easier but I’m not sure if I can justify dropping £600 or so on it. Any other suggestions (such as somewhere I can download a VMWare VM with e-Business Suite already installed) would be gratefully appreciated.

Our original plan was to cover Oracle BI Suite Standard Edition and Enterprise Edition, together with Oracle Warehouse Builder, and base the book on the 10g releases of the various products. The idea here was to take readers through the various tools in OBISE (Discoverer, Reports, Discoverer Portlets) and show how they could link in with the new tools (OBIEE, ODI and so forth). As we started writing the chapters though, it became more and more apparent that all of this would come together much more coherently in the 11g timeframe, with for example OWB being able to create OBIEE metadata, OBIEE Dashboards being able to incorporate Discoverer worksheets, Essbase coming in to the frame, OWB orchestrating the Oracle database bit and ODI having a more obvious role in bringing in non-Oracle data. There’s also a whole bunch of new functionality coming in around process integration (the “Action Framework”) that’ll be so central to OBIEE 11g and it’s integration with Fusion Middleware that we just didn’t want to leave this out.

In short, it made a lot more sense for us to wait for the 11g release of Oracle’s BI tools rather than write the book for what is obviously a “work in progress” in terms of integration and functionality. Coupled with the fact that, with publishing schedules as they are, the book would probably only be out for a few months before OBIEE11g and OWB11gR2 would start becoming available, holding fire and writing for these new versions made a lot of sense.

So the plan right now is to resume writing the book once the beta releases of OBIEE 11g and OWB11gR2 become available, and aim for handover of the manuscripts to the publishers towards the end of 2008, which gives us a publication date around the second calendar quarter of 2009. With a bit of luck this will coincide with the public release of the software, which means the book will be spot on in terms of topicality and should hopefully be something that’ll be of a lot of interest to readers of this blog. Whilst you can pre-order the book on Amazon (and I do appreciate the support here), in reality nothing will ship now until that time, and I’ll certainly keep you all posted on progress on the blog. In the meantime, thanks for the enquiries and interest and I promise it’ll be worth the wait.

Before I start, I review the new data warehouse tables I’ve imported into my OBIEE physical model.

Notice all the alias tables in there – I’m going to use those later on when creating separate logical dimension tables for each of my fact table date hierarchies. Now, I use the physical database diagrammer to check that all of my new warehouse tables are joined together properly.

This is particularly important as I’ve imported the views that OWB creates over dimension tables in instead of the actual tables themselves; this is because OWB in versions 10gR2 upwards adds additional rows (with negative dimension keys) into dimension tables for levels above the bottom level, so that you can join to the dimension at any level. For what I’m doing though, these extra rows are just confusing for users and so I import the views in instead, which filter these rows out. The views though, of course don’t have FK relationships with the fact tables, and so I have to use the OBIEE diagrammer to create these for me.

Doing this is fairly easy as each dimension table/view that OWB creates has the same column name, “dimension key” as the primary key column, and I join these back to the fact tables on the relevant fact table columns. You don’t join on the surrogate key columns, you join on these dimension key columns instead, which is a bit confusing first of all but when you get the hang of it, it (sort of) makes sense.

Now that my physical model is set up correctly, it’s time to re-map the dimensions using new logical table sources. To take an example, my Jobs dimension logical table has a business key of Job ID and a number of attributes. I leave this business key and attributes in place, but add a new column called “dimension key” which will hold the new dimension/surrogate key that the warehouse dimension table will use.

Then, as the Job ID business key will no longer be the primary key for this logical column, I delete it and create a new primary key that maps to the new dimension key column.

Finally, I map the columns from the warehouse job dimension table onto the logical job dimension table, which adds this new warehouse table as the logical table source for this logical table.

This process is then repeated for the other dimension tables until they’re all swapped over to use the new data warehouse tables. Then, I repeat the process for the fact tables, starting with the Appointments Fact table, which initially has four business keys that link it through to its logical dimensions.

I edit the logical table definition and remove these business keys, as I now need the fact table to join using the dimension keys I created earlier.

I then create a new primary key for this fact table that uses the new dimension keys.

Finally, I map the warehouse physical columns on to these new logical columns, so that the logical fact table’s logical table source now points through to the warehouse fact table.

In most cases, I don’t need to re-join the logical fact tables to the logical dimension tables as I previously created these joins as complex (logical) joins, which don’t specify actual logical columns and instead use the joins found in the physical layer, which now use the dimension key joins that OWB created.

I repeat the process for the other fact tables, replacing business keys in the logical fact table with the new dimension keys that the logical dimension tables now use, like this:

Once this is all done, I review the final model prior to testing it all out.

Now, the final test: do the reports I created right at the start, which ran against my single unified logical model that took data from the three different source system, still work now I’ve re-wired the model so that it now points to the data warehouse? I start off by trying the simple tabular report.

Bingo. The report still runs and returns the same data as before. What about the crosstab report?

Nice one. It runs the same, with no changes needed to the report, as the only real changes I’ve made are to the table keys, with the logical model adjusting automatically itself to the new datatypes as I mapped in the new logical table sources.

So, what do I conclude from all this? Well, firstly, it was a lot easier to re-wire the OBIEE logical model than I expected, principally because I’d built it independently of the original source systems first of all, then mapped them in and then afterwards switched this to the warehouse later on. The main change that I needed each time was to bring in the new dimension key that OWB created for each dimension, but this didn’t affect any of the existing reports as I could still keep the business key for each table, and the reports themselves didn’t display any key values, they just used them to join to other tables.

This approach should also work for slowly-changing dimensions – a dimension where we tracked attribute value changes (a Type 2 dimension) would just end up with more than one dimension key value for a given business key, but it’d still join back the same way and values for a particular dimension member would be aggregated as long as the dimension key wasn’t included in the report.

So, I was fairly pleased with this. The most time-consuming part was putting the OWB data model and mappings together, if Oracle can automate the process of generating the data warehouse schema and mapping data to it, this’ll make the whole process much much simpler. As it is though, it’s eminently do-able which will give me a lot more confidence proposing it at my talk later in the week.

In todays posting, I’m going to take the data held in these three application schemas and create a data warehouse out of them using Oracle Warehouse Builder 10.2.0.3. Before I do this though, one thing I have done is extend the logical model in OBIEE to three fact tables and a few more dimensions, so that it becomes a complete dimensional model over all of my source data. When I come to create the data warehouse, it’ll have fact tables corresponding to the four fact tables in my logical model, and dimensions to match the total set of dimensions in the model.

Before I start the process, there’s a few issues that I can see cropping up with this migration. Firstly, OWB is going to create surrogate keys (or to be more precise, a “dimension key”) for each of the dimension tables, and it’s these rather than business keys such as CUST_ID, PROD_ID and so on that are going to join to the fact tables. The fact tables themselves will have surrogate/dimension keys rather than business keys in them, and together these issues are going to mean that the tables I map to in OBIEE are going to be keyed on different fields than before.

Another issue that immediately springs to mind is how OBIEE will handle slowly-changing dimensions. OWB has automatic support for slowly-changing dimensions in the tool, but it’s not clear (at least to me) how OBIEE will handle this. I’m actually going to cheat on this a bit and just use Type 1 dimensions in my example, but I’ll give some thought as I go on to how Type 2 dimensions, which can have multiple entries per dimension member, partitioned by time, will be handled.

The final issue is around all the extraneous columns and rows that OBIEE adds to the tables it creates, particularly the dimension tables. OWB creates a surrogate key for each dimension level, as well as a “dimension key” field that acts as table-wide surrogate key, which has the benefit of allowing slowly-changing dimensions and separation from changes to source system keys, but to the inexperienced it seems like an awful lot of keys. In addition, you get extra rows in your dimension tables for all the levels above the bottom level in your hierarchy, which in conjunction with the dimension key I mentioned earlier allows you to join to the dimension table at any level of aggregation, but which could confuse people if they just want to list out customers, or products for example. There are ways around this (OWB automatically creates views that at least remove the extra rows, and in fact the dimension key concept is quite simple once you work it out) but again it’ll need a bit of thought.

Anyway, the first thing I do is go into Oracle Warehouse Builder and create modules for the three source applications and the data warehouse that I’m going to create.

Then, I create the dimension and fact objects, together with their supporting tables, in the data warehouse module. In this instance, I create them as relational (ROLAP) objects as it’s a bit trickier, currently, to import MOLAP objects into OBIEE.

Once that’s done, I create the mappings to load data from the source tables into the dimension objects. OWB takes care of all the surrogate key handling for me, all I have to do is map in the source data.

Now I can deploy the dimension objects and tables to the Oracle database, and run the mappings to load data into them.

At this point, my dimension tables have data loaded in to them. The next thing to do is to map the transactional information from my source systems into the fact tables. Again, OWB takes care of converting business keys into surrogate keys, the only tricky thing is the daft way that OWB specifies the date business key as a number, in the format YYYYMMDD, which means I have to convert it from the date format used by the source into this numeric format, before I can load it into the fact table.

Finally, once all the data is loaded into the warehouse, I can take stock of the final OWB project, which looks like this:

and then load the resulting objects into my OBIEE repository, like this – note the views over the dimensions that I’ve brought in, I’ll come back to this in tomorrows’ posting.

So, that’s the data warehouse created from the source data I brought in to OBIEE yesterday. Although these steps seem all fairly simple, it was quite a time-consuming task to do this – it took me a good few hours to set up all the dimension and fact objects, make sure all the datatype lengths are correct, create the mappings, add all the constants and conversions that were needed and so on, so I’m looking forward to some sort of automation of this process in the future – automating the creation of a physical schema from the logical model to create the data warehouse schema would be an obvious easy win, and it should even be possible to automate the creation of the data mappings from the source systems to the warehouse tables.

Tomorrow though, I’ll be mapping the logical model in my OBIEE metadata layer to these new warehouse tables, and trying to see whether it’s possible to preserve all the reports and avoid changing the logical model around. It’ll be interesting to see how the table key changes are handled, and how easy it is to get rid of all the extra information OWB adds to the warehouse tables.

All well and good, but how well does this work in practice? Well, there’s no better way to find out than to give it a go, so what I’m going to do over the next three days is work through three stages of the type of project I’m suggesting:

1. Firstly, we’ll start off with three operational systems (Orders, CRM and HR), and create a single logical model that maps on top of them.
2. We’ll then create some reports against this data and save them for checking later.
3. Then, we’ll use OWB to extract this data from the source systems and create a data warehouse
4. We’ll then re-map the logical model I created earlier to point to this new data source
5. Then, we’ll check the same reports and see whether they still work ok.

Starting up SQL Developer and taking a look at the three application data sources, the Orders schema contains a number of tables in a “normalized” design. It’s got an orders table, an order details table, various product tables and so on.

The CRM schema is a bit simpler, and contains details on customer interactions, their reason and their outcome. The customers it refers to are in the Orders schema.

So what we have here is a set of normalized tables that contains details on order activity, refunds, staff appointments and customer interactions. Out of these tables I therefore want to create a dimensional model, with the fact tables corresponding to the activities I just mentioned (orders, appointments, customer calls and so on) and the dimensions corresponding to the customer, product, employee and other lookup tables in the source schemas.

Starting off with a new repository then, the first thing I do is start creating this logical model. I’ll start off by modeling the facts first, then I’ll build out the dimensions, and only once that’s complete will I start thinking about mapping them to the source data.

I start off them by creating my first fact table, which will model the orders that I receive.

At this initial stage, I just create the logical columns within the fact table; the datatypes for each column will be defined later once I map source columns to them. Then, I repeat the process for the dimension tables, again just creating the column names and leaving the definition of the datatypes until later on. The point here is that you are modeling what the logical model should be, later on we’ll worry about mapping these objects to the source data.

Now, I create logical(complex) joins between the dimensions and fact table that I’ve just created.

Then, it’s a case of defining the dimensions for my logical model and setting the aggregation method for the measures, and the logical model is complete.

So I’m now at the point where I have a complete first cut of my logical model. At the moment, I’ve only got a single fact table, later on I’ll add some more, but for now my next step is to map in the source tables. To do this, I use the Import … From Database feature in Administrator and read in the source table metadata.

One of the most important things you have to do when importing source data into OBIEE is to make sure all the primary and foreign keys are registered in the physical model. I therefore go and create a physical database diagram for each schema, making sure all the tables are properly joined together – this for example is the Orders schema.

Then, I go and create joins between the different schemas, joining for example the tables in the HR schema to the Order schema via the Orders table, as employee data is references in orders via the Salesperson ID.

By repeating this across all of the source schemas, I make it possible for OBIEE to join together all the source data into a single, “virtual” source database.

Now it’s time to map the source data onto the logical model I created earlier. I start off by dragging the Order ID field from the Order Details physical table and dropping it on top of the Orders Fact logical table, which adds this physical table as a logical table source for the fact table and maps the Order ID logical column to the Order ID physical column.

I then repeat the process for the Order Line ID column, and the rest of the columns in this source table that have corresponding entries in the logical fact table.

The Order Details physical table only contains a part of the order information I want to put into my logical fact table – the most granular part that details the individual line items in the order. I also want to map across some data from the actual Orders table as this contains the order “header” information, such as the date of the order and the customer who made the order. To do this, I first map the existing fact table logical table source to this additional table, which I can do because the Orders and Order Details tables are joined together in the physical model.

Now I can just map in the additional columns from the Orders table to finish off this logical table’s mapping.

Once this step is complete, I’ve mapped in all the source columns for the logical fact table, with the fact table’s logical table source pointing to two source tables, the Orders and Order Details tables.

I then repeat the process for the logical dimension tables, editing the logical table sources as needed to provide access to additional lookup tables.

Once this is complete, I then edit the logical tables to add the primary keys, and then check the state of the logical model at this stage.

Everything is now complete except that I haven’t yet provided source table mappings for the time dimension. There’s a reason for this though, in that the source systems don’t have a “Times” table, all dates are held as Date datatypes in the various transaction tables and it’ll therefore be up to me to make a separate table out of them, together with derived fields such as Month, Day of Week and so on.

In my Orders fact table, time actually crops up a couple of times; there’s an Order Date and a Ship Date, and to be able to analyze these dates independently, I’ll need to create two time dimension tables, one for order data and one for ship date. I rename the Time dimension table I created earlier on to Order Date instead, and then map in the Order Date from the Orders table on to the Day ID logical column within it.

I then edit the logical table source for the Order Date logical table and use the Expression Editor to derive the various date components from the Order Date physical column.

We’re now ready to go. I create a simple Presentation model out of my Logical Model, start up Oracle BI Answers and run a couple of test reports.

The tabular report comes back OK. What about one that involves dates and a bit more analysis, say a pivot table report?

Not bad, it looks OK. So, tomorrow we’ll look at the next step – taking this operational data spread over three applications and copying it into a data warehouse, which we’ll do using Oracle Warehouse Builder 10gR2.