I have long felt that BI is only done for one of three reasons – the law says we must report things, it saves us money, or it makes us money; so if knowing something sooner gives us competitive advantage then surely that is a good thing. Knowing sooner is not enough though; it is also about being able to act on the information to facilitate a change in the organization that enhances return (or lowers costs). To my mind we are moving from the traditional “let’s look at this in aggregate” stance to a world where we ask “what is the significance of this newly observed fact”. This type of analysis requires a body of data to create a reference model and access to smart statistical tools to allow us to make judgments based on probabilities. Making such decisions based on dynamic events is not just for stock markets and bankers, the same principles apply in many sectors. I know of some restaurant chains that have investigated using centrally monitored sales across all outlets to dynamically adjust staff levels based on likely demand – staff are sent home, brought in, moved between outlets based on a predictive model that uses past trading patterns across many outlets.

As usual, most of the building blocks we need to do this are available to us, we just need a bit of creativity to join them together into an architecture. For this kind of use I feel that messaging should be core to the data capture – we want to look at single items of “fact” and do some statistical analysis on them before adding them to the data warehouse (or what ever form our data repository takes) so that the new fact can become part of the base data set we use to analyze the next fact to arrive. Micro-batch loading of log based change data is probably less suited here as we are:

1. adding to the latency by using discrete loads at fixed intervals and
2. the processing of many items at a time complicates the statistical analysis and alerting phases (after all if we get 2453 credit card transactions in a batch only a few will be potentially fraudulent).

After we capture a message from the source system we can pass the information through a chain of processes to analyze the information, propagate alerts based on the statistical significance of the item and add the data to the data store so that it becomes part of the knowledge. This last stage of adding the message to the data will probably need to be in a micro-batch mode rather than one-row-at-a-time-as-it-arrives – the latencies of adding fact to a conformed OLAP system (database, cubes, whatever) are such that single row additions will just take too much time, even if our target is an in-memory system. Here the art of the designer is to balance the availability of data, the time to reprocess the OLAP structures, the desire to keep the system up to date. It is always worth noting that for many data domains having to-the-second data is not that important as any new rows are unlikely to change the statistical results, however some subject domains will need access to all of the recent information including that which has not yet made it to the data warehouse, and here the creativity comes in.

One question that often comes up is “How is OWB different from ODI, after all they both do E-LT?” I have written a small series of blogs to be published over the next few months that look at this subject from the point of view of an OWB developer moving to ODI.

To start things off here is the first of the series where I am looking at OWB and ODI in high level terms and point out some of the key differences and similarities. I will be considering the two current releases (OWB 11.2 and ODI 11.1.1.5). Later blogs will look in more detail about the actual development of ETL process and how to orchestrate them.

Both ODI and OWB have a similar (I am being very simplistic here) three-component design of: a metadata repository, a development environment where the developer defines the processes and data flows and a runtime component that executes the code and flows. It is the “how” of these things that is different for the two tools.

Both are repository driven, that is the metadata that describes the ELT processes, data structures being accessed and host of other things is held in a database schema. For OWB the repository is pre-installed (the user needs to create a workspace though) in an Oracle 11gR2 database, optionally, the OWB repository can be installed into an other Oracle database if required. ODI’s repository is installed using Oracle Fusion Middleware’s Repository Creation Utility into a supported (and not necessarily Oracle) database. With ODI, the repository can be shared with other components that use the Fusion Middleware stack such as OBIEE 11g, whether this is desirable would depend on your circumstances and factors such as your organization’s software release process and network topology – just because it is possible to have all on one database does not make it desirable.

Cosmetically, there is a lot of similarity between the two development environments, they are both part of the same unified family of Java IDE applications as JDeveloper and SQLDeveloper; the look and feel is similar, for example double-clicking on a tab has the same effect (it toggles the tab’s panel between full-sized and windowed). What is different however is the content of the windows and navigators and that is a big topic for later postings.In practice, with OWB the key parts of the IDE are those for the development of MAPPINGS and (optionally) the design of process flows to orchestrate mappings. In the ODI world think INTERFACES for mappings and PACKAGES for process flows. This is simplistic though as ODI also has PROCEDURES (code developed in one of the ODI supported languages) and LOAD PLANS (multiple packages orchestrated to execute in serial or parallel). OWB mappings require the developer to include all of the components needed to facilitate the mapping – we connect source columns to target columns through a logic flow of joiners, filters, expressions, aggregates and a whole palette of other activities. Typically, this would generate a single, but large, SQL statement with much use of in-line views. ODI interfaces are simply about connecting source columns to target columns in a logical relationship (we also create expressions, joins and filters here) and allowing the physical implementation to be supplied by a knowledge module.

In its most common usage mode, OWB deploys its executable code into PL/SQL packages in the target database. Even pure SQL set-based insert code is wrapped into a package that contains the control and audit methods that allow it to execute under the control of the Control Center and the OWB runtime. The code generated by ODI depends on the knowledge modules used and might be native SQL which is executed directly against the target database by the Java agent executing the code. Again this is a big topic and more will follow in later blogs.

Take the postal code (or zip code) where does it fit in the hierarchy? Well the answer is might not fit at all. Postal codes were developed to help post offices deliver mail – and each postal authority did their own thing. The UK and the Netherlands have postal code systems that can identify a single street or even a cluster of houses of within a street. Other countries work on a code per town or group of nearby towns – so straight away we have a difference in grain; a few houses in the UK a few towns in France. In Germany postal codes relate to geographic areas but those areas are not aligned to the Bundesländer; on the other hand, France ties postal code to Department but there are anomalies notably where a river runs through a village and opposite banks share a postal code but are different Departments (and in one case, different regions). Some national postal codes are numeric, some area alphanumeric (like Canadian and UK ones), the length of the postcode varies between countries too.

Perhaps the sensible thing, especially if you are dealing with addresses from multiple countries, is to not use postal code as a level in geographical hierarchies. If you use them at all just make them as an attribute of the address and remember that they don’t always have geographical parents.

I think the key point about modelling geography is that just because you know how addresses work in your own country you can’t assume that they work like that in the country next door. If you have a requirement to report, for example, the efficiency of the postal service in delivering your goods by postal region you need to ensure that your reporting handles the anomalies and exceptions. As always, knowing your data is key to creating a correct model.

Look out for more Rittman Mead postings on ODI over the coming months

The more that I use ODI the more I like it, especially the recent 11g version with its ability to build source data sets within the GUI using set operators such as UNION and MINUS. Developing ETL processes can be as simple as connecting columns on the mapping tab and then specifying any knowledge modules needed to load, validate and store the data. However, the real challenge for me (and I love to be challenged) comes from making the generated code do exactly what I want it to do and in this ODI gives me immense flexibility. I can specify where individual transformations occur (source, stage, target or even a mix of all three), I am able to specify which columns are updated and which are inserted in an upsert type of incremental load, and, probably most significantly, I can adapt knowledge modules to do exactly what I want them to do.

Sometimes a change to a knowledge module can be as trivial as altering the name of the Oracle directory created for loading files as Oracle External Tables – this would be important if we need to process multiple external table locations within a single database schema. Other times we need to make more substantial alterations to the load, or even repurpose it. For example the Oracle MERGE incremental load includes a step using SQL MINUS to detect changes between source and target; we could change the table order in the MINUS operator so that we look for rows in the target that don’t exist in the source and thus build a knowledge module to do a “logical delete”.

Which for me means that ODI is not a cookie-cutter tool (though some people treat it so) but a way that allows me use my skill and judgement to do ETL in the way that I think that is most appropriate for the dataset.

Typically, I set indexes to be unusable by issuing an ALTER INDEX my_bitmap_index UNUSABLE and for partitioned tables ALTER INDEX my_bitmap_index PARTITION aug_10_part UNUSABLE, here I use a “smart partition name” that allows me to programatically determine the correct index partitions to manipulate – using the HIGH_VALUE of the partition from DBA_IND_PARTITIONS is problematic as it is a LONG data type and thus a tad convoluted to query (see Adrian Billington’s note on working with LONG columns).

However, when we use Oracle 11g interval partitioning the partition names are system generated; so how do we find the index’s partitions that need to be set unusable? There is no “PARTITION FOR” construct to alter an index’s partitions to be unusable, and the name of partition is not inherited from the table -it gets its own system generated name. There is still a way to do this and assuming that my_table is range partitioned by DATE interval; ALTER TABLE my_table MODIFY PARTITION FOR (TO_DATE(’1-Aug-2010′,’dd-mon-yyyy’)) UNUSABLE LOCAL INDEXES
Before interval partitions I have rebuilt unusable index partitions by writing a procedure to loop through all of the index partitions marked as unusable in the data dictionary and issue an ALTER INDEX …. REBUILD PARTITION command – here I can easily find the partition name to rebuild as the unusable marker is not held in a LONG, there is also an ORACLE package (DBMS_PCLXUTIL) that can do this; now, with Oracle 11g maybe the simplest thing though is to use ALTER TABLE my_table MODIFY PARTITION FOR (TO_DATE(’1-Aug-2010′,’dd-mon-yyyy’)) REBUILD UNUSABLE LOCAL INDEXES.

But before that I will take a small detour around what Oracle refers to as their Reference Data Warehouse and Business Intelligence Architecture. Here we are dividing the data warehouse into three “sections”: “staging” which is a local copy of data from the source systems; “foundation” which is typically a “process neutral” 3NF representation of the business data such as “customer”, “product” and “orders”. It is likely to be different in structure from the staged tables in that it could well be the merging of data from multiple sources, for example customer attributes could come from both CRM and ERP systems. This foundation layer is likely to be versioned (that is, whenever a dimensional attribute changes a new, current version is created) and non-aggregated. The data in this layer are our BI jewels; we don’t know what future analysis and data mining needs will be and by aggregating things we lose the flexibility we might need; remember there is no UNGROUP BY clause in SQL. The third tier is performance and access layer where we typically present optimized table structures to the query tools, it’s here that we have the aggregated tables, bitmap indexes and all of those other ‘traditional’ data warehouse features. This not really a revolutionary (or even new) architecture – I have been doing similar things in my data warehouse design since the 1990s.

One of the key things to note is that the staging tables for the dimensional data should be complete replicas of the source tables and not just a set of extracted rows provided by the source data owner. Here is the ideal place to bring in replication technology and hence the beginnings of a real time data warehouse. Fact (or more accurately in this case “events”) in the staging area are not going to be full replicas of the fact source but rather all of the events that have occurred since the last load, agin this can be achieved by realtime replication. Remember we only need to replicate the tables of interest and not all of the structures of the source applications. At first glance it might seem extravagant to have effectively three copies of dimensional data (one in each layer) and two of facts – but these days disk is cheap and it is also (tongue firmly in cheek) a good way to use some more of the disk space you had to buy to get the required data throughput.

Acquiring data in realtime is not going to be our problem, and if we can use the stage tables directly for reporting then we can say “job done” and not worry more. Our problems arise if we need to do significant work on the staged data to report over it. We might see problems with data quality, surrogate key management for dimensions, particularly slowly changing dimensions, and the need to aggregate facts to improve performance of the query tool.

I am not going to get into the debate on what to do about data quality, I have blogged about that in the past. The only thing I will say though is that the resolution to data quality problems should be in the source system(s) – data warehouses should report the same data as used in the transactional systems and if that requires a master data management program then so be it.

I suspect that this next point might be considered heresy, but if you have immutable business keys for your dimensional data (perhaps from a master data management system) then consider using them in the data warehouse – this will reduce the complexity of the ETL processes needed to push data from stage to the data warehouse, it might also remove a time dependency of pushing dimensional data through to the foundation and access layers in realtime. The need to track slowly changing type 2 dimensions (where we keep a history of change) might force the use of surrogate keys, but other approaches are possible that might avoid the need for surrogate keys being used on the fact tables; one approach is to split the non-volatile and SCD-1 attributes from the versioned (SCD-2) attributes and store the dimension in two tables, with the first table joined to the FACT table (on the business key) and the versioned table of SCD2 attributes joined to first dimension table on business key; queries against the second table will need to also pass a date so the correct version is selected, but this is not hard to achieve with most query tools. By far the easiest thing to do, though, is avoid SCD-2 all together; many organizations think they need to implement SCD-2, but when they come to use the system they find that SCD-1 actually fits the reporting requirements of the vast majority of their users.

If you ask someone on the outside of developing a (near) realtime data warehouse what the greatest challenge will be they probably would say “capturing the change” since they know we can already “do” data warehouses. I think that is wrong, capturing change is easy; the big problem is applying that change in a timely fashion to a data warehouse that also remains available for query. Adding relatively few rows of new fact to a table is trivial compared to the actions needed to validate, transform, apply keys, index, and publish the fact; and then think about the impact of merging that new fact into existing aggregate tables or materialized views. A lot of moving parts, a lot of challenge.

Realistically, we could populate an “atomic data store style” layer in realtime with what is in effect a versioned (timestamped, journalized or however you term it) replica of the source, a replica which is probably suited for realtime reporting but what we don’t get are the features of a data warehouse that we come to expect in a traditional star schema DW. We possibly miss out on: data validation through the ETL process, data enrichment and derived measures, conformed dimensions, slowly changing dimensions (especially type 2 SCD) through surrogate keys. It may well be that you don’t actually need a star model, after all one of the viable DW models for an Exadata warehouse is just that; a bunch of conventional tables joined on the natural business keys.

Another point to consider is that it is quite unlikely that all of the fact domains in a data warehouse need to be realtime ones; for example data sourced from a supplier’s EDI feed may arrive far less frequently than, say, sales transactions from the company’s web-store. Obviously, if we have realtime feed of sales, we must ensure we have all of the dimensional (reference) data loaded before a new transaction arrives, or else develop robust ways to handle this. This is a situation where we need business knowledge; if a new customer can be created at time of purchase (as often is the case for a web sale) we will need a realtime customer feed along with the realtime sales feed, but for banks with strict money laundering regulations customers are registered way before transactions occur, so a timely load of customer is likely to be sufficient.

Not only is it unlikely that all data feeds to a data warehouse need be realtime, it quite likely for some “facts” that only some measures are realtime measures. Consider sales: we know the quantity and the price charged to the customer at the time of the sale, but we may well not know the cost of goods until the time the order is fulfilled.

Non-realtime data warehouses often use a periodic batch data load paradigm; once a month, a week, a day, or whenever, we execute a batch process that extracts data from the source, identifies what has changed and applies that do the data warehouse. But what if we modify this batch to run much more frequently, say half-hourly. We are moving towards realtime. Loading data more frequently should reduce the volume of data in the individual loads, and less data should equate to a reduced batch times (but this is not likely to be a linear reduction in batch duration). But we are imposing some new challenges on the source extract – we will have frequently running queries that access multiple source rows, this will have an impact on the source performance – we will probably need to modify the extract code to robustly identify the contents of the extract windows, that is we must not miss or duplicate data. In addition we are imposing an impact on the target, we need a good method to publish the received data to the data warehouse without adversely affecting the query workload on the data warehouse and we must ensure that our micro-batch load-to-publish time is substantially shorter than the interval between the micro-batches.

However say “realtime data warehousing” to most people and they think of continuous data capture, possibly through a extract built on streams or SOA messaging from the source, but more than likely through synchronous or asynchronous change data capture using database triggers or redo logs. Again there is going to be some degree of latency between event and the data being replicated to the target. But now we have a design choice on the target. Do we consume the captured changes as continuous trickle-feed process? Or do we run a series of micro-batches to consume the data? By necessity true trickle-feed will move us into row by row processing and possibly significant impact on processes that need to aggregate data. I feel that most continuously-fed data warehouses will use some micro-batch for the majority of the DW transform and publish process, even if trickle-feed processing is used to populate an non-aggregated ODS style layer for the special cases when people need to see “now data”

As I mentioned in a previous blog, captured commit-based change can generate a lot of “noise”; commits associated with no data change, multiple updates to the same row, changes to columns of no interest to the DW system. How we choose to handle this in our load procedures will depend on what the business needs to see: final status for a row within the batch, treating clusters of row changes within a short period (such as a few seconds) as single change, or applying all changes (in the order they happened)

One of the points I make early on in my talk “Getting Real Data Warehouse Loading as it Happens” is that realtime business intelligence is not the same as realtime data warehousing; having one does not imply having the other.

People can, and do, direct their BI reporting at the source transactional system or systems and for these people there is no data warehouse involved; conversely others populate their data warehouse continuously through a realtime trickle-feed since that is the only viable way to load their data but choose to keep reporting aligned to an event in the past (such as midnight yesterday) so that users do not see changing data throughout the day.

So if it is possible to do realtime BI against the source systems why have a data warehouse? In my opinion the data warehouse is doing three things: it is protecting the transactional source from excessive, potentially sub-optimal queries; it is acting as a platform to conform data from multiple source systems and it allows the use of slowly changing dimensions.

Assuming we need realtime BI but do not want a realtime data warehouse we will need to mix transactional and reporting functionalities on the same system. Although it could be argued that a system such as Oracle’s Exadata V2 allows mixed reporting / transactional workloads few people have implemented Exadata yet and thus, for many people it means that they are mixing two distinctly different workloads on a platform designed for one workload. DML operations that are designed to insert or update a single row as quickly as possible are mixed with queries that access a great number of rows based on a combination of predicates that may well not be indexed on the OLTP system. Furthermore, some of the database features that assist performance in a data warehouse such as bitmap indexes and parallel query can actually inhibit performance of transactional systems. The fundamental problem of mixing workload is that if resource is busy doing one thing it is then not available to do something else and performance for all users suffers.

There are some things that we can do to reduce the BI query impact on the source. We can use data federation in a reporting application such as OBIEE and restrict the transactional source to current data and use a separate data warehouse, perhaps with pre-aggregated data to supply historic information; that is, we remove the need to repeatedly scan the whole of the transactional system to provide historic data that is no longer changing and by reducing the volume of data we are accessing we lessen the conflict with transactional activity. Of course we will still need a process that updates the data warehouse, but that can now be deferred to a later time. However, federation has its own drawbacks; where we once had one query we now have two and the results of these two queries need to be stitched together and that process has to happen on the BI server, where it consumes memory and processing resource to join the result sets. Another thing we could do is to use a replica database to be the source of reporting, thus reducing the impact on the transactional source (but not fully removing it as any from of replication has some impact as it will involve some process executing on or against the source). We could use a standby instance of the OLTP system for reporting, but we need to consider the licensing requirements of the replica, the location of the replica (it may be at a different site) and what happens if we need to use the standby system for its real purpose, business continuity – do we lose BI when that happens? Perhaps a better approach is to build a specific reporting replica. This replica can be the right size and have the appropriate product licensing for a reporting load. It does not have to be complete replica of the source system, just the tables of interest; if we have no reporting interest in audit and journal tables then we need not replicate them.

In fact we can get quite sophisticated with a reporting replica in that we change the database so that it is better geared for reporting; this might include dropping unhelpful indexes and possibly partitioning the data to allow partition pruning to occur. Adding bitmap indexes may be a step too far though as we are essentially receiving row by row updates and inserts and that results in lower throughput caused by table locking (actually multi-row locking) as the bitmaps change.

We can even combine the the federated approach with a database replica and use the replica as source for the recent transactions and the data warehouse for the historic data; and if we chose to implement our replica on a fast database, perhaps an in-memory one, we could get excellent performance with low impact on the transactional source.