Towards a Future Oracle BI Architecture?
February 12th, 2008 by Mark Rittman
One of the presentations I’m giving over the next couple of months is one for the UKOUG BI & Performance Management event on future Oracle BI architectures. The driver for this for me is around all the different options that are now available for building Oracle BI systems, now that we’ve got products from Siebel, products from Hyperion and of course the traditional Oracle BI tools such as Discoverer, Portal, Daily Business Intelligence and Oracle Reports.
In particular, having all these new tools opens up a number of questions around how you put together a BI system going into the future.
- Do you still build a data warehouse upfront, dealing with all the data and integration issues upfront, or do you use the data integration features of BI tools such as Oracle BI Suite Enterprise Edition to integrate the data on-the-fly?
- Do you plan and design your warehouse upfront as an IT initiative, or do you let the users design the reports and BI metadata and have those drive the warehouse design?
- How do you incorporate the planning and budgeting process into your warehouse design?
- Given the productivity of tools like OBIEE, can we significantly shorten the time to deliver BI projects?
- If we go for a mixed BI/DW architecture featuring data warehousing, ETL, BI tools and their metadata, planning tools, OLAP servers and so on, how do create a consistent level of security over our environment, and how do we capture metadata and metrics across the entire system?
- Finally, give the adoption of technologies such as SOA, Web Services, data service layers and the like, how do we incorporate these sources and activities into our BI architecture so we can leverage their features, incorporate their data and still stay relevant when not all business logic and data is held in databases?
- Do you still base your BI systems around a single, monolithic data warehouse, that integrates data into a single database, deals with all the data issues once and once only, applies Oracle performance techniques such as materialized views, bitmap indexes and partitioning, with the BI element of reports and graphs delivered towards the end of the project? Or do you use the prototyping and data integration features of tools such as Oracle BI Enterprise Edition to create your data warehouse on?
So you see, it’s quite an interesting, but tricky topic, but one that I think is very pertinent to Oracle BI users and customers given some of the feedback I’ve had at user group events – not quite knowing what to do next following all of Oracle’s acquisitions is the number one issue I hear from feedback sessions at events, and it’s this uncertainty I’m looking to address.
Anyway, to cut a long story short, an idea that I’ve been batting around for a while, and something I’ve discussed with people such as Doug Cackett and Andrew Bond in Oracle, is an attempt to try and pull together an architecture that recognizes the benefits of a properly structured and loaded data warehouse, but that incorporates some of the new options that are open to us now that tools like OBIEE are around.
Data Warehouses give us consistent, reliable data plus the benefits of database server technology designed to handle large sets of summary and detail-level data. However they take a long time to build and as far as BI systems are concerned, the value they provide dimishes rapidly the longer you take to deliver them. Your warehouse project might get signed off based on some immediate market opportunities your company has spotted, but if you take a year to deliver the system, by then the opportunity may well have gone.
Tools like OBIEE however allow you to put a BI system together now, mapping its metadata layer directly against the underlying source data, applying aggregates and caches to try and speed up query performance. In some cases, tools like OBIEE can actually allow you to integrate data across multiple systems, though this will never be as fast to query as data that actually resides in just one database. Over time, systems built like this get harder and harder to manage, but at least you get reports and analysis in people’s hands fast and the business aren’t usually interested in arcane technical discussions about the merits of a properly architected data warehouse.
The thing is though, what it you could combine the two approaches, having the business define the reports and the report metadata (in OBIEE terms, the semantic model) and with the metadata layer initially mapping through directly to the source data,. Then, as time goes by, you can migrate more and more of the data you’re reporting on to a proper data warehouse, all the time keeping the user’s reports going through the ability of the BI tool metadata to be “re-pointed” to the data warehouse as subject areas come online? Tools like OBIEE allow business users to define a logical dimensional model that query tools then work against, as a BI architect you can start with this metadata layer pointing to the source data and over time migrate it to proper data warehouse structures.
At the moment, building these warehouse structures requires you to define a dimensional model in a tool like OWB, then write your mappings to bring across data from source systems into the warehouse. In time though, features on the OBIEE product roadmap promise to considerably simplify this process, with OBIEE offering the option to persist the logical model it works with in a relational schema or a multi-dimensional database such as Essbase or Oracle OLAP, and in the future it wouldn’t take too much work for OBIEE to also generate the ETL routines in a tool such as Oracle Data Integrator to copy data from your source systems into this persistence (or in other words, data warehouse) layer.
So you see the benefit here is that we put the drivers for our BI system into the hands of the business users, allowing them to create their business model and the reports that make use of it, whilst in the background we can gradually consolidate the data they report on into a proper data warehouse, in future using features to come in OBIEE to automatically generate the warehouse model or cube, and the mappings that move the data from the source systems into this data store.
The way in which we move data from sources into the warehouse, or persistence layer, is likely to evolve over time as well. At present, 99% of the data movements I see on projects are data mappings that move data from one database table to another. In future though, we’re likely to see data coming in via web services, via enterprise service bus and messages, via business events and so on, and so the tools we use to move that data will need to evolve as well. Tools like Oracle Data Integrator are already able to handle data via these sources, whilst the next release of OWB is also slated to handle web service and message sources. Taking a step on, it’s worth thinking about architecting the data integration element of your BI system as a formal SOA “Data Services Layer” where data is abstracted away from your BI tools, integrated using a set of loosely-coupled services, and handled in a framework such as Oracle Fusion Middleware. Is this likely to be a dominant pattern in the near future? I can’t say for sure, but as organizations move to SOA architectures we likely to get our data more and more via non-traditional data warehouse sources.
Finally, what about the Hyperion tools, how are they going to fit into this architecture? Well in terms of specifics, who knows yet, but for now I’d say some likely integration points are in the BI tools metadata layer, in the dashboard frameworks that deliver planning, budgeting and operational BI reports, and I guess in time both planning & budgeting data, and the data on our operational performance, will more and more be available as services to be consumed by a wider BI architecture, so the distinction between reporting data, planning data, forecasting data, data derived from databases or from business events, and so on and so on, will eventually become blurred – they’ll all just be services in a service-orientated architecture.
Anyway, that’s some thoughts for me around how we can preserve the benefits of a data warehouse architecture but take advantage of the new capabilities of OBIEE, and also respond to the take-up of service-orientated architecure. I’d be keen to get other people’s feedback (especially if you’re going to the UKOUG event) to see whether this makes sense, whether it addresses the questions you’ve got around where Oracle’s BI architecture is going, whether you think it’s realistic to start off with an OBIEE model going against source data and eventually migrate it to running against a data warehouse, whether you think events and services will be big factors in BI architectures, and so on. Add some comments if you like, I’ll respond to the comments and follow-up later in the week.
February 13th, 2008 at 3:16 am
What you are describing in terms of the report now, build-later approach is what the Wherescape product Red claims to do, but I have only heard this from the marketing guys. I don’t know if it really delivers on this promise. It’s sort of ironic that the Wherescape folks considered their only real rival in this space to be Sunopsis.
February 13th, 2008 at 6:27 am
An inteeresting proposal that may be suitable for smaller more flexibe organisations. One issue I see is access to information ‘on the fly’. Many of the project I wok on ony allow data access out of hours, and only after aggregations have been built.
Another issue is performance, we often use datawarehouses to integrate data and appy the necessary structure (bitmaps, star schema, aggragates) to ensure that end user reports run quickly. We are working with source dataabases that contain billions of records, which cannot be analysed by Analytics on the fly, and so we usually recommend going down the SAS route for detailed analysis, and aggregates into the warehouse for summary reporting.
Thirdly, we always need to ensure a single version of the truth. For example, when sales managers get their Sales forecast reports for the weekly meeting with the team, everyone needs to have the same numbers. This is best achieved by controlling the published data in the warehouse.
February 13th, 2008 at 7:46 am
@Stewart – yes, it does have some similarities to the Wherescape product, I stumbled across this a couple of weeks ago. I guess a major difference is that (to my eye) Wherescape still looks like it’s a DW-driven product, even where it links in with Microstrategy – you build the DW first (albeit in a much faster, more productive way) and the tool then generates the BI metadata as part of the process, whereas the approach I’m suggesting is driven by OBIEE – the end user defines the logical model, IT links it to the source systems and then either manually starts creating a warehouse and ETL routines to start persisting some, then all of the data in a warehouse or cube, perhaps in time OBIEE can automate this process, automatically generating a DW dimensional model or cube together with the ETL routines to populate it. The key here is to capture the knowledge and drive of the business users whilst still taking advantage, in time, of the scalability and performance of a single data warehouse or cube. It’s an interesting comparison though, I’ll work through the Wherescape online demo and take a closer look, especially as you say it’s similar in concept to some of ODI.
@Adrian – another good point, and this of course is where the reality of corporate policies (only allowing access to data away from the source system and aggregated) and the difficulties of data aggregation (providing a single consistent data source when data is integrated as needed, and where you are constantly migrating source data into the warehouse) comes in. I guess the “single version of the truth” issue can be addressed by providing all data access through the OBIEE semantic model – what the model points to over time may change, but at any one point, the Sales measure in the business model will provide the same results to all users of the system, whilst with performance, pointing directly towards the source systems is an interim step whilst the warehouse is created and populated, and users may be willing to put up with slow access to a particular subject area if it means they can get data much earlier than normal. Good points though, ones I’ll need to address in the presentation.
February 13th, 2008 at 12:38 pm
Very useful article Mark that highlights an old issue. How does commitment to the IT highlighted benefits of a Data Warehouse get cemented in to the Business?
It’s a typical scenario. The Business approaches IT with the need for a reporting solution. IT say ‘great, lets build a Data Warehouse’. The Business says ‘fine, how long?’ and then faces fall at the answer. ‘OK’, says IT, ‘how about we use the fine features of this new technology to deliver you a rapid application as a short term deliverable and work on the Data Warehouse solution afterwards?’. ‘Excellent’ says IT.
A month or so of constructive co-operation between IT and the Business ensues, resulting in the delivery of a fast, seemingly complete, solution.
The Business is so happy that they bring forward a new requirement. ‘What about the Data Warehouse?’ asks IT. ‘Why do we need that?’ says the Business. ‘The data we need is there, it’s fast and everyone can access it’. ‘Oh but what about slowly changing dimensions, advanced aggregations and future manageability?’ pleads IT. Business Eyes glaze over.
A bit simplistic for sure but reflective of many situations I’ve been in. Now, don’t get me wrong, the Business pays so the Business rules but it does seem to me that IT have a stake in it’s success as well. A more mature approach is required from both parties. IT need to accept that not everything will end up in the DW (some requirements are just too short term) and the Business need to understand that the long term usage of a solution rely in cost effective, consistent warehousing solutions. Flexibility is required from IT, whilst Fidelity is required from the Business.
Technology will help but it’s going to prove difficult for the DW to keep up with the rapid focus changes of the Business if they are designing their own solutions all the time. A co-ordinated approach is still required. I guess then, in answer to your post, I think that, yes, you can start off against source data but that, sooner or later, the need will arise for the Data Warehouse and that that need should be communicated to all at the start of development and the benefits to be gained reinforced throughout.
February 13th, 2008 at 1:32 pm
What an excellent topic. “…not quite knowing what to do next following all of Oracle’s acquisitions…” seems to also be an issue that Oracle themselves are struggling with.
Knowing what direction to go down when you have Oracle DW, Hyperion Sys 9 and Siebel Analytics (OBIEE) all present in your organization is a tricky business, and with 11g the move is clearly towards db hosting for all data … so where is Hyperion going? Hosting out of Oracle DB’s in 11g, with MV-like refresh capabilities?
Some statements on medium and long-term strategy would be much appreciated.
February 13th, 2008 at 2:24 pm
Mark:
I agree with your points concerning the difference between the paradigm you described and the Wherescape one. And once again… what I know of the product is from the marketing guys combined with a project manager with very little technical information. Where it does resemble what you are describing a little closer is that the end result is a comprehensive metadata layer. For Red, that layer is manifested in terms of load routines… for OBIEE, it is a data access layer. You mentioned that perhaps OBIEE could offer both in the future… the “access-now” manifestation of the metadata layer combined with a “generate-later” manifestation that is implemented as ODI code. Red does allow live interaction with the data at every single point along the way, but that is certainly not the same thing.
It’s a very interesting view into what may be coming. My concerns are really echos of what Adrian already mentioned. I wonder if a mapped-metadata layer can do the kind of cleaning and conforming that is needed for an EDW. It’s certainly a logical possibility… but one that doesn’t (as far as I know) exist in reality as of yet. I guess I need to roll-up my sleeves and learn OBIEE backwards and forwards… especially since it may make my skills irrelevant :-)
February 15th, 2008 at 10:18 pm
[...] Comments Stewart Bryson on Towards a Future Oracle BI Architecture?David Aldridge on Towards a Future Oracle BI Architecture?Stuart Wallace on Towards a Future Oracle [...]
March 9th, 2008 at 3:49 pm
[...] Mark brought up a similar idea the other week in his ‘future BI architecture’ piece which also attracted some good comments. Indirectly, I mentioned alternatives to DWs in my article [...]
March 25th, 2008 at 10:22 am
[...] Mark brought up a similar idea the other week in his ‘future BI architecture’ piece which also attracted some good comments. Indirectly, I mentioned alternatives to DWs in my article [...]