A Future Oracle OBIEE Architecture

February 15th, 2008 by

Following on from my blog article earlier this week on future Oracle BI architectures, I’ve read through the comments and had a think about how things might work, and put together some thoughts on how organizations might use Oracle’s Enterprise Edition BI tools now and going into the future.

To recap, the idea here is to take the traditional BI architecture of a data warehouse, metadata layer and a query tool and update it for the heterogeneous, service-orientated world that is Oracle now. In this architecture, we’re going to take advantage of the ability for Oracle Business Intelligence Enterprise Edition Plus (OBIEE, previously known as Siebel Analytics) to map to multiple data sources (including applications, relational databases and OLAP servers) and create a single logical business model, and the fact that Oracle now have an ETL tool in Oracle Data Integrator that can extract from and load to a range of different databases, web services and other business services. We’re also going to try and think about how the tools from Hyperion will work in this architecture, both now in their unintegrated state and going into the future, when Oracle will no doubt try and integrate them with the OBIEE platform.

To start at a high level, the proposed future Oracle BI architecture will have three distinct layers;

  • a Data Services Layer that contains the data and associated ETL processes
  • a Business Logic Layer that maps to the Oracle BI Server, and
  • a Presentation Layer that maps to the Oracle BI Presentation Server and the Hyperion planning and performance management tools.

The idea of a data services layer comes from the SOA world, and is a way of creating an abstraction layer for SOA business processes that need to access business data. In our architecture, it contains the application data sources, the data warehouse when we create it, OLAP data sources, web service and business event data, and the ETL processes that moves and transforms data. The business logic layer is where the BI Server holds physical, logical and presentation models of your data, together with hierarchies, calculations, metrics, KPIs and access rules, with the presentation layer mapping to Oracle BI Presentation Server and the legacy Hyperion CPM tools that take data from the business logic layer in the former case, and the data services layer (at least initially) in the latter case.

To illustrate how the architecture might be developed, we can consider a typical organization who wishes to implement an Oracle business intelligence solution. For this organization, their deployment can be broken down into three separate phases:

  1. Initial deployment of pilot reports as “quick wins” and to determine detailed requirements
  2. Gradual consolidation of reporting data into a data warehouse
  3. Implementation of advanced functionality in the OBIEE 11g+ timeline

This assumes that the organization has no particular BI strategy in place at the moment, currently provides reports through disparate tools directly against source applications, and wishes to provide a reporting solution as part of a wider Oracle Fusion Middleware strategy.

1) Initial Pilot Deployment

Organizations looking to deploy BI solutions often have two conflicting drivers for how they approach their project; they want to get reports to users quickly, so that they can take advantage of the market opportunity that’s led them to want a BI solution, however the IT department are often concerned with the long-term viability of the reporting solution and wish to build it based on a data warehouse.

In the past, it was often difficult to cover both of these requirements as once you starting building reports and a metadata layer against your operational applications, it was difficult to re-point these towards a data warehouse once you started building one. With OBIEE though, organizations can create a metadata model that separates the logical objects that users report against from the physical database tables that provide data for them.

Therefore, in this first phase, we will create an initial metadata layer in the business logic layer using Oracle BI Server and Oracle BI Administration, and initially point it towards the source applications that contain the data users want to report on. Where possible, a single logical model will be created, however at this early stage it’s likely that the business logic later will contain individual logical, physical and presentation models for each source system being accessed.

Where appropriate, Oracle Data Integrator will be used to created pre-joined copies of certain source application tables, together with some limited aggregations, which will be placed in a database held within the data services layer. These tables will go on to form the kernel of the data warehouse that will be built in the next phase. Using this initial data services, business logic and presentation layers, the business can provide some initial reports, gain some quick wins and also provide a pilot platform that helps the process of gathering more detailed requirements.

2) Gradual Consolidation of Data into a Data Warehouse

The main phase in development follows the initial pilot and quick wins and looks to put in place a scalable, consolidated reporting environment. Where appropriate, data is extracted in real-time or batch from the source applications and consolidated into a classic data warehouse, with staging, atomic and dimensional (performance) layers, ideally using a database such as Oracle to provide indexing, summary management, in-database OLAP and storage of large amounts of detailed and summary-level data.

For more complex analytical needs, or if the business wishes to use planning, budgeting or financial consolidation tools, data can be loaded from the warehouse into an OLAP server such as Essbase, with all transformations and data loading taking place through Oracle Data Integrator and its repository. ODI also provides a means to integrate data via messages, enterprise service bus, web services and business events, with the overall data service layer refresh process being orchestrate through BPEL and ODI agents.

The primary route for data out of the data services layer is through this data warehouse. For data sources that are transitory in nature, or that have not been incorporated into the warehouse yet, direct access to these sources will still be supported with the business logic layer continuing to map to these sources as needed, as well as the new data warehouse.

The business logic layer, initially defined in the pilot phase, contains models describing the data sources within the data access layer, the business model over these data sources, and the customized views of the business model used by different parts of the organization.

Initially, most of the query tools within the presentation layer will access their data through this business model. Legacy Hyperion tools will for the time being bypass this layer and talk directly to the Essbase server in the data services layer.

In the initial phases of the BI implementation it is likely that multiple logical business models will exist in this layer whilst separate data sources are being combined into combined fact tables and conformed dimensions, the end goal though is to produce a single unified logical model that spans the entire organization and allows analysis across linked dimensions and facts.

3) Implementation of advanced functionality in the OBIEE 11g+ timeline

Everything in the architecture so far is supported by the current generation of OBIEE, ODI, Oracle SOA Suite and Hyperion tools. Going into the future though, there are obvious ways in which this architecture can be improved as integration continues between Oracle’s BI tools.

At present, OBIEE can generate aggregates to speed up user queries through the use of the Aggregate Persistence Wizard. It is likely that in future, OBIEE will also be able to generate a physical database schema to match a logical business model in the OBIEE repository, either in a relational database or as a multi-dimensional cube in Essbase or Oracle OLAP.

It’s also reasonably likely that OBIEE will also be capable of automatically generating ETL mappings for tools such as ODI, “function-shipping” the process of transforming data to an ETL tool rather than have the BI Server do it at runtime. To what extent these function-shipped ETL mappings will be able to handle complex mappings is obviously not known.

It is also likely that much of the metadata used by the former Hyperion tools will make it’s way into the OBIEE physical, business and presentation models, with the business model undergoing development so that it can hold items such as metrics, KPIs and goals as first-class data items.

Eventually, you can imagine the existing OLAP query tools used by Essbase (Web Intelligence etc) being replaced by the forthcoming release of Oracle Answers, which can present data as a multi-dimensional hierarchical model, negating the need for standalone OLAP query tools.

So what’s left? Well we haven’t tackled some of the “edge case” OBIEE tools such as Real-Time Decisions (this is likely to sit in the business logic layer and provide services for the presentation layer), and tools such as Brio, Discoverer and the like which are not part of Oracle’s strategic direction.

It also rather glosses over the slightly awkward manner in which ODI provides access to service and event-based data (more on that in a later posting), and of course bringing data together into a warehouse whilst at the same time providing consistent data for the business logic layer is of course a complex task, but this is I guess where the data mart automation tools coming in OBIEE 11g+ are likely to help out – the idea here is that persisting your report data in a dimensional model should be a fairly automated process, not one that has to be re-invented for every deployment.

For now though, hopefully it’s a fair stab I hope at pulling together a coherent, next-gen Oracle BI architecture, as usual comments are welcome and I’d appreciate any feedback before I start to put the presentation together.

Comments

  1. Stewart Bryson Says:

    This illuminates a lot of what was mentioned in the last posting. But my concerns center around the Data Services layer, and moving from phase 1 to phase 2. Web services, and any form of data extraction, are great for OLTP applications, capturing business processes and business entities as objects with attributes and methods. It makes code modular and easily callable from disparate applications.

    However, data abstraction in the form of services and objects is exactly what we don’t want in ETL. We need to code complex SQL statements with complex joins using non-standard Oracle functionality such as multi-table inserts, merge statements, error logging clauses, etc., in order to gain the type of performance needed to load a full data warehouse. Integrating ODI with it’s knowledge modules specifying these advanced processes may do the trick, but even with the knowledge modules, it seems difficult to implement a metadata layer that can take into consideration the web-services route and the ODI route when the former methodology depends on complete abstraction while the latter will suffer from it.

    If we abandon the notion of batch loading of the warehouse, then I think I can see where your vision could have legs. With a SOA framework where the service bus is loading the warehouse in real-time, then the performance of the batch becomes less an issue, and the logical abstraction is once again feasible. But in this type of implementation, the use of an ETL tool seems superfluous, as ODI and OWB are still targeted more at batch loading than anything else. Maybe their gradual evolution should also be taken into consideration to see this vision realized.

    I enjoyed the divergence into the philosophical for at least a moment. We are all so busy with the here and now, it’s nice to sometimes think about how things might (and maybe should) be.

  2. Stewart Bryson Says:

    In the first paragraph, I meant “data abstraction” not “data extraction”… whatever that is.

  3. Mark Rittman Says:

    Hi Stewart,

    Thanks for the feedback, this was just what i was looking for, thanks. By a data services layer, I’m not meaning to say that all data will be provided to OBIEE via web services, with data being integrated in real time/on demand, a.k.a. EII – the idea is that there are many service providers, the most prominent of which just happens to be the data warehouse. The warehouse is still loaded via batch from the various sources, some of which may be web services. In addition, OBIEE may still continue to get data directly from the source applications where appropriate, an in future, it may also source directly from web services, although I’d rather that they went through the ETL process and the data warehouse first of all.

    Perhaps I need to rename the data services layer to something less ambigious, as I’m not suggesting turning all data providers into web services – the warehouse is still central as is it’s ETL load processes. It’s more a case of creating an abstraction layer away from the complexity of the individual data sources, but as you say it can be intepreted as going away from traditional ETL batch loads into a warehouse towards more service-orientated data provision. I’ll have to give it some though.

    regards – Mark

  4. Stewart Bryson Says:

    I don’t think renaming “data services layer” is necessary. After re-reading, I think the mistake was mine. I was assuming a service bus that served as the center of the hub through which everything flowed. However, I think you make clear that the ETL processes do not have to run on the SOA bus to be part of the data services layer.

    I think I was also caught up (especially in your first posting) with the idea that the initial metadata mapping designed in phase 1 would also include everything logically needed to move to phase 2. It now seems like the movement to phase 2 would include the kind of work we always do when building a data warehouse. It looks like the important message here is that the switch would be an invisible one (except for increased performance) from the user perspective.

  5. Rittman Mead Consulting » Delving in to ODI’s Web Services Support Says:

    […] Comments Stewart Bryson on A Future Oracle OBIEE ArchitectureMark Rittman on A Future Oracle OBIEE ArchitectureStewart Bryson on A Future Oracle OBIEE […]

  6. Chandra Says:

    Great article as always!
    What is the difference between OBI Dashboard and Hyperion Dashboards, will there be a common Dashboarding solution as part of OBI EE+

  7. Rittman Mead Consulting » Blog Archive » Re-Wiring OBIEE Logical Models To Use A Data Warehouse : Part 1 Says:

    […] couple of weeks ago I posted an article about a next-generation Oracle BI architecture, where I advocated building a first cut OBIEE system against your operational applications, […]

  8. Rittman Mead Consulting » Blog Archive » Migrating OBIEE Logical Models to use a Data Warehouse : Part 2 Says:

    […] yesterdays posting, I took a look at some of the practicalities behind my recent article on OBIEE “next-generation” architectures. The idea behind this architecture is that you can use the connectivity features of OBIEE to […]

  9. Carl Petersen Says:

    In your article, you write: “These tables will go on to form the kernel of the data warehouse that will be built in the next phase. Using this initial data services, business logic and presentation layers, the business can provide some initial reports, gain some quick wins and also provide a pilot platform that helps the process of gathering more detailed requirements.”

    My question is, what is the likelihood that the business will give you time to go back and properly integrate the data into the data warehouse? The business has what they want, and now that they got something quickly, I would expect that they would want their next request filled just as quickly! Like the old saying says, “Pay now or pay later, their is no free lunch!”

  10. Mark Rittman Says:

    Hi Carl,

    Well, I guess it’s a cost/benefit thing – if they can get by without using a DW, well then they should do so, but I suspect performance will be an issue running against source data, and there are probably some areas of data integration that won’t be possible until a proper ETL process is in place, so the client may have to put up with lots of separate logical models rather than one single one. Yours is a good point though, especially if the main benefits of going down the DW route are mainly IT ones that the business does not perceive.

  11. Peter Scott Says:

    As Mark says, performance is an issue; but DWs also address the issue of data quality (or at least well designed data warehouses do!) when you have multiple source systems and need to conform dimensions at query time to cope with differing keys (or even descriptions) between sources you have an uphill struggle – the data warehouse is saleable to the business on grounds of better quality of information

  12. Grahame Crispin Says:

    Mark, ITV currently have a hybrid Oracle / Microsoft BI stack supporting silo’d BI and will do so for the forseeable future; we want to move to a common ITV-wide BI Platform that embraces what we’ve currently got but enables component interoperability wherever we can. Do you have any observations about how “A Future Oracle OBIEE Archiecture” can encompass other BI suppliers’ technologies – specifically Microsoft SSIS MS Reporting Services and Proclarity ?

  13. Mark Rittman Says:

    Hi Grahame,

    Good to hear from you. I think in terms of an architecture, it’s pretty vendor neutral at least in terms of data sources, presentation tools and so on – OBIEE (or at least the BI Server) is the key enabler through the single logical metadata model, it’s heterogeneous connectivity (to both Oracle, MS AS etc) and it’s ability to provide data for a range of reporting tools, including Reporting Services.

    Certainly you could use any ETL tool to provide the data warehouse layer – SSIS would be as good as any – with Proclarity performing the OLAP query tool role that Hyperion Web Analysis would provide in an all-Oracle stack.

    You should drop me an email offline if you get a chance – mark.rittman@rittmanmead.com – I’d be interested to hear about how you use your hybrid reporting technology and how it might fit in this sort of architecture.

    regards, Mark

Website Design & Build: tymedia.co.uk