This feature is based around Oracle Endeca Integrator Designer (“Integrator” for short), the client part of Integrator Suit that is used to design data loading graphs and optionally run them, and uses a “template project” which allows users to connect to an Oracle BI Server, run a select-query and then load repository data into an Endeca Server data domain. Some data domain attribute and indexing settings will also be configured, so that guided navigation and attribute/record search within Endeca Studio are set up for users looking to analyze the data we’re loading.

Two different scenarios come to mind that could make use of this feature. First, you might want to join some modeled data from your Oracle BI Server with other data that you have stored in Endeca Server; for example, allowing you to combine and analysis unstructured and structured data. In other words, you can create dashboards on your unstructured data that has been enriched by some descriptive structured data!

The second use case could be if you wanted to create some quick dashboards against data modeled within the BI Server repository, to find answers to new questions without the need to make extensive changes within the actual BI Repository – in other words, to carry out information discovery!

There are probably other use cases too, but whatever the reason you need to bring data from Oracle BI server to Integrator has made this very easy. In this post, therefore, I’m going to provide a step-by-step guide on how to setup this particular functionality, and load data from the OBI Server repository to the Oracle Endeca Server using the 3.0 release of OEID.

1. Firstly, open Integrator and select File > New > Project from the menu bar.

Note that with the earlier 2.4 release of Integrator, you had to make sure that a .jar plug-in file (found inside the installation package) for connecting to the Oracle BI Server had been installed in order to be able to see the “Load data from Oracle BI Server” wizard under Information Discovery category in the New Project displayed dialog. This step is not required for version 3.0. Click Load Data from OBI Server and press Next to proceed.

2. The Load Data from OBI Server wizard should be displayed. On the first page, enter the project name you are creating (for example, OBIConnection) or alternatively you can Use an existing project. In which case, after pressing Next again, the wizard will add new files to the project you have. I would suggest creating a new project to prevent any conflict, as you can always load the data into an existing data domain.

3. Next, In the Endeca Data Domain Configuration page, provide your up and running Endeca Server details and press Next to continue, for example:

Endeca Server Host: localhost

Endeca Server Port: 7002

Data Domain Name: Sample_Sales

The data domain name that you provide will be used to create a data domain as one of the initial steps of the Integrator project in a graph called “InitDataDomain.grf”.

Note, where the Endeca Server has been installed on a secure mode, you need to provide the ssl-listen port.

4. Now you’ll need to enter the OBI Server connection details such as User, Password, OBI Server host and OBI Server port; For example:

User: Weblogic

Password:  Password01

OBI Server host:  10.67.70.117

OBI Server port:  9703

9703 is the port that I could connect to. You should validate the credentials by Connecting to OBI Server and once you see the Connected message on the top of the dialog, proceed to the Next step.

5. You are now able to choose the repository you want and select the relevant objects/tables you need from the list of Tables by checking them. Press Next to continue.

6. The wizard will automatically prepare the required meta-data information for the attributes that you selected. Using this page, you can configure some OEID features such as Search Interface by editing the displayed values (Note that this property is only available for STRING attributes). Check the Edit Finished box (located towards the bottom left of the window) when you are happy with all configurations and then click Finish.

7.  Having now finished the wizard, a new project with all the configurations, metadata files, connection, SQL statement and graphs required to connect to the OBI Server will be added to the Navigator pane.

8.  You can now see the required database connection, as shown in the screen shot below, where the Oracle JDBC driver has been used.

9. It’s now time to run the Baseline graph in order to load the OBI Server repository data into the Oracle Endeca Data Domain. Expand the Graph folder within Navigator pane and open Baseline.grf. Press the green Run button from the toolbar. Check the Console to see Execution of graph successful.

Some graph components may require SSL configuration, if you are using a secure Endeca Server implementation; For example in the Load_Data graph, the Bulk_Add/Replace component has a SSL Enabled property that should be updated to true. Also, you’ll also need to change all http requests to https.

The “Load Data” graph uses a Query-Statement, which is located in the data-in directory and contains a SELECT statement from OBI Server repository objects.

10. Now its time to log into Endeca Studio and create a Data Source in the Control Panel pointing to the data domain you created in the Integrator project.

Test connection to ensure that it is working properly.

11.  One more change in OEID version 3.0 is that the old ‘Liferay portal’ term ‘Community’ has been updated to ‘Application’, which to me, feels more related to the subject of Information Discovery and thus makes sense, but that’s just a personal opinion. An ‘Application’ refers to a subject area where users can create different dashboards on different views of one and only one data-source.  Being in the home page of Oracle Endeca Studio, select New Application using the approved, newly created and validated data-source.

12.  OEID version 3.0 kindly, generously and automatically generates a new dashboard with all the required components such as a searchbox, a breadcrumb, a guided navigation and of course a results table.

So far we have created an Oracle Endeca Information Discovery dashboard upon our Oracle BI Server repository, but what is the advantage of doing this? Rather than being able to create reports very quickly, Perhaps joining some unstructured data from other data-sources to the Oracle BI Server repository and a bit of Text tagging, enrichment and salience analysis could be done in Integrator and will make the dashboards much more interesting.

Good luck to those of you trying this out.  Look out for my next post where I’ll explain how to read from Endeca Server and join its data to those coming from other sources.

There will also be experienced experts available before, during and after the event, providing a rare opportunity to get your questions answered by people who have been there and done it.

Click here for more information and to register.

Photos from the two events (a selection from Brighton are above, some from Atlanta below this paragraph) are available in these Flickr photo sets:

• “Rittman Mead BI Forum 2013, Brighton” (Flickr photo set)
• “Rittman Mead BI Forum 2013, Atlanta” (Flickr photo set)

As we always do, we’re also making the slides (where allowed by the speaker, and not under NDA) available for download using the links below, including the one-day Oracle Data Integration Masterclass provided by Stewart Bryson, Michael Rainey and myself. Note that Christian Screen’s and Jeremy Harms slides are actually online, so I don’t think you’ll be able to download them from whatever service is hosting them – sorry.

Oracle Data Integration Masterclass (Stewart Bryson, Michael Rainey, Mark Rittman, Rittman Mead)

• “Introduction to Oracle Data Integrator 11g”
• “ODI and the Oracle Reference Architecture for Information Management”
• “ODI and GoldenGate – A Perfect Match…”
• “ODI and Hadoop, MapReduce and Big Data Sources”
• “The Three R’s of ODI Fault Tolerance : Resuming, Restarting and Restoring”
• “Scripting and Automating ODI using Groovy and the ODI SDK”

Brighton RM BI Forum, May 8th – 10th 2013

• “OBIEE SampleApp 11.1.1.7 functional highlights” (Philippe Lions, Oracle Corporation)
• “OBI Performance Tuning – Real Customer Success Stories” (Antony Heljula, Peak Indicators Ltd)
• “Secrets of OBIEE implementation at LGI” (Marco Klaassens, Liberty Global)
• TED Session 1:  ”Why I want to be working with Business Intelligence in 5 years time” (Jon Mead, Rittman Mead)
• TED Session 3 : “Incrementally loading Exalytics using Notepad” (Antony Heljula, Peak Indicators Ltd)
• “Oracle Data Integrator 11g Best Practices. Busting your performance, deployment, and scheduling headaches.” (Uli Bethke/Maciek Kocon, Independent)
• “New Developments in BI Multi-tenancy and Cloud” (Adam Bloom, Oracle Corporation
• “The Magic of Aggregates” (Michael Wilcke, sumIT AG)
• “Integrating Oracle BI, BPM and BAM 11g: The complete cycle of information” (Edelweiss Kammermann, Awen Consulting)
• “Endeca – Beyond the Demos” (Adam Seed, Rittman Mead)

Atlanta RM BI Forum, May 15th – 17th 2013

• “It’s all in the genes – The power of Oracle Exadata and the Oracle Database” (Rene Kuipers, VX Company)
• “In Memory Analytics – Times Ten, Essbase 11.1.2.2 – Analysis – A Comparison” (Venkatakrishnan J, Rittman Mead)
• TED Session 3 : “A BI Publisher Beginner’s MacGyver-Hack for Financial Reporting with OBIEE: A Quickie!” (Jeremy Harms, CD Group)
• “Performance Tuning the BI Apps with a Performance Layer” (Jeff McQuigg, KPI Partners Inc)
• “Thinking Clearly about Performance” (Cary Millsap, Method R Corporation) – see also the accompanying technical paper
• ”Forecasting and Time Series Analysis in Oracle BI” (Tim & Dan Vlamis, Vlamis Software Solutions Inc)
• “Hadoop versus the Relational Data Warehouse.” (Alex Gorbachev, Pythian)
• “How to Create a Plug-In for Oracle BI 11g” (Christian Screen, Capgemini)
• “ODI and Hadoop / Big Data” (Alan Lee & Marty Gubar, Oracle Corporation)
• “BI Applications 11g and ODI” (Florian Schouten, Oracle Corporation)
• “OBIA 11G – What You Need To Know: Part 1″ (Kevin McGinley, Accenture)

So once again – thank you to everyone who came along, especially the speakers but also everyone from our Brighton and Atlanta offices who helped set the event up, and made sure it all ran so smoothly. See some of you again in Brighton and Atlanta next year, and our next outing is to ODTUG KScope’13 in New Orleans – another great event with the BI Track organised by Kevin McGinley – make sure you’re there!

This seamless nature of aggregate navigation means that testing specific aggregates are working can be fiddly. We want to ensure that the aggregates we’ve built are (i) being used when appropriate and (ii) showing the correct data. This is the particularly the case in Exalytics when aggregates are put into in-memory (TimesTen) by the Summary Advisor and we need to validate them.

Whilst the log file nqquery.log (or Usage Tracking table S_NQ_DB_ACCT) tells us pretty easily which table a query used, it is nice to be able to switch a query easily between possible aggregate sources to be able to compare the data. This blog demonstrates how we can use the INACTIVE_SCHEMAS variable (as described in my previous blog on loading Exalytics incrementally) to do this.

INACTIVE_SCHEMAS is a Logical SQL variable that tells the BI Server to exclude the specified physical schema(s) from consideration for resolving an inbound query. Normally, the BI Server will parse each incoming query through the RPD, and where a Logical Table has multiple Logical Table Sources it will evaluate each one to determine if it (a) can satisfy the query and (b) whether it will be the most efficient one to use. By using INACTIVE_SCHEMAS we can force the BI Server to ignore certain Logical Table Sources (those associated with the physical schema specified), ensuring that it just queries the source(s) we want it to.

In the following example, the data exists on both Oracle database, and TimesTen (in-memory). Whilst the example here is based on an Exalytics architecture, the principle should be exactly the same regardless of where the aggregates reside. This is how the RPD is set up for the Fact table in my example:

The GCBC_SALES schema on Oracle holds the unaggregated sales data, whilst the EXALYTICS schema on TimesTen has an aggregate of this data in it. The very simple report pictured here shows sales by month, and additionally uses a Logical SQL view to show the contents of the query being sent to the BI Server:

Looking at nqquery.log we can see the query by default hits the TimesTen source:

[...]
------------- Sending query to database named TimesTen aggregates
WITH
SAWITH0 AS (select distinct T1528.Sale_Amoun000000AD as c1,
     T1514.Month_YYYY000000D0 as c2
from
     SA_Month0000011E T1514,
     ag_sales_month T1528
[...]

Now, for thoroughness, let’s compare this to what’s in the TimesTen database, using a Direct Database Request:

OK, all looks good. But, is what we’ve aggregated into TimesTen matching what we’ve got in the source data on Oracle? Here was can use INACTIVE_SCHEMAS to force the BI Server to ignore TimesTen entirely. We can see from the nqquery.log that OBI has now gone back to the Oracle source of the data:

[...]
------------- Sending query to database named orcl
WITH
SAWITH0 AS (select sum(T117.FCAST_SAL_AMT) as c1,
     T127.MONTH_YYYYMM as c2
from
     GCBC_SALES.TIMES T127 /* Dim_TIMES */ ,
     GCBC_SALES.SALES T117 /* Fact_SALES */
[...]

and the report shows that actually we have a problem in our data, since what’s on the source doesn’t match the aggregate:

A Direct Database Request against Oracle confirms the data we’re seeing – we have a mismatch between our source and our aggregate:

This is the kind of testing that it is crucial to perform. Without proper testing, problems may only come to light in specific reports or scenarios, because by the very nature of aggregate navigation working silently and hidden from the user.

So this is the feature we can use to perform the testing, but below I demonstrate a much more flexible way that having to build multiple reports.

Implementing INACTIVE_SCHEMAS

Using INACTIVE_SCHEMAS in your report is very simple, and doesn’t require modification to your reports. Simply use a Variable Prompt to populate INACTIVE_SCHEMAS as a Request Variable. Disable the Apply button for instantaneous switching when the value is changed.

A Request Variable will be prepended it to any logical SQL sent to the BI Server. Save this prompt in your web catalog, and add it to any dashboard on which you want to test the aggregate:

Even better, if you set the security on the dashboard prompt such that only your admins have access to it, then you could put it on all of your dashboards as a diagnostic tool and only those users with the correct privilege will even see it:

Displaying the aggregate source name in the report

So far this is all negative , in that we are specifying the data source not to use. We can examine nqquery.log etc to confirm which source was used, but it’s hardly convenient to wade through log files each time we execute the report. Ripped off from Inspired by SampleApp is this trick:

1. Add a logical column to the fact table
2. Hard code the expression for the column in each Logical Table Source
3. Bring the column through to the relevant subject area
4. Incorporate it in reports as required, for example using a Narrative View.

Bringing it all together gives us this type of diagnostic view of our reports:

Summary

There’s a variety of ways to write bespoke test reports in OBI, but what I’ve demonstrated here is a very minimal way of overlaying a test capability on top of all existing dashboards. Simply create the Request Variable dashboard prompt, set the security so only admins etc can see it, and then add it in to each dashboard page as required.

In addition, the use of a ‘data source’ logical column in a fact table tied to each LTS can help indicate further where the data seen is coming from.

This introduction of ODI brings a number of potential benefits to customers and developers and gives Oracle the opportunity to simplify the product architecture, but bear in mind that there’s no migration path from the earlier 7.9.x releases to this version, with Informatica customers instead having to wait until the “patch set 2″ version due in the next twelve months; even then, migration between tools won’t be automatic, with existing Informatica-based installations expected to stay on Informatica unless they choose to re-implement using ODI.

So how does ODI work within this new release, and how has the DAC been replaced? Let’s take a look in this final piece in our short series on Oracle BI Apps 11.1.1.7.1, starting by looking at the overall role that ODI plays in the platform architecture.

Existing ODI developers will know that the tool uses two repositories, known as the Master and Work repositories, to store details of data sources and targets, mappings, data models and other aspects of an ETL project. Within the BI Apps these two repositories are stored in a schema called prefix_ODI_REPO, for example DEV_ODI_REPO, and are accompanied by a new schema called prefix_BIACOMP, again for example DEV_BIACOMP. The BIACOMP schema contains tables used by the various new WebLogic-based BI Apps supporting applications, and contain details of the functional setup of the BI Apps, load plans that have been generated and so forth. There’s also another schema called prefix_BIACOMP_IO which is used for read-write access to the BIACOMP schema, and all of these are held in a repository database alongside the usual schemas used for OBIEE, MDS and so forth.

The major difference in using ODI within this environment is that it’s treated as an “embedded” ETL tool, so that in most circumstances you won’t need to use ODI Studio itself to kick-off load plans, monitor their execution, set up sources and targets and so forth. This was the original vision for Informatica within the original BI Apps, but Oracle are able to do this far more effectively with ODI as they own all parts of the tech stack, can alter ODI to make it easier to embed, they’e got control over ODI’s various metadata APIs and so forth. What this means in practice is that the setup of the ODI topology (to connect to the ERP sources, and the target data warehouse) is done for you via a web-based application called the Oracle BI Applications Configuration Manager, and you can kick-off and then monitor your running ETL jobs from Configuration Manager and from ODI Console, the web-based operator tool that’s been around since the 11g release of ODI. The screenshot below shows Configuration Manager setting up the source database ODI topology entry, with the details that you provide then being pushed through to the ODI master repository:

Setting up a new BI Apps system involves using the Configuration Manager to define the connections through to the various source systems, then select the BI Apps modules (Financial Analytics, for example, and then the various subject areas within it) that you wish to implement. There are then a number of steps you can perform to set up system-wide settings, for example to select default currencies or languages, and then you come to run your first ODI load plan – which in this instance copies settings from your source system into the relevant tables in the BIACOMP schema, performing automatically the task that you had to do via the various domain configuration spreadsheets in the earlier 7.9.x releases – the screenshot below shows this ODI load plan listed out and having run successfully.

You can then view the execution steps and outcome either in ODI Console (embedded within Configuration Manager), or over at ODI Studio, using the Operator navigator.

Moving over to ODI Studio, the folders (or “adapters”) that in Informatica used to hold workflows and mappings for the various source systems, are contained with the BI Apps project within the Work repository and the Designer navigator. In the screenshot below you can also see the Fusion Apps adapter that’s not supported in this particular release, and the ETL Data Lineage adapter that should get enabled in an upcoming patch release.

In the screenshot above you can also see one of the loading tasks, SDE_ORA_APAgingBucketsDimenson, is a package that (were you to expand the Interfaces entry) makes reference to a regular, and also a temporary, interface.

Packages in ODI perform the same role as Informatica workflows in earlier releases of the BI Apps, and each package runs some steps to refresh variables, work out if its doing a full or incremental load, and then call the relevant ODI interface. Interfaces in ODI for the BI Apps typically load from other temporary interfaces, with these temporary interfaces performing the role of maplets in the Informatica version of the BI Apps, as you can see in the screenshot on the left below. On the right, you can see the flow for another mapping, along with one of the custom KMs that come as part of the BI Apps 11.1.1.7.1 package.

Individual packages are then assembled into the equivalent of BI Apps 7.9.x “execution plans” through a new JEE application called the Load Plan Generator, which also gets installed into ODI Studio as a plug-in so you can develop new data loading routines away from the full production setup. As you can see in the final screenshot below, these load plans are then visible from within ODI Studio (whether you generated them there, or from Configuration Manager), and like all ODI 11g load plans you can view the outcome of each load plan instance run, restart it if this feature is enabled, and so forth.

So there you have it – how ODI is used within the BI Apps 11.1.1.7.1. I’m going to take a break now as it’s almost time for the Atlanta run of the Rittman Mead BI Forum 2013, but once I’m back in the UK I’ll try and put something together for the blog on pulling together your first ETL run. Until then – have fun with the release.

Wednesday starts with the optional one-day masterclass, this year on Oracle Data Integration and led by myself, Stewart Bryson and Michael Rainey. I previewed the data integration masterclass previously on the blog, and the planned timetable for the masterclass looks like this:

Day 1 : Optional Oracle Data Integration Masterclass, followed by Registration, Drinks and Keynote/Meal

10.00 – 11.00 : Welcome, and Introduction to Oracle Data Integrator 11g (Stewart Bryson)
11.00 – 11.15 : Morning Coffee
11.15 – 11.45 : ODI and the Oracle Reference Architecture for Information Management (Stewart Bryson)
11.45 – 12.45 : ODI and GoldenGate – A Perfect Match… (Michael Rainey)
12.45 – 13.30 : Lunch
13.30 – 14.30 : ODI and Hadoop, MapReduce and Big Data Sources (Mark Rittman)
14.30 – 15.30 : The Three R’s of ODI Fault Tolerance : Resuming, Restarting and Restoring (Stewart Bryson)
15.30 – 16.30 : Scripting and Automating ODI using Groovy and the ODI SDK (Michael Rainey)

The event itself officially opens at 4pm on Wednesday, May 15th 2013 with registration taking place then, and a drinks reception in the hotel bar from 5pm to 6pm. At 6pm we have the Oracle keynote led by Jack Berkowitz and Philippe Lions, and then an informal meal in the hotel restaurant from 7pm – 10pm.

The main conference then opens at 8am on the Thursday morning, with registration open from 8am – 8.45am, opening remarks from myself at 8.45am and the first session starting at 9am. Here’s the timetable as planned for Thursday:

Day 2 : Main Conference Sessions, Guest Speaker and Gala Meal

8.45am – 9.00am : Opening Remarks Mark Rittman, Rittman Mead
9.00am – 10.am : Rene Kuipers, VX Company, “It’s all in the genes – The power of Oracle Exadata and the Oracle Database”
10.00am – 10.30am : Morning coffee
10.30am – 11.30am : Jack Berkowitz, Oracle : “OBI Presentation, Interaction and Mobility”
11.30am – 12.30am : Venkatakrishnan J, Rittman Mead, “In Memory Analytics – Times Ten, Essbase 11.1.2.2 – Analysis – A Comparison”
12.30pm – 1.15pm : Lunch
1.15pm – 1.30pm : TED Session 1 : Kevin McGinley – “OBIEE and OEID: What if…?”
1.30pm – 1.45pm : TED Session 2 : Jon Mead, Rittman Mead, “Why I want to be working with Business Intelligence in 5 years time”
1.45pm – 2.00pm : TED Session 3 : Jeremy Harms – “A BI Publisher Beginner’s MacGyver-Hack for Financial Reporting with OBIEE: A Quickie!”
2.15pm – 3.15pm : Alan Lee, Oracle, “Update on BI Metadata Architecture and Design Tool”
3.15pm – 3.45pm : Afternoon coffee and beers
3.45pm – 4.45pm : Jeff McQuigg, KPI Partners Inc, “Performance Tuning the BI Apps with a Performance Layer”

After the first day’s presentations we’ll take a short break, and then convene again back in the conference room at 5pm for our special guest speaker session, this year being provided by Method R’s Cary Millsap, who many of you will know from his Optimizing Oracle Performance book and his “response time” approach to performance tuning. Just after Cary’s session at around 6.30pm we’ll then be taken by coach to “4th and Swift”, the venue for the gala meal, where we’ll be from around 7pm through to around 10pm.

In this posting then, I’m going to take a closer look at this new product architecture, and I’ll follow it with a more detailed look at how the various bits of ODI functionality replace the workflows, mappings, transformation operators and execution plans provided in earlier releases by Informatica and the DAC. For anyone familiar with the previous, 7.9.x versions of the BI Applications, the architecture diagram below shows the five tiers that this product typically implemented; tiers for the source data and data warehouse/repository databases, an ETL tier for Informatica and the DAC server, then two more tiers for the OBIEE application server and the client web browser.

Communication between the tiers was – to put it politely – “loosely coupled”, with DAC task names corresponding with Informatica workflow names, each workflow containing a single mapping, and all of the connections and sources having to be named “just so”, so that every part of the stack could communicate with all the others. It worked, but it was a lot of work to implement and configure, and once it was up and running in most cases customers were scared to then change it, in case a name or a connection got out of sync and everything then stopped working. Plus – Informatica skills are scarce in the Oracle world, and the DAC is an extra piece of technology that few DBAs really understood properly.

The 11.1.1.7.1 release of the BI Apps simplifies this architecture by removing the separate ETL tier, and instead using Oracle Data Integrator as the embedded ETL tool, with its server functions running as JEE applications within the same WebLogic domain as OBIEE 11g, giving us the overall architecture in the diagram below.

Now anyone who read my series of posts back in 2009 on the 7.9.5.2 release of the BI Apps, which also used ODI as the embedded ETL tool, will know that whilst ODI 10g could do the job of loading data into the BI Apps data warehouse, it lacked the load orchestration capabilities of Informatica and the DAC and wasn’t really set up to dynamically generate what have become, in ODI 11g, load plans. BI Apps 7.9.5.2 turned-out to be a one-off release and in the intervening years Oracle have added the aforementioned load plans along with other functionality aimed at better supporting the BI Apps, along with two new JEE applications that run in WebLogic to replace the old DAC.  These new applications, along with the ODI JEE agent, ODI Console and the ODI SDK, are shown in the more detailed BI Applications 11.1.1.7.1 logical architecture diagram shown below.

Oracle BI Applications 11.1.1.7.1 has two main product tiers to it, made up of the following components:

• The Middleware (BI and ETL) tier; a WebLogic domain and associated system components, comprising BI components delivered as part of OBIEE 11.1.1.7 (including Essbase and related applications) as one managed server, and another managed server containing ODI Java components, including three new BI Apps-related ones; Configuration Manager, Functional Setup Manager, and ODI Load Plan Generator
• The Database (DW and Repositories) tier; for the time-being, Oracle only, and comprising a data warehouse schema (staging + performance layer), and a repository database containing the OBIEE repository schemas plus new ones to hold the ODI repository and other ETL/configuration metadata used for configuring your system.

Essbase at this stage is installed, but not used for the main BI applications, and all of it uses Fusion Middleware security (application roles and policies) along with the WebLogic Embedded LDAP server to hold users and groups. A special version of RCU is used to set up the new BI Apps-related schemas, and import data into them using Oracle database export files, so that the ODI repository, metadata tables and so forth are all populated prior to the first load taking place. Enterprise Manager Fusion Middleware Control is still used to manage and monitor the overall platform, but there’s now an entry for ODI along with Essbase, the latter of course being delivered as part of the 11.1.1.7 OBIEE platform release.

In the next posting in the series we’ll take a closer look at how ODI uses its JEE agent and mappings imported into its repository to load the BI Apps data warehouse, but what about the two new web-based configuration tools, Oracle BI Applications Configuration Manager (BIACM) and Oracle BI Applications Functional Setup Manager (FSM) – what do they do?

After you install OBIEE 11.1.1.7 and then the BI Applications 11.1.1.7.1, the BI Apps installer extends the BI domain to include FSM, BIACM and the ODI Load Plan Generator, along with some other supporting applications and libraries required for the full product. Load Plan Generator works behind the scenes to build new load plans in a similar way to the Execution Plan “Build” feature in the DAC, and the two web-based tools perform the following functions:

• Oracle BI Applications Configuration Manager performs system-wide setup tasks such as defining sources, selecting BI Apps modules and performing other, “one-only” tasks similar to the Setup feature in the DAC Console.
• Oracle BI Applications Functional Setup Manager is then used to list out, and track progress against, the various tasks required to configure the BI Applications modules, or “Offerings”, that you selected in the Configuration Manager

Most importantly though, these tools connect directly through to the ODI repository, so data sources you set up here will get pushed down to ODI as data servers in the ODI master repository; load plans you set up to, as in the screenshot below, load configuration tables, are ODI load plans and you can track their progress either from within ODI, or from within these applications themselves.

I haven’t had a chance to properly “diff” the RPD used in BI Apps 11.1.1.7.1 with the previous 7.9.x ones, or do a similar exercise for the underlying database data model, but on first glance the new RPD is at least recognisable, albeit with new sources and subject areas for the Fusion Apps, Oracle Transactional BI (OTBI), Real-Time Decisions and the like. The web catalog also looks familiar, but also has new content around the new applications along with additional content for the existing ones.

So, we’re at the point now where can start to think about loading data into the BI Apps data warehouse, and in tomorrows post we’ll take a look at what’s involved in a BI Apps 11.1.1.7.1 ETL load, and also look into how GoldenGate can now be used to extract and stage data prior to loading via ODI. Back tomorrow…

Although the focus in BI Apps 11.1.1.7.1 is on ODI as the ETL tool, this new release actually delivers a whole new product architecture along with new dashboards, new content, and a new security framework. In addition, there’s now an option to use Oracle GoldenGate to create a new layer in the BI Apps data warehouse data architecture that replicates source data into the warehouse environment, giving you the ability to run the more large-scale ETL processes when you like, rather than when there’s an ETL window for the source systems.

Let’s start off though with a summary of what’s new from a functional perspective, and also what limitations there are for this first release in terms of sources, scope and so forth. BI Apps 11.1.1.7.1 delivers the following set of new features and capabilities:

• Oracle Data Integrator as the embedded ETL tool, along with a whole new FMW11g-centric architecture and set of utilities
• Two new analytic applications - Student Information Analytics, and Indirect Spend Planning
• New content for existing analytic applications including Financial Analytics, HR, Projects, CRM and Procurement & Spend
• Dashboards that are now written for OBIEE 11g rather than 10g, including 11.1.1.7 visualisations such as performance tiles

Now although, in general terms, BI Apps 11.1.1.7.1 covers all (or most…) of the existing analytic application modules along with all of the 7.9.x-era sources (EBS, PeopleSoft, JDE and Siebel), there are some important restrictions that you’ll need to be aware of when making any plans to use this new release, starting with upgrade paths (or lack of them):

• There’s no automatic upgrade path from BI Apps 7.9.x, and no automated migration routine to take you from Informatica to ODI; if you want BI Apps 11.1.1.7.1 now, you’ll have to reimplement rather than upgrade, or you can wait for BI Apps 11.1 PS2 which will support upgrades from earlier releases, but (important to note) keeps you on Informatica – any move from Informatica to ODI will need to be done yourself, as a re-implementation
• Only Oracle database sources and targets are supported in this initial release, in practice not a real issue for new implementations, but worth bearing in mind if you planned to use Teradata, for example, as your target data warehouse platform
• Oracle Fusion Applications aren’t supported as a source either, yet, so anyone using this will need to stay on BI Apps 11.1.1.6.x until an upgrade version becomes available
• A few edge-case analytic applications and sources aren’t supported in this release yet – Enterprise Asset Management, for example, is not yet supported for any source, whereas some other applications only support more recent PeopleSoft versions and not JDE, for example. As always, get the most up-to-date supported sources and applications list from Oracle before making any major investment in an implementation or upgrade project.

From a technical perspective though the major difference in this release, compared to the 7.9.6.x versions that preceded it, is the use of Oracle Data Integrator 11g as the embedded ETL tool rather than Informatica. To be clear, Informatica will still be supported as an ETL option for the BI Apps going well into the foreseeable future, but Informatica users will need to wait for the PS2 release due in the next twelve months or so before they can upgrade to the new 11g platform.

In addition and perhaps more importantly, it’s not envisaged that Informatica customer will move over to ODI unless they use the upgrade as an opportunity to re-implement their system now on ODI, moving across customisations themselves and essentially starting with a clean sheet of paper (which may not be a bad thing, if you’re thinking of tidying up your system following years of upgrades, customisations and so forth). What this does mean though is no DAC, no Informatica server and client tools, a new (and hopefully simpler) way of setting up and configuring your system, and in-theory a more closely-integrated set of tools all based around the modern, standards-based Fusion Middleware 11g architecture. 

In this new world of ODI and the BI Apps, ODI load plans replace Informatica Workflows, whilst ODI packages and interfaces equate to Informatica mappings and maplets. The DAC is no more and is replaced by metadata within the ODI repository and other supporting schemas, with setup and configuration of the warehouse and ETL processes now carried out by two web-based tools, BI Applications Configuration Manager and Functional Setup Manager. The closer integration between these tool along with a chance for Oracle to re-think the BI Apps setup process should lead to easier configuration and customisations, but if you’re an Informatica developer it’s a whole new world, and the 11g platform makes a lot more use of Fusion Middleware platform functionality particularly around security and user provisioning.

So – all very exciting but quite daunting in terms of what needs to be learnt, and new processes that need to be thought through and put together before you can start making use of the new 11.1.1.7.1 feature set. We’ll start tomorrow then by taking a closer look at the BI Apps 11.1.1.7.1 technical architecture including the new configuration tools, and where ODI sits in the new product architecture, based on our first impressions of the product.

Registration for the Brighton BI Forum officially opens at 5pm on Wednesday evening, but before that we’re running an optional one-day Oracle Data Integration Masterclass at the hotel venue led by myself, Stewart Bryson and Michael Rainey. The masterclass will start at 10am and go through until about 4.30pm, with the following estimated timings:

Day 1 : Optional Oracle Data Integration Masterclass, followed by Registration, Drinks and Keynote/Meal

10.00 – 11.00 : Welcome, and Introduction to Oracle Data Integrator 11g (Stewart Bryson)
11.00 – 11.15 : Morning Coffee 
11.15 – 11.45 : ODI and the Oracle Reference Architecture for Information Management (Stewart Bryson)
11.45 – 12.45 : ODI and GoldenGate – A Perfect Match… (Michael Rainey)
12.45 – 13.30 : Lunch
13.30 – 14.30 : ODI and Hadoop, MapReduce and Big Data Sources (Mark Rittman)
14.30 – 15.30 : The Three R’s of ODI Fault Tolerance : Resuming, Restarting and Restoring (Stewart Bryson)
15.30 – 16.30 : Scripting and Automating ODI using Groovy and the ODI SDK (Michael Rainey)

Registration will then open at 5pm (17.00), with a drinks reception in the hotel bar between 6pm and 7pm (18.00 – 19.00), followed by the Oracle keynote and opening night meal in the hotel restaurant.

The main conference then opens at 8.45am on the Thursday morning, with the agenda for the first day looking like this:

Day 2 : Main Conference Sessions, Guest Speaker and Gala Meal

08.45 – 09.00 : Opening Remarks Mark Rittman, Rittman Mead
09.00 – 10.00 : “OBIEE SampleApp 11.1.1.7 functional highlights” (Philippe Lions, Oracle Corporation)
10.00 – 10.30 : Morning Coffee
10.30 – 11.30 : “OBI Performance Tuning – Real Customer Success Stories” (Antony Heljula, Peak Indicators Ltd)
11.30 – 12.30 : “Secrets of OBIEE implementation at LGI” (Marco Klaassens, Liberty Global)
12.30 – 13.15 : Lunch
13.15 – 13.30 : TED Session 1:  ”Why I want to be working with Business Intelligence in 5 years time” (Jon Mead, Rittman Mead)
13.30 – 13.45 : TED Session 2 : “HA, DR, CFC, WTF?” (Mike Durran, Oracle Corporation)
13.45 – 14.00 : TED Session 3 : “Incrementally loading Exalytics using Notepad” (Antony Heljula, Peak Indicators Ltd)
14.15 – 15.15 : “Oracle Data Integrator 11g Best Practices. Busting your performance, deployment, and scheduling headaches.” (Uli Bethke/Maciek Kocon, Independent)
15.15 – 15.45 : Afternoon Tea/Coffee/Beers
15.45 – 16.45 : “New Developments in BI Multi-tenancy and Cloud” (Adam Bloom, Oracle Corporation)

After the first day’s presentations we’ll take a short break, and then convene again back in the conference room for our special guest speaker session, this year being provided by Toby Potter from Datasift, who’ll talk to us about social media and how it can be used within BI and analytic applications. Then, we’ll be picked-up by coach from outside the Hotel Seattle and taken to the venue for our gala meal, The Ginger Pig pub and restaurant in Hove, just down the road from where Jon and I live (though don’t all expect to come back afterwards, whatever Borkur and Ragnar tell you when you leave the pub.)

18:00 – 18.45 Guest Keynote Toby Potter, Datasift – “Social Data and Business Intelligence”
19:00 – 19.30 Depart for Restaurant 
20:00 – 22:00 Gala Meal : The Ginger Pig, Hove

Don’t go too crazy on Thursday evening though, as we’re starting again early on Friday with sessions starting at 9am, going through until around 4.45 when we’ll close. Here’s Friday’s agenda:

Day 3 : Main Conference Sessions, and Close

9.00 – 10.00 :  “The magic of aggregates” (Michael Wilcke, sumIT AG)
10.00 – 10.30 : Morning Coffee
10.30 – 11.30 : “Update on BI Metadata Modeler and Metadata Architecture” (Philippe Lions, Oracle)
11.30 – 12.30 : “Integrating Oracle BI, BPM and BAM 11g: The complete cycle of information” (Edelweiss Kammermann, Awen Consulting)
12.30 – 13.15 : Lunch
13.15 – 14.15 : Debate “Big Data – Hype, or the Future or Oracle BI/DW?”
14.15 – 15.15 : “Endeca – Looking beyond the general demos” (Adam Seed, Rittman Mead)
15.15 – 15.30 : Afternoon Tea/Coffee/Beers
15.30 – 16.30 : “Virtualizing Exalytics” (Mike Durran, Oracle Corporation)
16.30 – 16.45 : Closing Remarks (Mark Rittman)

You’ll notice we’ve brought back the popular “debate” section this year, with this year’s topic being “Big Data – Hype, or the Future of BI/DW?”. I’ll be looking for volunteers to argue the case for either of the two sides in the debate, so if you’ve got a view on whether big data is going to be the salvation of BI, whether it’ll turn us in to the COBOL programmers of the future, or whether its just a load of hot air (or you just like having an argument), let me know when you arrive and we’ll pull the debating teams together.

Other than that – have a safe journey over, bring something warm and waterproof as the weather is typically England in May, and see at least some of you in Brighton today and tomorrow! 

The masterclass is made up of six sections, delivered by myself, Stewart and Michael, assumes a basic understanding of data warehousing and ETL tools but otherwise gets down into the detail of what we’ve found works well “in the field”.  Stewart Bryson, Oracle ACE and Managing Director for Rittman Mead America, will open the session with an overview of ODI and the Oracle Data Integration Suite, taking a look at the product history and walking the audience through the major elements of the ODI product architecture. If you’ve ever wondered what agents do within ODI, why there are two repositories and where WebLogic comes into it, Stewart’s session will make everything clear before we get into the rest of the details.

Then, after coffee, Stewart will carry on and talk about what’s called the Oracle Information Management Reference Architecture, Oracle’s next-generation blueprint for data warehousing and information management that combines the best of Kimball and Inmon with new thinking around “big data” and “data discovery”. ODI and Oracle Data Integration Suite is the enabling technology for this new framework. At Rittman Mead, we use this framework for the majority of our DW customer engagements and we’ll be talking later on in the masterclass about how big data sources, for example, can be leveraged by ODI and brought into your BI environment in the same way as any other regular, relational datasource.

The third section of the masterclass sees Michael Rainey take over the stage and talk to us about ODI’s integration with Oracle GoldenGate, Oracle’s data integration product for real-time analysis and data loading. Michael has taken part in several ODI & GoldenGate customer engagements over in the States, and has worked with Stewart in producing a number of custom ODI knowledge modules to better make use of this powerful new data integration tool. If you’ve read through any of Michael’s blog posts on ODI and Golden Gate and are interested in hearing a bit more detail on how it all works, as well as some real-world practical tips and tricks, this will be an invaluable session for you.

So far I’ve got away with just making the tea, but straight-after Michael is my session, where I’ll be talking about ODI and its new integration with Hadoop, NoSQL and the wider “big data” technology area. I’ve been covering ODI and Hadoop in some blog posts over the past week, but there’s only so much that I can get into a blog post and this session will be the first airing of this new material, where I’ll be demoing all the main integration points and talking about what works well, and where the main value is, with this very interesting new feature.

Then it’s back to Stewart again, where he’ll be talking about creating highly-resilient ETL code that’s also resumable, using features such as ODI 11g’s load plans and the Oracle Database’s resumable space allocation feature. Stewart and I were particularly keen to put together this session as it brings together work Stewart did a few years ago on fault-tolerant ETL in the Oracle Database, with some blog posts I put together over the 2012 Christmas break around highly-resilient ETL with ODI11g.  What this session does is explain the background to the ETL resilience features in the Oracle Database, and ODI’s use of WebLogic JEE agents, and demonstrates through some custom knowledge modules how they can be brought together for your project.

Finally, Michael concludes the masterclass with a look at a feature you’re probably vaguely aware of, intend to learn something about, but sounds a bit complex; Groovy scripting and the ODI SDK. In fact, like WLST scripting for OBIEE, learning Groovy and the SDK is the key to automating tedious tasks such as mass-importing and reverse-engineering tables and files, as well as making it possible to add functionality to ODI or integrate it with other standards-based products. In a session almost entirely made-up of live demos, Michael will take us through the basics of Groovy and the SDK, and show us a few examples of where this could add value to your data integration projects.

So there we have it – Brighton is now fully-booked up, but if you’ve already registered for the main event but want to come to the masterclass now too, you can log back into the registration site and update your booking to include the additional masterclass fee. Atlanta is running a week later and so still has a few main event passes left, and again if you’ve already registered for the main conference, use the link in your registration confirmation to go back in and add the masterclass to your booking. And – hopefully we’ll see you all in Brighton or Atlanta for the Rittman Mead BI Forum 2013 in the next two weeks!