Alerting end-users about a completed ETL task is a pretty common requirement. ODI 11g (or for that matter any ETL tool) has native routines to send emails directly from within the tool. But in the case of the specific customer case, they wanted to use BI EE 11g Agents for 3 main reasons

1. Use the features of BI EE – like along with the ETL task completion status they wanted a report or a set of reports to be distributed to the end users

2. They wanted dashboard based alerting for Administration users. Whenever an Admin user logs into the BI EE dashboard, they wanted the ability to pop-up a message denoting the completion status.

3. They have a bunch of BI EE reports built already on top of the ODI metadata which will capture information on number of failed records loaded etc (directly from the E$ and C$ tables).

So, it was pretty clear that using BI EE 11g scheduler is the way to go. But how do we call BI EE agents from ODI. There is no native integration between the two (perhaps a candidate for an enhancement request as i think this will be a pretty common requirement once BI Apps using ODI becomes available for all the modules). So the only option is to build our own integration. There are basically 2 ways of achieving this

1. Use SASCHINVOKE utility available for calling Agents from command line and then call the SASCHINVOKE utility from within ODI – This requires an ODI Agent to available on the same box as BI EE. A quick overview of the solution is given below

2. Use BI EE Web Services and call ExecuteIbotNow method in the iBotService – This method will be called directly from ODI 11g. The high level architecture diagram is given below

Lets see each of the above methods in detail now. We will start off with the first method i.e. using SAWSCHED utility.

SASCHINVOKE & Native ODI Agents:

This method assumes that there is a physical remote ODI agent running on the same box as BI EE 11g. For this to work, we will have to call the SASCHINVOKE utility from within a ODI package. For example, lets say we have an iBot in BI EE that is supposed to deliver a report/dashboard after the ETL job gets completed. First we start off with creating a package in ODI and then use the ODIOSCommand to call the SASCHINVOKE command line utility as shown below

Very Important: For this to work you need to have SSL enabled. According to note 1331931.1 in MOS, the ability to pass passwords in command line is now disabled(for security reasons). So, we cannot use this option if we do not have SSL enabled.

As you see in the above screenshot, SASCHINVOKE will always prompt for a password. To bypass this, just add an ECHO statement and pipe the output to the SASCHINVOKE command as shown below.

BI EE Web Services & ODI Packages:

In many cases, the first method above will not work because we might not have SSL enabled or we might not prefer passing the username & password in command line. One more drawback with the first one is, we have to host a separate ODI agent on the BI EE Machine. Of course we can work around it by just installing BI EE client on the ODI Agent machine (which will have SASCHINVOKE in the binary – and it can do remove invocations). But the problem with that is in pure Unix environments, we do not have the option of installing BI EE Client on the ODI Agent machine(only option is to setup a SSH public-private key and enable remote invocation). In such cases, we will have to rely on the Web Service API calls of BI Scheduler. Lets try to quickly understand how that works.

ODI supports native Web Service calls. It can either extract data (Through a Data Service) or can execute a remote service method. But the problem with BI EE web services is the fact that they are multiple-end point in nature. That is, we will have to first make a call to the SAWSession web service which will create a SessionID. This sessionid then needs to be passed on to other services like iBotService. To do that we need to do 3 main things within the ODI framework

1. Using ODIInvokeWebService, make a call to the SAWSession service. This will give a XML response with the SessionID embedded in the XML.

2. Use a java class (or using ODIXMLSplit & ODIXMLConcat) to extract the SessionID from the response XML and then create a Request XML with the SessionID and the path of the iBot embedded.

3. Using ODIInvokeWebService, make a call to the executeiBotNow method of the iBotService endpoint using the request XML generated above.

So, we start off with creating an ODI Package. We then call the ODIInvokeWebService component to make a call to the SAWSession service.

On execution of this service, the output response file will get stored in a file with the SessionID.

The next step is to extract the SessionID from the SOAP response and then generate a Request XML as shown below

To do this, i have attached a zip file here that contains the code for extracting the sessionID and converting it into a SOAP Request. This code basically uses the open source DOM4J API. I prefer Java Code instead of using ODIXMLCONCAT & ODIXMLSPLIT as both of them seem to have some known bugs. The snippet of the main code i have written for this conversion is given below

The code accepts 3 parameters(basically the runodiagent.bat file). Input SOAP Response file from the first Web Service Call. BI EE iBot full path that we intend to call. And the output file that will be generated, which will in turn act as a Request XML to the next Web Service Call. All the parameters will be called through the ODIOSCommand component as shown below

Once this file is generated, then this file is passed as an input to the next ODIInvokeWebService call as shown below

The final package will look something like this

On execution we should start getting the alerts right within the Dashboard (or through emails – whichever is setup in the iBot) after the ETL-ODI execution

From an OBIEE perspective, we started 2011 with the 11.1.1.3 release on general availability, and the 11.1.1.5 release made available about mid-way through the year. In hindsight, 11.1.1.3 was a bit of an “early adopter” release, with all the main functionality working but a few hassles around installation, upgrades and “fit and finish” issues. Many of our customers and partners adopted OBIEE 11g at this point, but quite a few held back for 11.1.1.5 which is generally considered a “fully-working” version (albeit with the usual bugs, issues etc that you get with any piece of enterprise software). Looking back at 2011, probably the biggest drivers for 11g adoption were Mobile (particularly the iPad client), the new Answers and Dashboards UI, and management via Enterprise Manager and WLST scripting. Without much fanfare, the Action Framework has also been widely adopted, but more for adding “missing functionality” such as Essbase writeback rather than integration with web services and workflows. Something I hear is going down well but don’t see much evidence in the field is Scorecard & Strategy Management, though I would expect to see more of this as Oracle push “Oracle BI Foundation” as the base license deal rather than “Oracle Business Intelligence Enterprise Edition”, as this will remove the price premium for scorecards (and mobile) and remove one of the hurdles for adoption.

As well as OBIEE 11g releases, we saw the 11.1.1.5 release of Oracle Data Integrator (review here, here and here), and confirmation that OWB is now at the end-of-the-line, along with Discoverer, with only maintenance releases coming along in future. Almost every new ETL project I see now is based on ODI (rather than OWB) so in the field, this was a de-facto change anyway, though it’ll be interesting too see what options emerge for OWB customers looking to migrate to ODI – keep an eye on this blog as we’ll be running a seminar in London in February on just this topic.

2011 was also the year we ran our third-ever Rittman Mead BI Forum, running again in Brighton and also this time, in Atlanta in America. I was very pleased to be joined by Tony Heljula for the one-day Masterclass before each event, where we talked about OBIEE 11g topics such as architecture, data modeling, SOA integration and Spatial Integration; Tony’s a great speaker and it was fun to collaborate on the topics and deliver them together over the two locations. Next year, Kevin McGinley from Accenture is delivering the masterclasses, and he’ll be known to our US readers as winner of the “Best Speaker” award at last year’s US BI Forum, and “Best Paper” award at ODTUG KScope’11. The call for papers for next year’s BI Forum is now open, and running until the end of January, so if you want to propose a session for either the UK, US or both events, take a look at the announcement page now!

Rittman Mead were certainly out on our travels in 2011, with our OBIEE 11g Training Days events running in the first half of the year in Atlanta (twice), Bangalore (Twice), London, Johnannesburg and Brussels, and our team speaking at events such as Collaborate’11 in Orlando, UKOUG 2011 in Birmingham, ODTUG KScope’11 in Long Beach, Oracle Open World in San Francisco, ODTUG BI/EPM Symposium in Sydney, AIOUG Sangam’11 in Bangalore, DW Global Leaders’ events in Athens, Rome and San Francisco, and plenty of others I’ve probably forgotten.

We encourage all of our staff to speak at conferences and our policy is that, if you get a paper accepted anywhere, we’ll give you the time-off to speak and cover your expenses. Thanks also to the Oracle ACE and Oracle ACE Director Program and to Vikki, Lillian and Justin at the Oracle Technology Network, who’ve funded a lot of my speaking engagements on behalf of the ACE Director program, and who provide complementary Open World attendee passes for myself as an ACE Director and our Oracle ACEs, Stewart Bryson and Venkatakrishnan J.

It’s also been a year where many of our staff have started to gain a name for themselves as specialists in certain areas. Stewart Bryson, for example, has been speaking and writing all year about real-time BI, and the new development paradigm that you can adopt when combining Exadata, OBIEE and Agile methodologies, whilst Venkat has been building on his reputation as the “go-to” expert for OBIEE and Hyperion Integration. Borkur has spoken at several conferences on Oracle Golden Gate, whilst Mike Vickers is getting a name for himself as a speaker on BI methodologies. Behind all these experts though is our team of BI, DW and EPM enthusiasts, taking this message and their own particular skill-sets to our customers around the world, sharing what we’ve learned and hopefully “raising the bar” for customer implementations.

For Rittman Mead as a company, it’s been an exciting year with lots of growth. We’ve now got offices in the UK, US, India, Brussels and Australia, new training centres in Brighton and Bangalore, and we’ve been winning awards such as the UKOUG BI Partner of the Year award 2011/12. Thanks again to all of our customers, partners and staff, and I’m looking forward to getting out again and visiting our various country offices in 2012.

So what about 2012? What are we likely to see, both from Oracle, from the industry in general, and from Rittman Mead? Well, from an OBIEE perspective, we’ll most likely see the 11.1.1.6 release fairly early in 2012, with probably a minor release (11.1.1.6.1?) sometime later in 2012, going on how Oracle do OBIEE releases at the moment (one major, one minor, each year). Going on from 11.1.1.6, there’s some general themes that Oracle are working to with OBIEE:

• Making Mobile a first-class client – perhaps adding support for Android, re-vamping the UI, maybe making it possible to author reports from mobile, maybe even re-thinking the UI so it’s not just a dashboard on an iPad, but maybe delivering BI in different ways that are perhaps more suitable to a mobile device
• Developer productivity – making it simpler/quicker to develop ETL from a logical model, and to process customizations in source systems, plus better support for team working, version control, more granular metadata stored in middleware repositories, and hopefully running end-to-end from ETL through to the front-end reports
• Appliances – Exalytics will be out in early 2012, combining hardware + software into a “it just works” appliance that also harnesses the “Exa-” engineered systems approach.
• Building out BI innovation – 11g so far has been about Fusion Middleware integration, and supporting the Fusion Apps. Hopefully we’ll see more pure-BI innovation in future releases, to enable OBIEE to take on some of the specialist, niche vendors such as Qlikview and Tibco (Spotfire) whilst providing enterprise features that are Oracle’s traditional strength
• Steady improvement in Hyperion Integration – I think it’s fair to say that Essbase/Hyperion integration for OBIEE is still a “work in progress” so we’re looking forward to seeing improved Essbase integration, metadata access support for Planning, integration of Essbase/HFM with BI Apps, and some replacement solution for integrating the OBIEE dashboard with Hyperion Workspace

There are also some general industry trends that we’re also very excited about for 2012. Real-Time BI (in it’s many forms) is becoming more and more “the expected norm”, but there are lots of challenges in terms of how we do data movement/ETL, how we present a consistent set of numbers to users, how we may meaningful decisions based on data that’s come in a few seconds ago, and how do we present what can potentially be large amounts of sensor-driven, machine-captured “big data” to users without them becoming overwhelmed by data?

Unstructured data will most certainly be something Oracle will be talking about a lot in 2012 following the acquisition of Endeca, with my prediction being that a future version of the OBIEE 11g BI Server will support Endeca as a data source (but how will Endeca’s data be modelled as a star schema?), and Endeca Latitude front-end features being gradually incorporated into OBIEE 11g’s front-end, in the same way that Hyperion WebAnalysis and Oracle Discoverer features have made their way into the 11g release. I’m not sure Endeca / Unstructured data will ever be “mass market”, but it’s an obvious acquisition for Oracle and I’ll be looking forward to seeing unstructured data features making their way into Oracle’s BI platform.

Another area we’ll be focusing on in 2012 will be in-memory BI. Exalytics is just around the corner, and the new version of TimesTen that it’ll ship with, plus the in-memory version of Essbase, should make split-second, lightning-quick analysis of large sets of data a possibility. The biggest barrier I see to user adoption (well, apart from data quality) is slow queries on dashboards, so Exalytics’s in-memory databases plus the 1TB of RAM it’ll ship with will be definitely welcome. But how well will in-memory combine with real-time, and how will the TimesTen in-memory database perform compared to Exadata, OLAP + materialized views, or even the file-based results cache that OBIEE already comes with? Check out our presentations at RMOUG Training Days in February, and the Hotsos Symposium in March, for the results of our testing.

The BI Apps is also an area that’s due for a lot of changes in 2012. BI Apps 11g is already out, but you’d be forgiven for not noticing, as up until recently it’s been on controlled release, and it still only covers the Fusion Applications as a data source, with support for Apps Unlimited (EBS, Siebel, PSFT etc) coming in the next 12 months or so. We should also start to see some innovation in the product itself, with closer integration with OBIEE in terms of pushing through customizations, integration with Essbase and HFM, and general reduction in the workload in terms of upgrades, data loading and team development. Probably for most existing BI Apps 7.9.x + Informatica customers, not much will be happening in the next 12 months, but keep an eye on product announcements and expect lots of activity in 2013.

Cloud-based BI is a particular interest of the company and in particular Jon Mead who presented on the topic at Open World last September, based on trends towards cloud-based applications in the industry and demand from customers to simplify their systems and reduce their costs. We’ll be introducing an “Exalytics in the Cloud” option in 2012, offering customers the ability to access our cloud-based Exalytics machine and replicate, via GoldenGate, their data into our hosted, secure environment and have us manage their BI system. As always, the challenges with BI in the cloud are firstly, replicating large amounts of data securely into the cloud, and secondly, trusting a vendor to manage your data for you, but ETL technology and bandwidth are making the first less of an issue, and for the second, a proper “best practices”, always-patched, 24×7 monitored remote system is often more secure and better managed than something in-house. Look out for lots on this from us over the next 12 months.

We’ll be speaking at lots of events in 2012, including RMOUG Training Days in Denver in February, Hotsos Symposium in Dallas in March, Collaborate’12 in Las Vegas in April and ODTUG KScope’12 in San Antonio in June. Keep an eye on our Events page for dates as they are announced, and also on our Public Scheduled Training Events page for details of OBIEE 11g, ODI 11g and BI Apps 7.9.6.3 courses running regularly at our training centres in Brighton, Atlanta, Bangalore and Melbourne.

Finally – at last, 2012 will be the year that my book is released. I’ve had some excellent support and co-operation from the Oracle Product Development team over the past few months, and so the book will be bang-up-to-date when it comes out, covering 11.1.1.6.x and Exalytics and including all the new visualizations, team development features, security capabilities, scorecard features, and data source support that’ll be around in the second half of 2012 when the book comes out. I’ve now written 12 of the 15 chapters, writing the Answers + Dashboards chapter over Christmas, and I’ve just got Security, Clustering and High Availability and Scorecards & KPIs to write, plus all of the technical edits from Mike Durran and Venkatakrishnan J to incorporate, plus adding new features introduced in 11.1.1.6 since the original chapters were written against 11.1.1.3 and 11.1.1.5. Expect to see the book on the shelves at the Oracle Open World bookstore, and a return to more regular blogging from me once I’ve finished the book and all the NDAs are lifted!

So that’s it from me now – Happy New Year to you all from Rittman Mead in the UK, Europe, USA, India and Australia, and see you all again hopefully in early 2012.

Our most popular courses at the moment, delivered typically on-site for customers by consultants and trainers based out of our UK, Brussels, Atlanta, Bangalore and Melbourne offices are as follows:

- TRN202 OBIEE 11g Bootcamp (5 days)
- TRN201 OBIEE 10g Bootcamp (5 days)
- TRN403 ODI 11g Bootcamp (5 days), and
- TRN205 OBIEE 11g for End-Users (2 days)

Increasingly though we’ve been asked by companies looking to send just one or two delegates to a course, whether we can run public, scheduled courses that they can come to. Well now they can, as we’ve just announced our public training schedule for the first half of 2012, which features our TRN202 OBIEE 11g Bootcamp, TRN403 ODI 11g Bootcamp, and (coming soon) our TRN303 OBIA 7.9.6.3 Bootcamp.

Aimed at companies who want to train their staff but don’t have the need to train an entire department, these are the same courses that we teach on-site for customers, delivered by consultants and trainers such as myself, Borkur Steingrimsson, Stewart Bryson and Venkatakrishnan J. Bookings are open now, and this is a great way to use up any training budget left for 2011 by booking training for early 2012. Hopefully we’ll see some of you soon!

One question that often comes up is “How is OWB different from ODI, after all they both do E-LT?” I have written a small series of blogs to be published over the next few months that look at this subject from the point of view of an OWB developer moving to ODI.

To start things off here is the first of the series where I am looking at OWB and ODI in high level terms and point out some of the key differences and similarities. I will be considering the two current releases (OWB 11.2 and ODI 11.1.1.5). Later blogs will look in more detail about the actual development of ETL process and how to orchestrate them.

Both ODI and OWB have a similar (I am being very simplistic here) three-component design of: a metadata repository, a development environment where the developer defines the processes and data flows and a runtime component that executes the code and flows. It is the “how” of these things that is different for the two tools.

Both are repository driven, that is the metadata that describes the ELT processes, data structures being accessed and host of other things is held in a database schema. For OWB the repository is pre-installed (the user needs to create a workspace though) in an Oracle 11gR2 database, optionally, the OWB repository can be installed into an other Oracle database if required. ODI’s repository is installed using Oracle Fusion Middleware’s Repository Creation Utility into a supported (and not necessarily Oracle) database. With ODI, the repository can be shared with other components that use the Fusion Middleware stack such as OBIEE 11g, whether this is desirable would depend on your circumstances and factors such as your organization’s software release process and network topology – just because it is possible to have all on one database does not make it desirable.

Cosmetically, there is a lot of similarity between the two development environments, they are both part of the same unified family of Java IDE applications as JDeveloper and SQLDeveloper; the look and feel is similar, for example double-clicking on a tab has the same effect (it toggles the tab’s panel between full-sized and windowed). What is different however is the content of the windows and navigators and that is a big topic for later postings.In practice, with OWB the key parts of the IDE are those for the development of MAPPINGS and (optionally) the design of process flows to orchestrate mappings. In the ODI world think INTERFACES for mappings and PACKAGES for process flows. This is simplistic though as ODI also has PROCEDURES (code developed in one of the ODI supported languages) and LOAD PLANS (multiple packages orchestrated to execute in serial or parallel). OWB mappings require the developer to include all of the components needed to facilitate the mapping – we connect source columns to target columns through a logic flow of joiners, filters, expressions, aggregates and a whole palette of other activities. Typically, this would generate a single, but large, SQL statement with much use of in-line views. ODI interfaces are simply about connecting source columns to target columns in a logical relationship (we also create expressions, joins and filters here) and allowing the physical implementation to be supplied by a knowledge module.

In its most common usage mode, OWB deploys its executable code into PL/SQL packages in the target database. Even pure SQL set-based insert code is wrapped into a package that contains the control and audit methods that allow it to execute under the control of the Control Center and the OWB runtime. The code generated by ODI depends on the knowledge modules used and might be native SQL which is executed directly against the target database by the Java agent executing the code. Again this is a big topic and more will follow in later blogs.

At the time I said:

“Fast forward to 2009, and if I was recruiting an Oracle Business Intelligence developer now, what would I be looking for? Ideally, most if not all of the following product skills:

• Oracle Business Intelligence Enterprise Edition, usually with a specialization in either the back-end (the BI Server) or front-end (Answers, Delivers, BI Publisher), would be mandatory
• Essbase skills would be very desirable, and would be required far more than Oracle OLAP skills in the past.
• Oracle BI Application experience is a definite bonus, with the real skill in being able to apply customizations to the ETL and RPD layers
• Another very desirable skill is being able to implement at least Hyperion Planning, and ideally some more of the Hyperion financial applications. It’s rare though to find people with both good OBIEE and Hyperion skills, those that do can virtually write their own salary cheque.
• In terms of ETL, OWB is now “mainstream” and is a mandatory skill. ODI is rising in importance, but I don’t see many “classic BI” projects that use it, at least at the moment.

Compared to five years ago, skills in the Oracle database, and in particular with the Oracle Application Server, are less of a priority, as security for example is usually handled now by the OBIEE BI Server, and fast access to data is typically done through Essbase. Also, now that Oracle’s BI tools (and Fusion Middleware) are designed to work across heterogeneous platforms, its more useful now for developers to have experience with Active Directory, for example, compared to Oracle Internet Directory, and it’s also useful to have knowledge of other database platforms such as Microsoft SQL Server, Microsoft Analysis Services and the like as Oracle’s BI tools are increasingly being sold to non-Oracle database customers.”

So how have things changed, now it’s 2011? Well, some things are much the same; OBIEE, if you’re an Oracle BI developer, is a mandatory skill, and as back in 2009, you tend to specialize in the backend (RPD development, possibly Informatica if you’re an Oracle BI Applications developer), or the frontend, creating the analyses and dashboards that are most visible to the end users. Take away all of the stuff about WebLogic, Enterprise Manager and Fusion Middleware, and the core of OBIEE hasn’t changed much since the days of Siebel Analytics, and we typically see the same old problems, misunderstandings and so on with RPD development that we’ve seen since first transitioning to the technology back in 2005. Expect things to change fairly rapidly in the next couple of years though, with a new Administration tool, possibly a new (or alternative) XML or database-based repository, and a lot of work going on within Oracle to make the whole process of developing and deploying repositories a lot more automated.

One skill that an Oracle BI developer will need, and that wasn’t such a requirement back in 2009 (even, in my view at the time, becoming a less relevant skill), is application server skills, specifically Oracle WebLogic Server. For the time being at least, if you’re deploying OBIEE 11g, you’re deploying WebLogic Server, and having skills in such areas as WebLogic security, WebLogic Scripting Tool, the WebLogic Admin Console, and technologies such as admin and managed servers, node managers and OPMN is pretty much a requirement if you’re going to be an end-to-end OBIEE 11g developer. You could pretty-much install OBIEE 10g and forget about it, but nowadays you need to have some systems administration skills if only to get the product installed and running.

Another skill I downplayed in 2009 but is increasingly relevant now, is security. OBIEE 10g took care of security and connectivity to LDAP servers from within the Administration tool, but now you need to know Oracle Platform Security Services, application roles and policies, and all the associated technologies around WebLogic’s implementation of security. In fact, one major skill you need now is the ability to read and digest reams of documentation, made all the worse by OBIEE’s content being folded in with the rest of the Fusion Middleware product line, making it tricky to pick out how, for example, you can connect OBIEE to third-party LDAP servers, or how to perform upgrades between releases of 11g. It’s all a lot more complicated now, and you can’t just charge in, ignore the documentation and not do your homework.

Essbase was something I highlighted back in 2009, and at the time, it seemed fairly imminent that all Essbase projects would use Oracle BI technology, and all Oracle BI projects would include an element of Essbase. Now, i’m not so sure; Essbase is going gangbusters and if you’re an Essbase specialist, you can more or less name your price, but for OBIEE projects, we see a bit of Essbase, but it’s not gone mainstream yet. Realistically, whllst OBIEE 11g solved a number of 10g-era problems with OBIEE/Essbase integration, it’s just introduced another set of new ones, and I don’t see many WebAnalysis, Financial Management or Interactive Reporting customers moving to OBIEE unless their application was pretty-much relational in form, in the first place. If you’ve got Essbase skills, it’s certainly a bonus, but unless you’re prepared to get to expert level, you’ll probably find most Essbase projects are handled by specialist Hyperion consultants, and your job is probably more likely to be raising SRs and working with the client to try and fix the OBIEE to Essbase connectivity issues.

I smile to myself when I look back at the comment on OWB. Certainly, in terms of new projects, OWB is dead now and every customers is moving towards ODI. Maybe it’s because it’s all that Oracle will sell on new projects, maybe it’s because customers don’t want to invest in products that have been sunsetted, or maybe it’s because ODI is better, but nowadays, you need to know ODI if you’re an OBIEE developer. This means knowing things like how to create knowledge modules, how to migrate projects from environment to environment, and how to use new features such as ODI OBIEE data lineage. But if you’re an OBIEE developer and you’ve got time to learn one more skill, make it ODI.

And so finally on to the Oracle BI Applications. My experience has been that most BI Applications projects have been handled by the big ERP integrators, occasionally calling on specialists if the OBIEE element goes wrong. Either you’re an RPD, DAC and Informatica developer, or you’re an OBIEE, RPD and OWB/ODI developer, and if you’re the former you’re probably a contractor or working for a large SI. The major bonus skill that you’d have as a BI Apps developer is knowledge of either Siebel or Oracle E-Business Suite as a data source, but all of this is going to change in the next couple of years as Oracle transition to the Fusion Applications, with the 11g release of the BI Apps focusing on the Fusion Applications first and then only with support for Apps Unlimited (basically, all the legacy ERP suites Oracle developed or acquired) coming later on. The big question though, at least in my mind, is whether you can still be a BI Apps developer in the future without also being a Fusion Apps developer, and a Fusion Development (ADF, JDeveloper, SOA Suite) developer; the 11g BI Apps will be embedded in the Fusion Apps, and whilst it’s possible to develop for the 7.9.x release of BI Apps as a standalone piece of technology, I wonder how much you’ll be able to achieve in isolation from the Fusion Apps.

So, to summarize, and to update my 2009 comments about what skills an Oracle BI developer needs, in 2011 I think you need:

• OBIEE 11g skills, both in terms of new functionality (mapping, analyses, KPIs and Scorecards etc) and new infrastructure (WebLogic, EM, OPSS etc)
• A smattering of Essbase skills, focused mainly on the integration with OBIEE and Essbase (and the many workarounds and gotchas)
• Good ODI skills, both in terms of the basics, but also being able to write knowledge modules, integrate with OBIEE, deployment and migration
• Solid database skills – OBIEE gave the illusion through aggregates etc that database tuning was redundant, but time has shown it’s by far the biggest success factor in a project – get the database design and optimisation wrong, and your project is toast. You need to know partitioning, materialized views, index types, and increasingly, you need to get yourself on an Exadata project as customers are buying the technology but you can’t teach it to yourself at home
• BI Apps skills, but watch out for everything changing when BI Apps 11g comes out, and be prepared to learn the Fusion Apps and JDeveloper if you want to stay in the game
• Looking to the future, keep an eye on technologies such as in-memory (TimesTen), mid-tier caching (Coherence), plus technologies such as Business Activity Monitoring (BAM), “big data” (Hadoop, large data sets, NoSQL), complex event processing and maybe products such as Qlikview, just in case Oracle buys them, or at least to know what the competition are up to, or more importantly pitching to your boss

The other thing to bear in mind of course, if you’re an Oracle BI developer, is that you need to have great business, communication and data modeling skills. But that’s another topic in itself, and maybe one of my colleagues will have some views on that.

Something that’s always been a weakness with Oracle’s BI&DW tools (and indeed, with most vendors’ tools) is the multiple, fragmented repositories that they use. When you create data objects and data mappings in ODI, there’s no in-built link between those and the repository objects you create with the OBIEE Administration tool, and it’s also tricky to trace data elements from analyses and dashboards through to the OBIEE repository, let alone the ETL tool and the underlying data source. The Oracle Business Intelligence Data Lineage feature with ODI 11.1.1.5 looks to address this by linking the ODI and OBIEE repositories and allowing you to query lineage and impact across both sources.

To install the OBIEE Data Lineage feature, you need to make sure you’re on ODI 11.1.1.5 and then download the ODI Companion CD along with ODI itself. Once you’ve done this, you run a Data Lineage Wizard that ships as part of the Companion CD, and you’re given three options to work with.

You can set this up for both OBIEE 10g or 11g, but the install is a bit simpler and automated with 11g. To set things up, select Install Lineage in OBIEE Server, make sure the BI Server and Presentation Server are both stopped, and pick an RPD for the wizard to merge the physical, logical and subject area contents in to.

The wizard installs objects in your repository (RPD), together with analyses in your presentation catalog. Once its run, you can take a look at what’s been created.

There’s a couple of steps to change the supplied connection pool details to point to your combined ODI and master repositories (this works best if you’ve used the RCU to create your ODI repositories), but once it’s all done and you’ve checked the connections, you’ve got a set of tables that hold combined ODI and OBIEE lineage data that then needs to be initially loaded using the wizard.

Selecting the Export Metadata from OBIEE and Refresh Lineage option copies repository information from your RPD file, and analyses and dashboard information from your presentation catalog, into a set of new tables in your ODI repository. You need to run this step every time your OBIEE or ODI repository information changes, and there are supplied scripts that you can use to automate the process.

Once you’ve supplied the connection details to your OBIEE repository and catalog, you then match up the ODI models in your ODI repository with the physical databases in your OBIEE repository, so that the lineage links can be created.

Once it’s all run and setup, you can then start querying your data lineage using the supplied analyses, or create your own. In the example below, I’ve selected a subject area table from the OBIEE repository, and the analysis initially shows me the columns that it contains.

Pressing the Lineage button shows me some more information about a selected subject area column, including where it came from in the source database.

A “Runtime Stats” analysis shows me the history of agent executions within my repository, allowing me to analyze past runs, identify long-running process and spot where errors and warnings are getting raised.

You can also add contextual links to analyses in your catalog, to allow users to display details on where the data items came from in their report. For example, in the dashboard below, I’ve added a link as a text item under the analysis, which the user can then click on to display lineage data.

Clicking on the link, which is actually some embedded HTML in a text item added to the dashboard, calls another report and passes across the name of the request we’re interested in, giving us a contextual lineage report.

From that point on, I can drill-down further through the business model and mapping columns, physical layer columns, through to the ODI repository objects that were used to populate the request data.

So there we are, the last of the new ODI 11.1.1.5 features that I wanted to cover. Take a look through the ODI 11.1.1.5 New Features PDF if you’re interested in hearing what else is new with this release.

The 11.1.1.5 release of ODI is available for download on OTN and Edelivery, and you can either do a fresh install, or upgrade from either the 10g or 11.1.1.3 version of ODI. The 10g to 11g upgrade process, like OBIEE, is carried out using the Upgrade Assistant, whilst the 11.1.1.3 to 11.1.1.5 upgrade, again like OBIEE, is the awful “Fusion Middleware Patching” process that involves running lots of scripts, upgrading repositories and so on. Once you’re there though, one of the first major new features you’ll see in 11.1.1.5 is Load Plans.

As a quick recap, ETL routines using ODI are made up of Interfaces, which are typically organized into packages. You can either create packages, like this, where you drag individual interfaces onto the canvas, and each interface is then executed serially, one after the other as one session:

Or you can compile the individual interfaces into “scenarios”, and then use the scenarios in a package instead.

The advantage of using scenarios rather than interfaces in a package, apart from the fact you’re working with a fixed, compiled version of the interface that in theory won’t change from run-to-run, is that each scenario can be run in it’s own session, and execute asynchronously, with an ODI tool at the end that checks their statuses and fails over if one of the child sessions errors.

In fact running interfaces and packages in parallel, something fairly common in large ETL routines, was something fairly tricky in ODI before 11.1.1.5. Typically you’d use the compiled scenarios trick to get things running asynchronously, and you’d also set up and run several agents each of which handled a certain number of sessions before load-balancing off to another agent. Oracle did try and tackle this problem, along with sequencing and combining of sets of interfaces, with the 7.9.5.2 release of the BI Apps which used ODI, along with a tool called the Configuration Manager, to create load routines out of subject areas each of which had dependencies, run orders and so on. It wasn’t as powerful as the DAC, but combined with packages it went some of the way to meeting the BI Apps requirements, albeit in a pretty inflexible way.

ODI 11.1.1.5 introduces a new concept to handle this requirement, called Load Plans. Load Plans are executable objects that you create in your work repository, can be compiled themselves and imported into an Execution Work Repository, can run from the command-line, and are a way of adding nesting, sequencing, parallelism, error handling and restarting to your ODI project.

Going on the same example as before, where we are first loading some staging data and then insert/updating into some warehouse tables. a typical ODI 11.1.1.5 Load Plan would look like this:

When you create a load plan, individual steps are added in which can either be a Serial Step, a Parallel Step, a step to run an individual Scenario, and a Case step for conditional logic.

In the example above, we have a serial step at the start to refresh some project variables, then we load in parallel the staging tables used to feed the dimensions. Later on, we have a CASE step that checks a project variable and then only executes the rest of the plan if DW_REFRESH=1.

Something that will interest Stewart Bryson, is the ability to define resume/restart settings for individual steps and groups of steps. The default action for an interface scenario failing is for the whole load plan to fail, but you can define “exception”, procedures and other steps to, for example, flashback a table or set of tables, or run some kind of clean-up routine.

These exceptions can then be associated with steps and groupings, so that the exception is run after a step failure which will then either re-run, re-start from the parent-process or similarly for the whole load plan.

Presumably this is the sort of new feature that’s making it’s way into ODI because of requirements coming from the BI Apps, but it should be useful for most developers working with ODI. It doesn’t replace the concept of packages – packages are typically “transaction-style’ ETL routines that don’t get edited after deployment and can also take advantage of loops, which load plans don’t have (but then again, load plans have conditional execution using CASE … WHEN … ELSE). Uli Bethke, who uses ODI extensively, came up with some initial limitations that he found with load plans which are worth taking a look at, and this posting by FX from the ODI team goes into the feature in a bit more detail.

To wrap-up this mini-series, tomorrow I’ll take a look at a new ODI 11.1.1.5 feature that’ll also be interesting to OBIEE developers – Oracle Business Intelligence Enterprise Edition Data Lineage.

One such new feature is the ability to join together two or more data sets using set operators, such as UNION, INTERSECT or MINUS. To take an example, consider a situation where you have some order data sitting in a table that you want to load into a staging area, like this:

Now imagine that you had some additional orders data, but this time it was sitting in a file, containing the same columns but obviously stored separately to the table data. You could read from the two sources over two separate interfaces, or in ODI 11g you can press the Add/Remove Dataset… button, like this:

Pressing this button brings up a dialog that lets you give each dataset a name, add a new dataset reference and then select the set operator to combine them, from the list of UNION, UNION ALL, MINUS and INTERSECT.

Once you’ve added this second dataset reference, ODI adds a tab to the bottom of the Source Datastore canvas, and you can then drag and drop the file datasource into your interface, with the two sources then being combined through your chosen set operator.

Another new feature is Temporary Interfaces. If you saw my postings on the 7.9.5.2 release of the Oracle BI Applications back in 2009, you’d have seen how Oracle reproduced the maplets feature within Informatica with temporary, or “yellow” interfaces. Temporary interfaces are different from regular ODI interfaces in that they can become a data source for another interface, allowing you to encapsulate mapping functionality and re-use it across wider ETL processes. Temporary interfaces have now made their way into ODI 11g and are a handy way of breaking down a more complex mapping procedure into more manageable chunks.

To take an example, in the example above we combined a table and file source into a single interface source through a set operator, which then provided us with some order line item informaton. Now consider a situation where we want the output of this interface to be available as a data source for another interface, so that we can combine the line-level data with some order-level information such as the order date, salesperson and ship date. To turn the original interface into a temporary interface, ensure that you don’t drag a target datastore into the interface, instead drag and drop the columns you require into the target datastore area, and in the Temporary Target Properties panel at the bottom of the screen, give the output table a name, for example ORDER_DETAILS_TEMP.

Now, when the temporary interface is executed, it creates an populates in this case a table called ORDER_DETAILS_TEMP, and if you include the temporary interface in another mapping, you can join to it and read from it just like any other datastore.

The default behaviour for temporary interfaces is to persist the intermediate results (the output of the temporary interface) in a database table, which in the example above, would make sense as part of the data is coming in from a file. But in the case where the temporary interface is only working with table data, it would be nice if we could render the temporary interface as just a SELECT statement, which we could embed in the calling interface as a sub-query or sub-select.

This is actually now possible with ODI 11g, using a feature called “Derived Select for Temporary Interfaces”. To take an example, consider a situation where you are building up a complex, multi-step interface where first, you aggregate orders by customer, and then you want to add an additional column to this dataset containing the customer order rank. You could start by creating a temporary interface that aggregated orders by customer, and then create a second interface that has the first temporary interface as a data source.

Then, instead of having the first temporary interface persist its output as a temporary table, you instead navigate to the Source Properties panel in the Interface Editor, and select the Use Temporary Interface as Derived Table (Sub-Select) checkbox, like this:

The benefit of going down this route is that you’ll cut down on the disk I/O, and disk space required, by removing the need to stage the intermediate results to a temporary table. The downside is that the SQL used in the main interface is going to be more complex, might take up more memory and might be a bit trickier to debug if there’s an error. But in general, if you’re using a fairly beefy database server to do your ETL, this new feature will make sense more often than not.

So there you go – some new interface and ETL features in the initial 11.1.1.3 release of ODI 11g. But hold on – as Peter Scott mentioned, there’s just been a new 11.1.1.5 release of ODI as well, so we’ll take a look tomorrow at two new features that come with this updated version – Load Plans, and OBIEE Lineage.

Look out for more Rittman Mead postings on ODI over the coming months

Taking a step back and looking at the ETL process we felt there was a lot of complexity unnecessarily created by decomposing the process into a number of program units or mappings. In our view this process creates the following problems:

• A large amount of processing time was wasted on the inter-communication of these mappings.
• Unnessary temporary storage objects and created and populated in the database.
• A separate technology is required to orchestrate all the mappings.
• It encouraged multiple developers to work on the ETL process thereby increasing the risk of mis-communication and mis-aligned interfaces.

In response to this Rittman Mead have developed the One Mapping Paradigm. We believe that you should put all your ETL code into one mapping, and as such have called this approach the One Mapping Paradigm (OMP). The goal of this approach is to encapsulate your entire ETL routine into one mapping or program unit.

We feel this approach adheres to some of the fundamental tennets of software development: encapsulation (everything is in the one mapping) and decoupling (there are no external dependencies). Further it completely negates the need for old bugbear re-usability, you now don’t even need to re-use code, just use it once, all in the same mapping. Most importantly OMP will also provides a reduction in development costs: you now only need one developer.

Our extensive research has also developed a series of steps you can follow to deliver your One Mapping. You should note that the One Mapping that OMP generates will be extremely complex, only by following these can you address the complexity of the mapping that will be generated.

OMP follows a black hole development approach where it is crucial for the developer to do as much development as possible without any outside interfere from either peers or the business. This allows the developer to focus solely on the development task in hand, which is a must when developing extremely complex code. It is also essential that the developer is allowed to proceed as far through the process as possible without stopping for other distracting activities like testing. In order to follow the OMP I have built the following example using Oracle Warehouse Builder.

• Step 1: source objects – create new mapping a drag all your source objects onto the canvas – it is important to arrange these in a straight line on the left hand side of the canvas.
• Step 2: add all your join operators to combine the data. A couple of tips here, (1) add predicates into the join conditions to avoid using filter operators (2) keep the data transition lines as straight as possible for performance reasons.
• Step 3: add any expression or transformational operators required – these should really be added to the middle of the canvas.
• Step 4: add all your target tables – these are added to the right hand side of your canvas. You are in the home straight now, but you may find this the trickiest part and we recommend using at least a 29″ monitor to complete this process.
• Step 5: unit test – note there is no orchestration or integration required, as you only have One Mapping.
• Step 6: release to production – you can just release you mapping straight into production, overwriting whatever was there before. There is no system or integration testing required as there is only one piece of code. UAT is further bypassed as your unit testing verifies whether the entire ETL process works or not.

We are looking for beta testers for this concept, so if you want to try the OMP for your ETL code, please contact me at omp@rittmanmead.com.