Oracle Data Integrator 12c release : Part 2

November 4th, 2013 by

In the first part of this article, we discovered the new mappings and reusable mappings, the debugger and the support of OWB jobs in the freshly released ODI 12c. Let’s continue to cover the new features in ODI 12c with the Weblogic Management Framework, the new component KMs and some features to improve performances or ease of use.

ODI12c Logo

WebLogic Management Framework

Another big change with this new version is the new WebLogic Management Framework used to manage the standalone agent in replacement of OPMN (Oracle Process Manager and Notification). Based on Weblogic Server, it brings a unified management of Fusion Middleware components and it provides a wizard for the configuration – the Fusion Middleware Configuration Wizzard.

ODI12c CAM

Practically the standalone agent is now installed outside of the ODI installation, in a domain similar to WLS domains. The benefits of it, apart from the centralised management, is that it supports Weblogic Scripting Tool (WLST) and can be monitored through Enterprise Manage Fusion Middleware Control or Enterprise Manager Cloud Control. The standalone agent can be launched manually or through the Node Manager.


Component Knowledge Modules

Next to the standard KMs – now called Template KMs -, a new type of knowledge modules has been introduced in this release. Going one step further in the reusability, the Component KMs contain steps that are defined once and shared among several of them. Instead of being templates of code with some call to the Susbtitution API at runtime , these KMs are actually compiled libraries that will generate the code based on the components present in a mapping.

The Component KMs are provided out-of-the-box and are not editable. You can check the type of a KM in the physical tab of a mapping.

Component Knowledge Module

Knowledge Module Editor

Don’t worry, you can still import, edit and create Template KMs. You can now find them in the folder <ODI_HOME>/odi/sdk/xml-reference.

Good news, the KM Editor has been reviewed to avoid us to constantly switch from one tab to another. Now when the focus is set on a KM task, it is directly displayed in the Property Editor. In addition the command field supports syntax highlighting and auto-completion (!!). Options are also now directly managed from the KM editor and not from the Project tree.

ODI12c KM Editor

In-Session Parallelism

With ODI 12c, it is now possible to have the extract tasks (LKMs) running in parallel. It’s actually done by default. If two sources are located in the same execution unit on the physical tab, they will run in parallel. If you want a sequential execution, you can drag and drop one of your units onto a blank area. A new execution unit will be created and ODI will choose in which order it will be loaded.

ODI 12c In-session Parallelism

Thanks to this the execution time can be reduced, especially if your sources come from different dataservers.

In the topology, you can now define the number of threads per Session for your physical agents to avoid one session to get all the resources.

maxim_thread_session_ds

Parallel Target Table Load

With ODI 11g, if two interfaces loading data in the same datastore are executed at the same time or if the same interface is executed twice, you can face some problem. For instance a session might delete a worktable in which the other session wants to insert data.

This now belongs to the past. With 12c, a new “Use Unique Temporary Object Names” checkbox appears in the Physical tab of your mapping. If you select it, the worktables for every session will have a unique name. You are now sure that another session won’t delete it or insert other data.

I can hear a little voice : “Ok but what if the execution of the mapping fails? With these unique name, these tables won’t be deleted the next time we run the mapping.” Don’t worry, the ODI team thought about everything. A task in a KM can now be created under the section “Mapping Cleanup”. It will be executed at the end of the mapping even if it fails. That’s a nice feature that I will also use for my logging tasks!

ODI12c Cleanup Tasks

“But eh, what if the agent crashes? These cleanup tasks won’t be executed anyway.” No worries… A new ODI Tool is now available : OdiRemoveTemporaryObjects. It runs every cleanup tasks that have not run normally and by default it is automatically called when a agent restarts. Little voice 0 – 2 ODI Team.

Session Blueprint

When doing real-time integration with ODI, you can have a lot of jobs running every few seconds or minutes. For each call of a scenario, the ODI 11g agent retrieves all the steps and tasks it should execute from the repository and write it synchronously in the logs before starting to execute anything. After the execution it deletes the logs that shouldn’t be kept when the log level is low. At the end of the day, even if you set the logs to a low level, you end up with a lot of redo logs and archive logs in your database.

Instead of retrieving the scenario for every execution, the ODI 12c agent now only retrieves a Session Blueprint once and keep it in the cache. As it is cached in the agent, there is no need to get anything from the repository if the job runs again a few minutes later. The agent also write – asynchronous – only the needed logs defined by the log level, as it only relies on the blueprint for it’s execution and not on the logs. The parameters related to blueprints are available in the definition of the physical agents in the topology.

ODI12c Blueprint

In summary, thanks to the Session Blueprints the overhead of executing session is greatly reduced. The agent needs to retrieve less data from the repository, doesn’t insert-and-delete logs anymore and write it asynchronously. Isn’t it great?

Datastore Change Notification

This happened to me once again last week : we needed to rename a datastore column in ODI but we couldn’t because it was used as a target datastore in an interface. The solution is to remove the mapping expression for this column in every interface where the datastore is used as a target.

It will not happen anymore with 12c. Now, you can change it in the model and a change notification will be displayed in every mapping using this datastore. In the following example, I removed the LOAD_DATE column.

ODI 12c Change Notification

Wallet

Ever faced this dilemma in 11g ? Giving the Master Repository password to everyone is not safe, but saving it in the Login Credentials is not better if a developer forgets to lock his screen when he is away from keyboard.

Once again, 12c is there to help you. You can now store your Login Credentials in a Wallet, itself protected by a password. One password to rule them all… and keep them safe!

ODI 12c Wallet

Try it !

If you want to give a try to ODI 12c and follow the Getting Started Guide, you can download a Pre-Built Virtual Machine. It’s running on Oracle Enterprise Linux with an Oracle XE Database 11.2.0. Of course, you will need Virtual Box to run it.

See you soon

Needless to say I’m very exited about this release. Combining the greatness of the ODI architecture with the new flow-based paradigm and all the new features, ODI is now a very mature tool and the best-of-breed for Data Integration regardless of the source or the target technology.

Keep watching this blog, there are more posts to come about data integration! Stewart Bryson already announced in his last post he would talk about the ODI 12c JEE agent and Michael Rainey started with GoldenGate 12c New Features. Who will shoot next?

[Update] Peter Scott will also present ODI 12c in his session at the Bulgarian Oracle Users Group Autumn Conference on 22nd November 2013.  Stewart and myself will also talk about it at RMOUG Training Days in February.

Tags: , , , , , , , , ,

Comments

  1. Antoine Drieux Says:

    Very nices new features. Thanks for this article.

    By the way, on the oracle download site, the archive I’ve downloaded just has a “opatch” installer. Do you know if a “.exe” installer exists ?

  2. Jérôme Françoisse Says:

    Hi Antoine,

    There should be a jar file (odi_121200.jar) next to the opatch zip file in the archive you downloaded. As it’s a generic installer, you should be able to run it on any 64-bits system supported (Linux, Windows, …).

    Cheers,
    JeromeFr

  3. Peter Scott Says:

    In addition to the future posts that Jérôme mentions, I will be at the Bulgarian User Group Conference on the 22 November giving an updated (12c) version of my “ODI for OWB developers:” talk

  4. David Allan Says:

    Hi Jérôme

    Thanks for sharing, looks like you have been very busy!

    The maximum number of threads per agent is also a new configuration, so you can throttle it along with each session. You can then design with max parallel capabilities and throttle the runtime.

    Cheers
    David

  5. VS Says:

    Is it really nothing left to us to improve in ODI? Parallelism, automatic clean-up, compiled KMs – all of that just force every developer to think about business logic implementation. No more exciting?

  6. Mayuri Nallamothu Says:

    In OBIA 11.1.1.7.1, how to setup datalineage?
    I see there is loadplan Datalineage and dashboards are already there in the analytics. Is there any recommended way to run the datalineage load plan manually? I see the documentation for previous version to setup datalineage using ODIcompanion software but I see nothing for 11.1.1.7.1. Please advice.

    Thanks,
    Mayuri.

  7. Stewart Bryson Says:

    @Mayuri: Your question is regarding OBIA, which is not yet supported on ODI 12c. So I’m not sure this is the right post to raise your question.

  8. Antoine Drieux Says:

    Hi. I’ve difficulties to see what are the “component km”.

    They are already compiled, not modifiable, and are not anymore “template code with API methods”. But it seems to be far away from the original sunopsis concept, don’t you think ?

  9. David Allan Says:

    Hi Antoine

    You can still write AND use all KMs as in Sunopsis/ODI – all of the odiRef substitution reference API is there and working in ODI 12c. You can still write LKM/IKM/CKM/JKM etc The component KMs an internalized piece of ODI code generation for now, although Jerome has highlighted them.

    It was necessary to introduce component-style KMs to support arbitrary assembly of map components (this is a benefit) and for that to be engineered in a fashion to minimize code replication and build software components in a modular manner (these are benefits). The architectural changes to the code generation based on components and component KMs takes care of this plus you can still leverage LKMs, IKMs, CKMs, JKMs and all that good stuff.

    Cheers
    David

  10. Jérôme Françoisse Says:

    Thanks for your answer David.

    By the way, your article about Parallel Table Load is a must-read.

Website Design & Build: tymedia.co.uk