Michael is well known in the Discoverer world as the author of the Oracle Press Discoverer 10g Handbook (for which I guest-authored the chapter on Discoverer for OLAP, trivia fans), is a regular contributor to the Discoverer OTN Forum and of course the IOUG and ODTUG user groups.

Michael will be answering questions not just on Discoverer, but also OBIEE and other BI technologies, so if you’ve got some time free on the 14th, pop over with your questions and say “hello”.

The tool is a part of the Administration toolset, so it’s only available on the Microsoft Windows platform. The program file is called MigrateEUL.exe and can be found in the OracleBI/server/Bin/ directory. The migration tool must be run on a machine with an existing OBIEE Administrator installation. This is a command line utility, so get ready to do some work in the Windows shell …

In this first release the migration utility only supports the generation of a repository file (.RPD) based on the business areas (BAs) that are contained in the exported .EEX file. This means that no Discoverer Workbooks can be migrated yet. This is planned for a later release. But what what we do get is a complete physical, business and presentation layer. This also includes join definitions, hierarchies for dimensions and calculations. When workbook migration will be made available we should expect to get a presentation services web catalog as well. Also due to this limitation, some things in the Discoverer EUL can not be migrated, such as optional conditions that would normally be represented as saved filters in OBIEE web catalog.

First, to be able to do this migration you must make sure you are already running on the latest release of Oracle Discoverer, 10.1.2.X (if your production environment is not running in this version, I’d suggest you get the latest release of the Discoverer Desktop tools, create a dummy EUL in your test environment and use that to quickly migrate your EUL, so that you can still go on with testing this migration utility). Using the Discoverer Administrator tool, we must first export the desired BAs to a file, as I have already mentioned. Once that is done, you simply run the command line utility, passing the name of the .EEX file as an argument:

- MigrateEUL.exe vidstr.eex

The generated results will be placed in the same folder as the input .EEX file

• vidstr.rpd – This is the resulting OBIEE repository
• vidstr.migration.log – Progress of the migration
• vidstr.exception.log – Any exceptions such as folders and joins that could not be migrated

The Logs

The migration.log will look something like this:

  Oracle BI SE - EE Migration Assistant Version 10.1.3.4.0
  Repository creation started...
  ****************************************************************
  Business Area : Video Store Tutorial
  ****************************************************************
  Custom Folder : Alternate Sort Store Size
  Custom Folder : Alternate Sort Days
  -- Processing EUL Joins....
  -- Processing Folder(s) based on EUL Joins....
  ## List of Folder(s) with SKIPPED JOINS / PATHS
  FOLDER_ID SELECTED = ### FOLDER_NAME JOIN_NAME
  --------------------------------------------
  ### FOLDER CONTAINER MAPPING ###
  Mapping For Folder : Alternate Sort Days : Alternate Sort Days
  Mapping For Folder : Alternate Sort Store Size : Alternate Sort Store Size
  Mapping For Folder : Calendar Date : Calendar Date
  Mapping For Folder : Fiscal Date : Fiscal Date
  Mapping For Folder : Products : Products
  Mapping For Folder : Sales Facts : Sales Facts
  Mapping For Folder : Store : Store
  Mapping For Folder : Target Sales : Target Sales
  -- Processing Complex Folder(s)....
  ** Complex Folder : Monthly Sales Analysis
  Complex Folder is not based on any folder in the current Business Area : Video Store Tutorial
  Complex Folder cannot be migrated!!!

  -- Processing Dimension(s)....
  ** Dimension Video Analysis Calendar Time Hierarchy based on a Complex Folder is not supported!!!
  ** Dimension Video Analysis Store Hierarchy based on a Complex Folder is not supported!!!

  ************* CREATING PHYSICAL LAYER *************
  -- Creating Database...[DONE]
  -- Creating Connection Pool...[DONE]
  -- Creating Physical Table(s)...
  -- Physical Table(s) creation...[DONE]
  -- Creating Physical Join(s)...
  -- Physical Join(s) creation...[DONE]
  ************* PHYSICAL LAYER CREATION DONE *************

  ************* CREATING LOGICAL LAYER *************
  -- Creating Subject Area...[DONE]
  -- Creating Logical Table(s)...
  -- Creating Logical Join(s)...
  -- Logical Join(s) creation...[DONE]
  -- Creating Calculation(s)...
  - Creating Complex Folder Calculation(s)...
  -- Calculation(s) creation...[DONE]
  -- Logical Table(s) creation...[DONE]
  -- Creating Dimension(s)...
  Dimension : Calendar Date Hierarchy
  Dimension : Fiscal Calendar Hierarchy
  Dimension : Product Hierarchy
  Dimension : Store Hierarchy
  -- Dimension(s) creation...[DONE]
  ************* LOGICAL LAYER CREATION DONE *************
  ************* CREATING PRESENTATION LAYER *************
  -- Creating Presentation Folder(s)...
  Folder : Video Analysis Information
  *** Admin Optional Condition Year is 1998 or 1999 will not be migrated to a repository!!!
  *** Admin Optional Condition Department is Video Rental or Video Sale will not be migrated to a repository!!!
  Folder : Stores and Sales Details
  Folder : Products
  Folder : Store
  Folder : Calendar Date
  Folder : Fiscal Date
  Folder : Sales Facts
  Folder : Target Sales
  -- Presentation Folder(s) creation...[DONE]

  ************* PRESENTATION LAYER CREATION DONE *************

  ------------------- All Business Area(s) processed -----------------
  -- Creating Security Filter(s)...
  -- Security Filter(s) creation...[DONE]
  -- Creating User(s)...

  EUL
  PUBLIC
  VIDEO5

  -- User(s) creation...[DONE]

  The migrated repository is saved at C:\TEMP\vidstrEUL.rpd
  ------------------------------------------
  EUL MIGRATION SUCCESSFUL
  ------------------------------------------

Anything that doesn’t get migrated or fails during the execution of the tool will get looged in to the the migration log as well as the execption log. The exception.log will only list, as the name suggests, the exceptions that occurred during the pass through.

  Oracle BI SE - EE Migration Assistant Version 10.1.3.4.0
  ****************************************************************
  Business Area : Video Store Tutorial
  ****************************************************************
  ## List of Folder(s) with SKIPPED JOINS / PATHS
  FOLDER_ID SELECTED = ### FOLDER_NAME JOIN_NAME
  ——————————————————————

  ** Complex Folder : Monthly Sales Analysis
  Complex Folder is not based on any folder in the current Business Area : Video Store Tutorial
  Complex Folder cannot be migrated!!!
  ** Dimension Video Analysis Calendar Time Hierarchy based on a Complex Folder is not supported!!!
  ** Dimension Video Analysis Store Hierarchy based on a Complex Folder is not supported!!!
  *** Admin Optional Condition Year is 1998 or 1999 will not be migrated to a repository!!!
  *** Admin Optional Condition Department is Video Rental or Video Sale will not be migrated to a repository!!!

The interesting things we see here are:

• Complex folders that are based on other complex folders are not supported (yet?)
• Hierarchies based on Complex Folders are not supported
• Optional Conditions are not supported (as mentioned above) as they’d need to be placed in the web catalog

Configuration Parameters

The migration tool reads configuration parameters from a file called MigrationConfig.properties that is stored in the same directory as the executable. SInce the only command line argument the tool expects is the name of the Discoverer .EEX, all other configurations need to be stated in this properties file.

Looking at the MigrationConfig.properties file we can see that the following is the default setting:

  CreateAggregatedCols=FALSE
  CreateSeperateRPDs=FALSE
  ExcludeJoins=
  ConsiderMultiplePaths=TRUE
  IncludePathsForFolders=
  DataSourceName=
  Username=

Sticking with the Discoverer Video Store Tutorial, here is what the business area looks like in the Administration tool:

After we run the migration tool, we get a new .RPD called vidstr.rpd. Opening this repository in the OBIEE Administration tool, we must first set the properties of the connection pool, so that the BI Server can connect to the database, namely the TNS connection name, username and password.

Validate and save the repository. Notice the warning that the Administrator user has been created using a blank password.

Now would be a good time to shut down the BI Server as well as the BI Presentation Server.

Copy the vidstr.rpd file to the OracleBI/server/Repository/ directory. Change the OracleBI/Server/Config/NQSConfig.ini configuration file to point to the new repository.

  #Star = samplesales.rpd, DEFAULT;
  Star = vidstr.rpd, DEFAULT;

Before we go on, we might also want to create a new web catalog to test the migration in Answers. Edit the OracleBIData/web/conf/instanceconfig.xml file to point to a new catalog. Simply giving it a new name will create a new blank catalog once the BI Presentation Service has been started.

  <CatalogPath>E:/OracleBIData/web/catalog/vidstr</CatalogPath>

Restart the BI Server and the BI Presentation Server services and log in to the Oracle BI web application and we will see that the “Business Area: Video Store Tutorial” Subject Area is now available for querying.

To conclude this, we can see that it is quite straight forward to migrate a Discoverer EUL to OBIEE. However, there are still some fundamental differences between the architecture of the two products that make it difficult to automatically migrate everything. But what we do get is a very good head start. The Video Store Sample I used here is a very simple business area, where as your EUL might contain quite a lot more complicated logic. Things can get hairy very fast when your Discoverer join configurations are slightly more complicated than a simple star. But don’t hesitate to give it a go, even if just to demonstrate what your current Discoverer reporting environment might look like, in the spiffy new OBIEE …

The past few weeks have been fairly hectic what with the preparation for my Open World sessions, adding some new material to the OBIEE course and pulling the seminar material together for my new Oracle 11g Data Warehousing seminar that’s running for the first time in Denmark next week. Apart from that I’ve been teaching and consulting in OBIEE over in Galway (a very nice place), doing some work with Borkur on a Discoverer project in London, and appearing on the panel for a BI Thought Leaders’ forum that Imperial College kindly arranged last week. Between now and Open World I’ve got the three days course delivery this week, then back to London for more Discoverer work, then I’m doing some technical consulting for an Oracle BI Applications project back in the UK for two days, then off to Denmark for the DW seminar, then back to the UK for a day before flying off to San Francisco. In all it’s turning into one of those months where we’re working flat out and often away from home, in my experience it’s often now (autumn) and the early part of the year (February through to May) when we’re the most busiest and this year Open World falls right into the middle of it. Hopefully come the end of September things will have calmed down a bit at least in terms of R&D work, however I suspect the first OBIEE 11g beta will arrive around then so I doubt I’ll actually get much spare time.

Apart from consulting and training, the other thing we’ve been working on is pulling an agenda together for the first of our “Oracle BI Training Days” events, where we’ll be hiring out a venue in London for three days and offering intensive, expert-level training on OBIEE and the Oracle BI Applications for a small, public audience, plus a social event on one of the evenings where we’ll all get a chance to go out for a meal and shoot the breeze on Oracle BI development. There’ll be more news on this during the next week or so, but if you’re interested the dates are likely to be the 22nd through to the 24th of October, the venue will be Canary Wharf London, and we’ll post more details soon.

If you try and bring this arrangement in to the Physical Layer of your OBIEE semantic model, the administration tool will initially allow this and your repository verifies as being correct.

However if you then drag and drop the tables into the logical business model layer, all three lookup tables are shown as dimension tables, as you would expect,

and the BI Administrator tool shows an error message when you try and verify the repository, saying that circular joins aren’t supported.

In reality, you don’t want three dimension tables (departments, regions, employees) joining to the fact table (appointments), you just want one, so you’re going to have to use the mapping ability in the business model and mapping layer to join these three lookup tables together under one logical table source. For the employee table though, rather than join it through this circular join, you need to create an alias for it in the physical layer (in my case I call it CIRC_JOIN_REGEMP) like this:

and join the region table to this rather than the employees table.

Then you can create a single logical dimension table within your logical business model, and map the logical table source to the join between the four lookup tables, including your new alias for the employee table.

When you add the new columns into the logical table and then run a report, the figures come up as expected, with the regional manager details displayed correctly, the alias having resolved the circular join.

Another issue that often comes up with Discoverer is “Fan Traps”. Fan traps are an oddity of SQL where, if you have say a dimension table full of account details, and two facts that join to it, say with sales figures in one and budget figures in the other, logically if you query the two together you should get a set of results like this, taken from the Oracle document “Oracle Discoverer’s Fan Trap Resolution – Correct Results Every Time”

The problem is though, that if you issue the obvious query in SQL, the results come back wrong, because SQL does a cartesian join between the two fact tables and over-counts the figures.

SQL> select a.acc_name,
2  sum(sales),
3  sum(budget)
4  from fan_trap_accounts a, fan_trap_sales s, fan_trap_budget b
5  where a.acc_id = s.acc_id
6  and   a.acc_id = b.acc_id
7  group by a.acc_name
8  order by a.acc_name
9  /

ACC_NAME             SUM(SALES) SUM(BUDGET)
-------------------- ---------- -----------
ACCOUNT1                    900        1050
ACCOUNT2                    130         200
ACCOUNT3                    600         750
ACCOUNT4                    600         600

Now Discoverer handles this quite well, and automatically rewrites the query internally to use two in-line views, one to join the sales and accounts tables, and one to join the budget and accounts tables, and then join the results together with the correct aggregation applied. So how does OBIEE handle this? Does it ignore the potential error and give you the wrong figures, or does it even disallow this sort of schema design in the first place? Let’s take a look.

The BI Administrator tool doesn’t have a problem with this sort of arrangement in the physical layer:

The logical business model is fine as well, with the sales and budget tables being displayed as fact tables, and the accounts table as a dimension, as you would expect.

In fact, when you run a query, using the same selection as our SQL statement, the figures do in fact come out correct.

So what’s happening? Well, if you take a look at the query log and see how the logical, and physical, SQL queries are put together, you can see that OBIEE is doing what Discoverer does, but within the BI Server rather than writing a single SQL statement that uses inline views. The BI Server effectively creates two inline views that in turn lead to two physical SQL statements being sent to Oracle, the BI Server then joins them together and returns the data back as a single result set.

-------------------- Execution plan:

RqBreakFilter <<2750>>[1] [for database 0:0,0]
    RqList <<2641>> [for database 0:0,0]
        case  when D903.c1 is not null then D903.c1 when D903.c2 is not null then D903.c2 end  as c1 GB [for database 3023:5710,44],
        D903.c3 as c2 GB [for database 3023:5710,44],
        D903.c4 as c3 GB [for database 3023:5710,44]
    Child Nodes (RqJoinSpec): <<2752>> [for database 0:0,0]
        (
            RqList <<2795>> [for database 0:0,0]
                D902.c1 as c1 [for database 3023:5710,44],
                D901.c1 as c2 [for database 3023:5710,44],
                D901.c2 as c3 GB [for database 3023:5710,44],
                D902.c2 as c4 GB [for database 3023:5710,44]
            Child Nodes (RqJoinSpec): <<2798>> [for database 0:0,0]

                    (
                        RqList <<2653>> [for database 3023:5710:ora11g,44]
                            FAN_TRAP_ACCOUNTS.ACC_NAME as c1 GB [for database 3023:5710,44],
                            sum(FAN_TRAP_SALES.SALES by [ FAN_TRAP_ACCOUNTS.ACC_NAME] ) as c2 GB [for database 3023:5710,44]
                        Child Nodes (RqJoinSpec): <<2686>> [for database 3023:5710:ora11g,44]
                            FAN_TRAP_ACCOUNTS T6678
                            FAN_TRAP_SALES T6685
                        DetailFilter: FAN_TRAP_ACCOUNTS.ACC_ID = FAN_TRAP_SALES.ACC_ID [for database 0:0]
                        GroupBy: [ FAN_TRAP_ACCOUNTS.ACC_NAME]  [for database 3023:5710,44]
                        OrderBy: c1 asc [for database 3023:5710,44]
                    ) as D901 FullOuterStitchJoin <<2744>> On D901.c1 =NullsEqual D902.c1; actual join vectors:  [ 0 ] =  [ 0 ]

                    (
                        RqList <<2690>> [for database 3023:5710:ora11g,44]
                            FAN_TRAP_ACCOUNTS.ACC_NAME as c1 GB [for database 3023:5710,44],
                            sum(FAN_TRAP_BUDGET.BUDGET by [ FAN_TRAP_ACCOUNTS.ACC_NAME] ) as c2 GB [for database 3023:5710,44]
                        Child Nodes (RqJoinSpec): <<2723>> [for database 3023:5710:ora11g,44]
                            FAN_TRAP_ACCOUNTS T6678
                            FAN_TRAP_BUDGET T6681
                        DetailFilter: FAN_TRAP_ACCOUNTS.ACC_ID = FAN_TRAP_BUDGET.ACC_ID [for database 0:0]
                        GroupBy: [ FAN_TRAP_ACCOUNTS.ACC_NAME]  [for database 3023:5710,44]
                        OrderBy: c1 asc [for database 3023:5710,44]
                    ) as D902
        ) as D903
    OrderBy: c1 asc [for database 0:0,0]

+++Administrator:2a0000:2a0001:----2008/08/26 17:24:48

-------------------- Sending query to database named ora11g (id: <<2653>>):

select T6678.ACC_NAME as c1,
     sum(T6685.SALES) as c2
from
     CUST_ORDER_HISTORY.FAN_TRAP_ACCOUNTS T6678,
     CUST_ORDER_HISTORY.FAN_TRAP_SALES T6685
where  ( T6678.ACC_ID = T6685.ACC_ID )
group by T6678.ACC_NAME
order by c1

+++Administrator:2a0000:2a0001:----2008/08/26 17:24:48

-------------------- Sending query to database named ora11g (id: <<2690>>):

select T6678.ACC_NAME as c1,
     sum(T6681.BUDGET) as c2
from
     CUST_ORDER_HISTORY.FAN_TRAP_ACCOUNTS T6678,
     CUST_ORDER_HISTORY.FAN_TRAP_BUDGET T6681
where  ( T6678.ACC_ID = T6681.ACC_ID )
group by T6678.ACC_NAME
order by c1

Now this strictly speaking isn’t as efficient as Discoverer’s approach, which uses a single SQL statement and does the join of the two inline views within the database, but it’s still pretty clever and would work for platforms that don’t support inline views. In a way its an “on the fly” creation of two logical table sources without you having to set this up explicitly in the logical business model.

One thing I haven’t tried out yet is how OBIEE handles queries that, in Discoverer, would cause a fan-trap detection error, i.e. queries that it can’t rewrite to use two or more inline views. I suspect that these sorts of scenarios, that usually involve facts of differing granularity, where circular joins exist or where the keys to the tables don’t correspond would get picked up by the general data modeling rules in OBIEE, but if anyone’s got any experiences with this then by all means leave a comment.

If you’re wondering what prompted this posting, I’m putting the finishing touches to my Oracle Open World presentation on OBIEE data modeling, and looking to cover off a few data modeling “oddities”. I’ll try and cover support for ragged/unbalanced/value-based hierarchies in a posting tomorrow, and finish up with how the logical business model supports the Kimball concept of Bridge Tables, but no-one seems to use them.