HPCM primarily has 3 main sweet spots

1. Every company will have indirect costs. For example, a Consulting company where Revenue is obtained through driving projects will have a lot of indirect costs like HR Costs, Admin Costs etc. HPCM provides an ability to allocate the costs back to the projects so that proper project profitability is derived. How the costs are allocated will be defined through HPCM itself. For example, if a company is running say 3 consulting projects with 20, 30 & 40 resources each, then the indirect costs are allocated back to the project based on the number of people(or time logged etc) in each project.

2. Every company will be storing the incoming Revenue & Costs in a Ledger. Due to various reasons, even the direct costs & revenue might not actually be tied back to a project (Consulting company example above). So, there might be a need to allocate the project based Direct Costs & Revenue as well to the Projects (allocation possibly by head count etc).

3. Allocation of Costs is pretty dynamic in nature depending on the type of business. It can vary quite frequently. So, the key is to ensure that the allocation logic can be changed frequently and easily. In addition, one more key point is to find the lineage back to the source on how the costs are obtained.

HPCM provides all the 3 above. If you are an Essbase or a Planning person, you could argue that we can do the same thing using these 2 products itself. Though true, in many cases Cost & Revenue allocation rules are defined by Functional Users. So, it is not possible for Functional users to create Business Rules/Calculation Scripts every time there is a change. In addition though Essbase is very good, it is very difficult to do a data lineage from a calculation script, to find out how the costs are allocated. Thats the main reason why, HPCM is a solution positioned primarily at Business/Functional users for providing that cost & Revenue Breakdown.

Though i have mentioned that HPCM is a functional tool, its underlying technology is very interesting. It has a relational metadata that stores the metadata related to HPCM. In addition each HPCM application will have 2 Essbase databases. One is Block Storage cube which will be used for the allocation & calculations. The other is a reporting Aggregate Storage cube which will be used for reporting. Data push from BSO and ASO is automatically available out of the box from HPCM. Also, one important point to note, change to dimensions, change to metadata, pushing data from BSO to ASO are all achieved within HPCM without writing any external code/scripts. Everything is done out of the box. This architecture is shown below.

In addition, the most interesting aspect of HPCM is the way it handles dimensions. This is what we will be covering in this article today. HPCM uses EPMA for managing its dimensions & attributes. HPCM as an application has 3 types of dimensions

1. System Dimensions – There are 2 System Dimensions – Measures & AllocationType. Generally while creating a HPCM application through the Wizard we can pre-create these 2 dimensions. AllocationType is used by HPCM internally for doing allocations. It is generally not needed to make any changes to this dimension. But Measures dimension is the most important dimension that HPCM uses for pushing costs & allocating them. We can create custom members in the Measure dimension if needed.

2. POV Dimensions – HPCM supports upto 4 point of view dimensions. These dimensions are generally for storing Year, Period, Scenario & Version. True to their names, they generally are used as POVs and are not used directly in any allocation (the POVs are always fixed in the calculation scripts).

3. Business Dimensions – Business Dimensions are those dimensions where allocations happen. These dimensions drive the allocation logic.

In addition HPCM also supports alias and attribute dimensions. For this article, i will use a simple case of demonstrating how to go about allocating HR Costs in a Consulting Company recognising its revenue through Projects. Lets make an assumption that on a monthly basis we record the HR Costs (including Salary paid to HR, other misc costs) etc. Lets also assume that we have 3 projects running in the company with the following break-up

a. Project A – 300 people full time
b. Project B – 500 people full time
c. Proejct C – 200 people full time

We will start off with creating the application through the Application Wizard (pre-create System Dimensions) and then we shall define the necessary dimensions.

We basically have 2 business dimensions – one for Accounts which will hold the HR Costs. Then we have the Project dimension which will record the revenue and costs specific to the project.

Lets deploy this application and then login to the application through Workspace.

In HPCM all allocation happens through stages. Stages is where allocation happen. HPCM supports uppto 9 stages. Each stage also supports intra stage allocation. Lets try to understand what this means from a Multi-Dimensional Essbase database Standpoint. In our example, we will have 2 stages. The first stage will have just the Accounts dimension – basically HRCosts in Accounts dimension will flow from Stage 1 to Stage 2 and will get allocated in Stage 2. So, Stage 2 will contain both the Accounts and Project dimensions.

After creating these 2 stages(ensure you also have a POV defined) lets go ahead and deploy this to Essbase from the Manage Database screen(both Calculation & Reporting Database). What you will notice is 3 things

1. There will be 2 essbase databases one suffixed with letter C and the other suffixed with letter R. C database is the Block Storage database that is used for allocation. R database is the Aggregated Storage database that will be used for reporting.

2. You will notice that for each stage there will be a corresponding set of dimensions prefixed by the Stage prefix given at the time of creation. So effectively, if there are 2 stages with 2 dimensions each, then Essbase will have 4 dimensions (though the 2 dimensions might be the same in EPMA).

3. You will also notice that each dimension will have a dummy member called NoMember. This is one of the most important members that controls the grain of the data. This member is the key in loading multi-grain data for allocation into HPCM.

Now that we have the Essbase cubes deployed as well, lets try to understand how the allocation logic works. To begin with lets assume for the month of Jan 2011, the HR Costs for the Consulting Company is say 1000 USD as shown below

This is the input data into Stage 1. So to load this in we will have be creating a text file and load it directly into Essbase. There are 3 options to load data into HPCM

1. Manual Data Entry – HPCM provides a screen where we can update data manually.

2. Staging Tables – We can load the data temporarily into a set of staging tables, and then from within HPCM we can push the data from Staging tables into Essbase.

3. Directly loading data into Essbase

In our case, we will load the data directly into Essbase as that will give more clarity on how HPCM works. For doing data load for Stage 1, remember we have a total of 9 dimensions in Essbase (Measures,AllocationType,Year, Scenario, Period, Version, ACAccount,PRAccount,PRProject). But our input data of HRCosts comes at a grain of only 7 dimensions (Measures,AllocationType,Year, Scenario, Period, Version, ACAccount). So, load this in we will have use the NoMember intersection of the remaining 2 dimensions (PRAccount & PRProject). The input data file to Essbase is shown below

CostInput,DirectAllocation,2011,Jan,Actual,Working,HRCosts,[PRAccounts].[NoMember],[PRProgram].[NoMember],1000
Our idea is to allocate the 1000 USD down to the 3 projects for the January Month. So, our end result should look like this

If you notice, the 1000 USD is split across the 3 projects based on the overall number of resources(Resource Count for each project/Total Resources*HRCost) in each project. So basically for the allocation to happen, we need to load the Resource Count data. Resource Count data for all projects and the individual projects have to be loaded as shown below

FixedDriverValue,DirectAllocation,2011,Jan,Working,Actual,HRCosts,HRCosts,Project A,300
FixedDriverValue,DirectAllocation,2011,Jan,Working,Actual,HRCosts,HRCosts,Project B,500
FixedDriverValue,DirectAllocation,2011,Jan,Working,Actual,HRCosts,HRCosts,Project C,200
Weight,DirectAllocation,2011,Jan,Working,Actual,HRCosts,[PRAccounts].[NoMember],[PRProgram].[NoMember],1000

In the above data file there are 2 things we can notice

1. We have used a measure called FixedDriverValue and Weight. We will see their significance a little bit later. For now think of them as intersections that will hold the Resource Count Data.

2. The first 3 records above have HRCosts repeated twice to load into both the Accounts dimension we have in the Essbase Cube. Again we will see the significance of why we are doing this below.

So far we have loaded all the necessary data into the Essbase Cubes and have also setup the stages. The next step is in defining the Allocation Logic. This is done through a concept called Drivers. From the Perspective of HPCM, drivers define how the allocation values get pushed from source to target. HPCM supports different allocations like Activity Based Costing etc through the concept of Drivers. In our case, the Driver for allocation is the Resource Count. Just to recap, within Essbase now we have dimensions catering to two stages – Stage 1 and Stage 2. This is illustrated below. So basically we have 3 separate sub-cubes each having its own intersection.

In the above diagram, whenever we want to do allocation there can be different ways of doing it. The 3 most common ways are

1. Allocation Based on Source – Here all the driver values are obtained from the source stage and data from source stage is then assigned to the Destination Stage based on the Source driver values.

2. Allocation Based on Destination – Here all the driver values are obtained from Destination stage and data from source stage is then assigned to the Destination Stage based on the Destination driver values.

3. Allocation Based on Source & Destination – Also known as Assignments – Here all the driver values are obtained from the intersection of Source & Destination and data from source stage is then assigned to the Destination Stage based on the Assignment driver values.

If you look at our Driver data above (3 records containing FixedDriverValue & 1 containing Weight), you can see that our driver (Resource Count) is loaded in 2 ways. First the FixedDriverValue is loaded at the intersection of Source & Destination stages. The second driver Weight is loaded at the Source Stage intersection. We have chosen these 2 measures (FixedDriverValue & Weight). We could have chosen any other measure like Rate, Volume etc. But each measure has a logical meaning and it is better to stick to the ones we think is logically close to what we are trying to do. If we are not able to use the existing measures and if they don’t relate to our driver names then we can create custom measures. So, this allocation handling through drivers is defined through the Driver Definition screen in HPCM.

As you see in our formula we are basically doing a division of each Resource Count from Stage 2 by the Total Resource Count in Stage 1. HPCM will automatically multiply the resulting values of the driver to the CostInput measure (input data) thereby allocating.

Once we have defined the driver the next step is to assign the driver to both the stages. HPCM allows us to have multiple drivers for each stage. And even within a single dimension in a stage we can define exceptions so that multiple drivers can be assigned within the same dimension. This way we can have allocation logic based not only on Resource Count but also on say Clocked Time of a resource in certain cases. Driver assignment is done through the Driver Selection screen in HPCM.

After the assignments, one other important feature of HPCM is its ability to assign allocations on a cell by cell basis. So, what we can do is, for each cell in the Source Stage we can assign the Destination cell(s) in the Destination Stage. That way we can clearly find out what intersections will get affected by what allocation logic. If you had to do them through say Essbase Calculation Scripts alone, it would have been such a laborious task not only while doing the allocations but also in maintaining them. In our example, lets assign the Data intersection of HRCosts in Stage 1 to the 3 Destination Projects in Stage 2 as shown below

Now lets run the Calculation & then immediately transfer the data to the ASO cube from the Manage Calculation screen.

If you now look at the data you can clearly see that each project now will have the corresponding costs allocated.

To validate how the costs have flown through the stages, HPCM provides an option to do a stage balancing report. This will show us how the costs have moved from stage to stage. In addition one biggest advantage of HPCM is, if there are any unassigned costs in the source, those will automatically move into Idle Costs (or we can configure it throw an error to ensure all costs are always allocated).

In terms of reporting, the ASO cube that HPCM provides will be more than sufficient. But based on what i have seen, we will have a lot of un-necessary dimensions especially when we are not concerned about the inter-stage essbase dimensions. In such cases, we can build our own ASO cube and extract only the necessary intersections that we need. But for now, i will showcase how BI EE 11g can be used for reporting again the native HPCM ASO cube. I will quickly show the lineage breakup reports as well

We start off with importing the ASO cube into BI EE which is pretty straight-forward as shown below.

Then lets start with building the report just for looking at our Stage 1 data. Remember Stage 1 input HRCosts data is loaded against NoMember intersections of the other dimensions. So we will have to explicitly filter to arrive at the right data.

Similarly to get the final allocated data we will have to apply filters as shown below

In the same way we can get the driver values directly from the cube using a simple BI EE report. All of these are shown in the form of a dashboard – showing how the costs from stage to stage using the drivers.

1. This does not have a ETL layer – Basically this is the second time i am seeing a BI Apps suite without a ETL layer (last time being the CPM Analytics which is no more available). So the reports/dashboards directly go against HFM & Close Management. Interestingly there is also a Single Sign-on support. So essentially we can start using this as an alternative to Hyperion Financial Reporting (functionality is still behind when compared with HFR – but at least to some extent wherever possible) that has been the standard & only reporting tool available so far on HFM.

2. This has support not only for data but also for HFM Process Management. Essentially we can not only report on the data but to an extent on the process management. This has been the sore point in HFM considering there are no built in tools outside of HFR that can leverage this (but of course there is always the API). Even ODI or HyperionDIM(based on Informatica) cannot extract Process Management related details out of HFM. So, we always had to rely on external adapters or direct API calls to do this. That is not the case anymore.

3. This has support for the newly released Close Management suite. Considering the fact that HFM & Close Management go together, it makes sense to have these 2 together supported out of the box.

4. The entire integration is based on the connector that i had blogged about earlier here. So, no extra configuration/setup (in terms of binaries – though there is a install process which just sets up the reports/dashboards) is required for this to work.

The architecture of this suite is given below

As mentioned in the earlier linked post, this suite uses the ADM driver to connect and extract data out of HFM. So, there are 2 things to keep in mind

1. This suite requires you to be 11.1.2.1.102 version of EPM(earlier releases are supported though – 9.3.3 and 11.1.1.3) – That is you need to apply one more patch on the HFM side to basically add a new feature to the HFM driver (for process management).

2. You need to be on BI EE 11.1.1.5.1 MLR patch set for this to work without any issues. More than the above EPM certification, 11.1.1.5.1 patch is very critical for the connectivity to work. But interestingly this MLR patch is not recommended for all customers. I am not sure why it is this way but looking at the Readme of this patch, it looks like 11.1.1.5.1 patch is meant only for customers who require HFM connectivity and Fusion integration. But for other customers who don’t need this, the patch recommends not to upgrade at all which is interesting as you would expect any latest patch set to be valid for all customer environments.

Once you have both the above, download the HFM Analytics suite from Edelivery. The install itself is straight forward.

Ensure that you are using a Custom install. Looks like there is a bug in the installer. If you do a full install, the installer does not seem to install the web catalog.

The installer basically installs a RPD, Web Catalog and a configuration utility. One of the advantages of using HFM is, we will always have a constant set of dimensions (8 system dimensions + 4 custom dimensions). So, oracle has basically fixed the RPD but the web catalog structure will vary based on the number of custom dimensions used. That is where the configuration utility comes into play. The default password of the repository is welcome1. Let’s open this RPD and take a look at the contents

Basically FCM is modelled using the relational metadata directly. But the HFM one uses the ADM Driver. To complete the configuration, we need to run the configuration utility which gets installed along with the software.

We start with specifying the connection details of both the BI instance and HFM instance. In addition we need to provide the RPD and Web Catalog location details that comes along with the software. Interestingly the configuration utility injects new content into the web catalog dynamically based on our selection. We start with selecting the Entity values that we are interested in for reporting.

Then we need to provide the following account grouping details

1. Balance Sheet Accounts

2. P & L Accounts

3. Cash Flow Accounts

4. Performance Indicator Accounts

Actually there are more groupings but at a high level the above are the basic classifications required.

If we have any Custom dimensions configured then those values need to be configured as well. Basically the values chosen for custom dimensions (along with default values) will be used for the web catalog prompts and filters. Initially it might be a bit confusing but once we have web catalog, it sort of makes sense how the grouping actually works.

Issues:

Though it works and does give a high level complete picture of the HFM financials, there still are certain grey areas in this suite (atleast for me). I will list them below. If anyone has any other insights in this, i will be interested in hearing them.

1. License – This is one area of the product that i am not able to understand. Based on what i heard in Open World, this requires an extra license. But again if you look at the underpinnings of how this works, there is nothing that stops a customer to build the same set of reports and dashboards with just a HFM and BI EE 11g license. In fact, to make this work (on my demo data sets), i myself had to understand the reports and make the requisite changes to the filters. So, anyone can build the same set of reports and dashboards all by themselves without having to license this(i.e. just use the BI EE 11g-HFM connector). Or i am hoping i am just wrong in this case.

2. Journal Entries – As far as i see ADM driver is pretty limited in its functionality. There is no way to just create a report with just the journal entries. For that we still have to rely on Hyperion Financial Reports.

3. Active/Inactive Entities – IC Transactions etc – Same comment as above. The connectivity does not support reporting for Active/Inactive entities of a period, or for just creating a report containing just the Inter Company transactions etc.

5. Custom Dimension names – Somehow the connector is sort of hard-coded to use Custom1, Custom2 etc dimension names. If we had used a different name in the Aliases field then the connector throws an error. This looks more to be a bug in the connector. So, if you have a custom named dimensions then revert them to Custom1, Custom2 etc until this is fixed.

6. BI EE 11.1.1.5 – If you are using this without the MLR patch, the reports will work. But none of the custom attribute columns will work. I guess that is due to the ADM client driver that is bundled as part of the BI EE 11.1.1.5. But hopefully once you upgrade this issue should go away.

To illustrate what I’m talking about, take a simple analysis created against the Sample.Basic Essbase database, which I’ve imported into an OBIEE 11.1.1.5. repository using the default settings, and then created physical cube columns for the UDAs within the Market dimension, and the Default alias table within the Product dimension.

Notice how, at this stage, the measure hierarchy has been brought in as-is, with the full hierarchy preserved and a single measure, in a physical cube column called Basic – measure. To get access to a particular measure (for example, Profit), you’ll need to include this measure in an analysis criteria and filter on the Gen2, Measures, Gen3, Measures or Gen4, Measures logical column; or alternatively, access the measure dimension via a hierarchical column, and drill-down or use selection steps until you get to the one you want.

So let’s start then by creating an analysis where I have the Marketing and Payroll measures along the columns, and the East and West regions on the rows, and the numbers are for the Budget scenario. In MDX terms, using the essmsh command-line tool this would look something like:

MAXL> select {Measures.Marketing, Measures.Payroll} ON COLUMNS,
2>           {Market.East, Market.West, Market.South} ON ROWS
3> from Sample.Basic
4> where {Scenario.Budget};

Axis-1              (Marketing)         (Payroll)
+-------------------+-------------------+-------------------
(East)                            11210                7100
(West)                            16800               11660
(South)                            6120                5090

In OBIEE 11g, an equivalent analysis would look like this, using a pivot table view:

The criteria to create such an analysis would look like this:

Now this criteria is relatively simple, as we’ve selected measures that are within the same generation (Marketing and Payroll, both within the third generation in the Measure hierarchy). In reality, your measures are likely to come from different measure hierarchy generations, which is the reason you’ll often flatten the measures in the OBIEE physical model, but I’ll come on to the why and wherefores of that in a moment. For now though, the criteria is fairly simple and results in the following MDX sent to the Essbase server.

With
set [_Market2]  as '{Distinct({[Market].[East], [Market].[South], [Market].[West]})}'
set [_Measures4]  as '{Distinct({[Measures].[Marketing], [Measures].[Payroll]})}'
select
{ [Attribute Calculations]
} on columns,
NON EMPTY {crossjoin({[_Market2]},{[_Measures4]})} properties GEN_NUMBER, [Measures].[MEMBER_UNIQUE_NAME], [Measures].[Memnor], [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]
where ([Scenario].[Budget])

OK, not so bad. It’s a variation on what we coded by hand (using WITH blocks to define the set of rows and columns upfront, and a crossjoin to return the product of the two dimension member sets, but it’s recognizable as being more or less equivalent to our hand-coded MDX.

So what if we try a simple calculation? Let’s add the Marketing and Payroll measures together to create a new measure called Direct Costs. In MDX, I’d write this as:

MAXL> with member Measures.[Direct Costs] as
2>            'Measures.Marketing + Measures.Payroll'
3> select {Measures.Marketing, Measures.Payroll, Measures.[Direct Costs]} ON COLUMNS,
4>           {Market.East, Market.West, Market.South} ON ROWS
5> from Sample.Basic
6> where {Scenario.Budget};

Axis-1              (Marketing)         (Payroll)           (Direct Costs)
+-------------------+-------------------+-------------------+-------------------
(East)                            11210                7100               18310
(West)                            16800               11660               28460
(South)                            6120                5090               11210

So let’s try and create this now using OBIEE. Going back to the analysis criteria, I look for a way to add the two measures together.

OK, so this is interesting. There’s no Marketing or Payroll measure to select and use as part of an expression, as the measure name is actually a dimension member and there’s only one measure as-such. This is the drawback with bringing your measure dimension into OBIEE – you don’t then have individual measures you can select from to include in an analysis, create or include in calculations, and so on.

Now one way around this is to use the new “selection steps” feature in OBIEE 11g to calculate this value post-aggregation, after the initial filtering has happened and using the resulting result set. I therefore create a new selection step, like this:

and Bob is indeed your uncle.

So apart from figuring out the selection step move, this was actually quite straightforward. So how does the MDX look?

With
set [_Market2]  as '{Distinct({[Market].[East], [Market].[South], [Market].[West]})}'
set [_Measures4]  as '{Distinct({[Measures].[Marketing], [Measures].[Payroll]})}'
set [_Measures4FILTER1]  as '{Distinct({[Measures].[Payroll]})}'
set [_Measures4FILTER2]  as '{Distinct({[Measures].[Marketing]})}'
member [Attribute Calculations].[_MSCMFILTER1] as 'AGGREGATE(Intersect([_Measures4FILTER1], {[Measures].CurrentMember}))', SOLVE_ORDER = 102
member [Attribute Calculations].[_MSCMFILTER2] as 'AGGREGATE(Intersect([_Measures4FILTER2], {[Measures].CurrentMember}))', SOLVE_ORDER = 102
select
{ [Attribute Calculations],
[Attribute Calculations].[_MSCMFILTER1],
[Attribute Calculations].[_MSCMFILTER2]
} on columns,
NON EMPTY {crossjoin({[_Market2]},{[_Measures4]})} properties GEN_NUMBER, [Measures].[MEMBER_UNIQUE_NAME], [Measures].[Memnor], [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]
where ([Scenario].[Budget])

OK, so what’s going on here is the situation Venkat described back in some earlier blog posts, where selection steps involves the BI Server generating MDX that calculates lots of additional values which it then combines back together to give you your result set. Running this MDX statement through the MAXL command shell gives us this output:

Axis-1              Axis-1.properties   (Attribute Calculat (_MSCMFILTER1)      (_MSCMFILTER2)
+-------------------+-------------------+-------------------+-------------------+-------------------
(East, Marketing)   (GEN_NUMBER = 2, ty               11210            #Missing               11210
(East, Payroll)     (GEN_NUMBER = 2, ty                7100                7100            #Missing
(South, Marketing)  (GEN_NUMBER = 2, ty                6120            #Missing                6120
(South, Payroll)    (GEN_NUMBER = 2, ty                5090                5090            #Missing
(West, Marketing)   (GEN_NUMBER = 2, ty               16800            #Missing               16800
(West, Payroll)     (GEN_NUMBER = 2, ty               11660               11660            #Missing

The BI Server takes this result set, plus some additional member information it gets via another MDX query, and adds the results together as per the selection step calculation to give you your direct costs calculation. So whilst it’s very easy for the end-user to create their calculation (and in the case of using selection steps, just as easy to select measures from different measure hierarchy generations), but it’s not actually Essbase that’s doing the calculation, its the BI Server (and in fact Essbase is doing a lot of extra work to return all these subtotals, at least compared to my hand-written MDX).

An alternative to using selection steps though, would be to flatten the measure hierarchy into a flat list of measures, the same way that OBIEE 10g did by default when importing Essbase outlines. How does this affect the MDX that’s generated for a calculation like this?

I start then by duplicating the Essbase database in the OBIEE repository physical layer, and then select the option to flatten the measure hierarchy, giving me a physical model looking like this:

Now creating my calculation in the analysis criteria is really easy, as I’ve got individual measures for the Marketing and Payroll members, which I can just add together to create a new measure in the criteria.

So how do the numbers look? One advantage of working with a flattened list of measures, is that you can use a regular table view to display the results, rather than having to use a pivot table view as you do with a measure hierarchy. Taking a first look, the figures look correct:

What about the MDX? Well, it’s actually a lot closer to the MDX I wrote by hand; instead of adding the two measures together, it creates a set of the two (giving the same result, in this instance), but other than that, it looks pretty close to the MDX I wrote.

With
set [_Market2]  as '{Distinct({[Market].[East], [Market].[South], [Market].[West]})}'
select
{ [Measures].[Payroll],
[Measures].[Marketing]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]
where ([Scenario].[Budget])

Just as a quick check, I create another analysis that randomly adds up, divides and otherwise applies simple operators to some measures.

Now if I were to write an MDX statement to retrieve numbers like this, I’d probably write something along the lines of:

MAXL> with member Measures.[Sales Divided by Marketing]
2> as 'Measures.Sales / Measures.Marketing'
3> member Measures.[Sales Plus Total Expenses]
4> as 'Measures.Sales + Measures.[Total Expenses]'
5> member Measures.[Sales times Margin]
6> as 'Measures.Sales * Measures.Margin'
7> select {Measures.Sales, Measures.Marketing, Measures.[Sales Divided by Marketing], Measures.[Sales Plus Total Expenses], Measures.[Sales times Margin]} ON COLUMNS,
8> {Market.children} ON ROWS
9> from [Sample.Basic]
10> ;

Axis-1              (Sales)             (Marketing)         (Sales Divided by M (Sales Plus Total E (Sales times Margin
+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------
(East)                            87398               14721    5.93696080429319              112708          4323666458
(West)                           132931               23529    5.64966636916146              173096          9308626206
(South)                           50846                8281    6.14007970051926               66504          1469246016
(Central)                        129680               19706     6.5807368314219              164544          9482979680

Looking at the MDX generated by the BI Server though, using flattened measures, gives us a bit of a surprise:

With
set [_Market2]  as '[Market].Generations(2).members'
select
{ [Measures].[Marketing],
[Measures].[Sales],
[Measures].[Total Expenses],
[Measures].[Margin]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]

All the BI Server does, then, is just request the basic values for measures from Essbase. The BI Server then performs the calculations, and you’ll find this happens all over the place. Where you would have assumed that the BI Server would ship-down calculations to the Essbase server, instead it just requests the basic data and performs the calculation in-memory, in the BI Server.

But if you particularly want to make use of an Essbase MDX function, there is actually a way you can do this, and it’s through the EVALUATE, EVALUATE_AGGR and EVALUATE_PREDICATE functions within OBIEE. These functions provide a way to pass through a request to the underlying datasource for an analysis to use a particular function, and you can use this approach to make use of Oracle database functions that either don’t have an equivalent within OBIEE, or perhaps you’ve written yourself. But you can also use this approach for working with Essbase sources, and it even allows you to do things that would be tricky to do in the ROLAP-world of OBIEE but are easy to do within a multi-dimensional database such as Essbase.

Take, for example, a situation were we want to list out, in an analysis, a list of markets together with sales amount AND the average sale amount for the states within each of the regions. In this case, getting the average sale amount for each region’s constituent states would actually be quite tricky with a regular relational source, as the dimension members your calculating the average for are at a different level of granularity to the one used in the query, but for Essbase this is easy. Let’s take a look at how we’d write this in MDX first.

MAXL> with
2>   set regions  as 'Market.children'
3>   member Measures.Children_AVG as 'AVG(Market.currentmember.children,Measures.Sales)'
4> select
5>   { Measures.Sales,
6>     Measures.Children_AVG
7>   } on columns,
8>   NON EMPTY {regions} on rows
9> from [Sample.Basic];

Axis-1              (Sales)             (Children_AVG)
+-------------------+-------------------+-------------------
(East)                            87398             17479.6
(West)                           132931             26586.2
(South)                           50846             12711.5
(Central)                        129680    21613.3333333333

Now moving over to OBIEE, what we want to use now is an EVALUATE_AGGR function, to leverage the AVG MDX function. This function takes two arguments:

AVG(set, numeric_value)

In other words, you pass it a set of members (or “tuples” in Essbase-speak), and a number to perform the average over. So now, if we want to use this in OBIEE to return an aggregated value, we need to substitute a dimension member reference and a measure reference which in turn will be passed down, along with the call to the AVG MDX function, to Essbase.

To do this I create a new logical column in the business model holding my Essbase data (the one with flattened measures), and edit the logical table source expression list to reference this function.

I also set the default aggregation type for this new column to AGGR_EXTERNAL, and then copy it across to the presentation layer subject area.

Now I can create an analysis that lists out the regions, their total sales, and then the average sale value for the states below each region, something that it’d actually be quite tricky to do with a relational source.

Taking a look at the MDX generated by the BI Server, it looks along the same lines as the one we wrote by hand:

With
set [_Market2]  as '[Market].Generations(2).members'
member [Measures].[_MSCM1] as 'AVG(Market.currentmember.children,[Measures].[Sales])', SOLVE_ORDER = 101
select
{ [Measures].[Sales],
[Measures].[_MSCM1]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]

Now what we’ve seen there is an example of an MDX function that returns an aggregated value, hence our use of EVALUATE_AGGR. But we can also return values that can be used as attributes for a dimension member, for example the count of immediate “children” under a dimension member. In this example, if we were to write a query in MDX that returned total sales for a region and also the number of states underneath it, the query would look like this:

MAXL> WITH
2> member Measures.Num_Children AS
3>   'count(Market.currentmember.children)'
4> select {measures.sales, measures.num_children} on columns,
5> {market.children} on rows
6> from Sample.Basic;

Axis-1              (Sales)             (Num_Children)
+-------------------+-------------------+-------------------
(East)                            87398                   5
(West)                           132931                   5
(South)                           50846                   4
(Central)                        129680                   6

To make use of this dimension member count in OBIEE, we’d need to use the EVALUATE function, and pass to it a reference to the currently selected dimension members’ children, and the name of the dimension that we’ll be calculating this for.

Taking a look at the MDX generated by the BI Server, it’s as you’d expect.

With
set [_Market2]  as '[Market].Generations(2).members'
member [Measures].[_MSCM1] as 'count([Market].Generations(1).dimension.currentmember.children)'
select
{ [Measures].[Sales],
[Measures].[_MSCM1]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]

So it’s a bit of a mixed picture with calculations with an Essbase source. On the one hand, OBIEE doesn’t really make much use at all of MDX functions, instead retrieving just the basic measure values from the Essbase database and performing the calculations itself, within the BI Server. In many cases this is fine, but it can trip you up if you’re expecting to offload processing to the Essbase server but instead, you find the BI Server is actually doing most of the work. In some cases there’s good reasons for the approach Oracle have taken – not all MDX functions work across all storage types, for example – but it’s worth being aware of this before you get too deep into working with Essbase.

The flip side of this though is being able to access MDX functions directly via the EVALUATE_AGGR and EVALUATE functions. These functions actually give you the ability to perform calculations, across levels and dimensions, that you would even be able to do with regular sources and OBIEE functions, and on most projects using Essbase sources you’re going to be using these functions alot, both to compensate for lack of push-down from the BI Server, but mostly to take advantage of the MDX functions and cross-hierarchy and level calculations that you can perform from Essbase.

There is one exception to all this though, and it’s an interesting one. The one type of OBIEE function that does get pushed down correctly to Essbase, is time-series functions. The AGO, TODATE and (new in 11g) PERIODROLLING do indeed get translated down to their equivalent MDX functions, which means that you can use the time-series calculations and wizard available in OBIEE to create all your time-based derived measures.

For example, consider a situation where you wish to calculate the value of sales for the previous quarter. In OBIEE, you’d create a time-series calculation looking like this:

Then your analysis would look like this:

Taking a look at the MDX generated by the BI Server, it does indeed use the PARALLELPERIOD MDX function.

With
set [_Year2]  as '[Year].Generations(2).members'
member [Measures].[_MSCM1] as '(ParallelPeriod([Year].[Quarter], 1,[Year].currentmember), [Measures].[Sales])'
select
{ [Measures].[Sales],
[Measures].[_MSCM1]
} on columns,
NON EMPTY {{[_Year2]}} properties GEN_NUMBER, [Year].[MEMBER_UNIQUE_NAME], [Year].[Memnor] on rows
from [Sample.Basic]

So, it’s been a long post, but what observations and rules-of-thumb can we draw therefore for Essbase sources and calculations?

• Measure hierarchies are great for end-users, particularly if they come from a traditional Essbase background. Coupled with selection steps, it’s easy to select measures from a hierarchy and perform calculations all from the Answers UI
• But measure hierarchies can often generate quite complex and costly MDX queries, and it’s often easier, as a developer, to work with flattened lists of measures. You can create these in a separate physical database model, or combine a measure hierarchy with a flattened list of measures in the same business model
• OBIEE out-of-the-box actually makes very little use of MDX functions in the Essbase database, instead choosing in most cases to just request measure values and then performing any calculations itself
• The exception to this is time-series queries, which do get pushed-down to their equivalent MDX function
• In some cases, you can compensate for this lack of push-down through the EVALUATE and EVALUATE_AGGR functions, but they don’t cover all situations and also limit you in terms of drill-down etc. EVALUATE and EVALUATE_AGGR also provide you with some extra functionality though, as they let you harness the ability of Essbase to calculate numbers across different hierarchy levels and dimensions.

That’s it for Essbase for a while now; keep an eye on the blog for a bit more on OBIEE 11g and Oracle OLAP, plus possibly some content on OBIEE 11g and Microsoft Analysis Services, to see how well support for Microsoft’s OLAP server compares to Essbase.

These two blog posts by Christian Berg back in 2010, based on OBIEE 10.1.3.4.1 and Essbase 9.3.1, describe the problem well and unless you were prepared to do some unsupported hacking with UDML and Notepad, working with an evolving Essbase outline was pretty much impossible with OBIEE 10g, and when 11g came out many customers re-defined their Essbase-derived physical model hierarchies as value-based to get around this problem.

But value-based hierarchies have their own issues, such as not being able to make use of UDAs or alias tables, or as outlined in this previous posting, not being suitable for vertical federation (drill-to-detail) with relational and file sources. So, it was with interest that I noticed a new section in the Essbase metadata modeling section of the OBIEE 11g documentation (11.1.1.3 and 11.1.1.5) entitled “About Incremental Import”, that says:

“You can choose to incrementally import Essbase metadata. In other words, you can perform an initial import, and then import again. You might want to import incrementally when information in the data source has changed, or when your first import only included a subset of the metadata. Note the following about incremental import:

• When you re-import metadata that already exists in the Physical layer, a message appears, warning you that your Physical objects will be overwritten.
• If you delete data in the source, re-importing the metadata does not automatically perform the deletion in the Physical layer. Instead, you must manually delete the corresponding Physical objects.
• If you rename an object in the source, the renamed object is imported as a new object. In this case, both the old object and the new (renamed) object are displayed in the Physical layer.
• In general, customizations that you have performed on the Physical layer data, such as determining the alias column to use for display, are retained after an incremental import. If you want to revert to the default imported view, you must delete the existing Physical layer objects and then re-import the metadata.”

So, how does this work, and how easy is it to implement? With this question in mind, I thought I’d give it a try.

To start off, I’ve imported a copy of the Sample.Basic Essbase database into the Physical layer of an Oracle BI Repository. Running this import creates a physical database named after the Essbase server (essdemo), a physical catalog named after the Essbase application (Sample), and a cube table named after the Essbase database (Basic). I performed the import using OBIEE 11.1.1.5 with all the import options left at their default value.

As you can see from the screenshot below, the Market physical dimension has three UDAs associated with it; Major Market, Small Market and New Market.

So, as my first test, what if I use Essbase Administration Services to add a new UDA value to the West Member called Development Market, a UDA value that’s new to the outline and therefor wouldn’t have been imported into the OBIEE repository in the past?

I save the updated outline and then go and re-run the Import Metadata wizard in the Oracle BI Administration tool. Selecting the same Essbase database brings up a warning asking me to confirm I want to re-import.

After running the re-import, I check the Market dimension, and yes, it contains the new UDA value.

But what if what the Administration tool has done, is just overwrite the old cube table definition with the new one? This could cause me problems if I’d fine-tuned the physical model previously by, for example, setting the hierarchy type for the Market dimension to Fully Balanced, or I’d taken the Measure hierarchy and flattened it into a flat list of measures.

So let’s perform these customizations on the physical model, by firstly setting the Market hierarchy type to Fully Balanced (I would choose Value, but you can’t access UDAs if you set your hierarchy to Value):

I then right-click on the cube table and select the Convert measure dimension to flat measures option.

So at this point, I’m looking to preserve the change I’ve made to the hierarchy type for the Market dimension, and also the new set of flattened measures in the cube table. Note that converting the measures to flat measures still leaves the Measure dimension in the cube table, and I would generally delete this from the business model after dragging and dropping the physical model into the Business Model and Mapping layer.

After saving the changed physical model, I then go back into Essbase Administration Services and add another UDA to one of the Market dimension members.

I then use the Import Metadata wizard in the Oracle BI Administration tool to bring in the updated outline. Checking the resulting updated cube table, the new UDA is there as expected.

The flattened measures are still there, but unfortunately, the Hierarchy Type for the Market hierarchy has been set back to Unbalanced.

So now, even thought I know that I can’t make use of UDAs when I set the Market hierarchy type to Value, I’ll set it to this anyway, and then go and add another UDA to the dimension, to see if it does preserve the Value hierarchy type setting after a re-import. I set the Hierarchy Type to Value:

I then add another UDA to the outline, re-import it using the Import Metadata wizard, and interestingly find that the Value hierarchy-type setting has been preserved. If I then switch the Hierarchy Type back to Unbalanced, Fully Balanced or any other level-based hierarchy type, I can then also access the new UDA that was added to the outline.

So the rule here seems to be: if you’ve added UDAs to your outline, and you re-import the outline using OBIEE 11.1.1.5 (I’ve also checked this for 11.1.1.3, it’s the same behaviour), existing changes to the outline are preserved, except Hierarchy Type settings where all level-based hierarchy types are reset back to Unbalanced. Any changes you’ve made to flatten the measure hierarchy, for example, are preserved OK.

So what about aliases? If we add a new alias table to the Essbase outline, and use it to add a new alias value to, for example, the East member, what happens?

At the moment, if I view the list of Alias tables imported into the cube table definition, I can see the standard Member_Name, Default and Long Names aliases.

Now again, as with UDAs, you can’t access alias tables when you set your Essbase-derived physical hierarchy type to Value, so I set it back to Fully Balanced and then import the updated outline.

As with UDAs, the new alias table has been brought in with the re-import, the hierarchy type has been reset back to Unbalanced, but other that that, the rest of the cube table definition seems unchanged.

So what about something a bit more drastic? What if one of the hierarchies in your Essbase outline suddenly grew an extra generation (which would correspond to a new hierarchy level and set of cube columns in the OBIEE physical layer), which in 10g would mean you were pretty-much out-of-luck. To test this out, first I add a new generation to the Scenario hierarchy, adding Initial Budget and Revised Budget members as children to the existing Budget member. I save the outline and restructure the cube, in preparation for importing into the OBIEE repository.

At the moment, the Scenario dimension and hierarchy within the cube table has two levels, corresponding to the two generations in the Essbase hierarchy.

I’ve also created a corresponding business mode, and subject area, off of this cube table, so that I can see how easy it’ll be to incorporate the additional levels into already-derived business models and subject areas.

So, here we go. I use the Import Metadata wizard to re-import the database definition into the physical layer, and check out the updated physical model.

And there we have it. The new child members in the Scenario dimension hierarchy have caused a new level to be created in the physical model, something that (unless i’m much mistaken) wasn’t possible in OBIEE 10g. And this is quite a big deal – for many of our customers, not being able to refresh their physical model off of a changed outline was a showstopper, and it’s been the major driver towards adoption of value-based hierarchies in OBIEE 11g. The fact that this is now possible is actually quite big news, and I’m surprised the dev team within Oracle responsible for OBIEE/Essbase integration hasn’t publicised this – I only came across it when trawling through the online docs whilst writing the Essbase chapter of the OBIEE 11g book.

So how easy it is to incorporate these changes into  your business model, which remember is automatically generated for you when you drag and drop your cube table into the business model and mapping layer of the Oracle BI Repository. To find out, I start by dragging three of the four cube columns that are contained within the new Gen3, Scenario physical level across to the correspondng business model logical table, like this:

Then I add this additional key column to the logical table key, like this:

Next, I add a new child level to the existing Scenario logical dimension, and add these three logical columns to the level.

Now I add the logical level keys, copying how they are set up for the other levels.

Then, to finish off the business model changes, I set the detail (content) level for the logical dimension and logical fact table sources, to reflect the additional level that’s in the Scenario dimension.

Finally, I update the subject area in the physical model, deleting the old presentation hierarchy within the Scenario presentation table and replacing it with the updated one from the business model and mapping layer.

Now I can verify the updated repository, and then create a sample analysis. First off, I create an analysis that just includes the updated Scenario dimension, to check that the new measure comes through as expected.

You can see the two new scenarios listed under the Budget member. Now I add a measure into the analysis, which should lead to these two new members being removed, as no data has been recorded against these scenarios.

Very good. The hierarchical column indicates that there are members under the Budget member, but if you try and click on the “expand” icon, nothing happens.

So we’ve shown that we can import new generations into the repository, and then feed these through to the business model and presentation layers, so that users can make use of the new members at this new level. Interestingly, if you then remove these two new members so that the hierarchy goes back to the original number of generations, and then re-import the outline back into the repository, the Administration tool removes the additional generation from the physical model, so you’re back to two generations again (contrary to what the manual says, actually). You’ve then got to go back and adjust your business model and subject area to reflect the new number of generations, but at least it’s possible to delete generations this way, as well as add them.

But what about something bigger – say, adding a dimension to an Essbase database? Given what we’ve seen so far, it’s likely that the incremental import from the outline will pick up the new dimension, but is it then possible to model it in the business model and subject area?

To test this out, I create another copy of the Sample.Basic Essbase database, delete the Population attribute dimension, and then import it into the Oracle BI Repository using the BI Administration tool. For good measure, I change the Market dimension hierarchy type to Fully Balanced, and convert the measure dimension to a flat list of measures.

I also create a corresponding business model and subject area for the database.

Then, I restore the missing attribute dimension to the source Essbase database (by dropping it, and then re-copying it from Sample.Basic using Essbase Administration Services), and then use the Administration tool to re-import the outline again. How does it work out?

Well, as you can see, the Population attribute dimension has been imported incrementally, and the measure dimension is kept as a flat list. Interestingly, the Market dimension’s hierarchy has been reset back to Unbalanced, which indicates that the re-import process sets all the level-based hierarchies back to Unbalanced regardless of whether the hierarchy has any other changes. So how easy is it to introduce this new attribute dimension to the business model?

Dragging and dropping the new attribute dimension from the Physical into the Business Model and Mapping layer adds two items to the business model; a logical table for the cube columns, and a logical dimension, for the physical hierarchy and physical levels. Notice how the logical table has the “fact’ indicator in it’s icon? To remove this, I create a logical join between this new logical table and the fact logical table, like this:

I also set the content level for the fact table logical table source, to reference the detail-level from the new Population logical dimension hierarchy.

Finally, I copy the new logical table over to the Presentation layer of the repository and add it to the existing subject area, along with it’s logical dimension. Now, to try it out in an analysis.

So there you go. It all appears to work fine. So to me, it looks like the limitation that we had with OBIEE 10g, where we couldn’t incrementally refresh a physical model in the repository based off of an Essbase database, is no longer the case. There’s a couple of gotchas – level-based hierarchies, even for those that haven’t been touched – get reset back to Unbalanced from any other level-based hierarchy setting, and the actual process of amending the business model and subject area are a bit tricky (at one point, I kept hitting a drilling error with the new scenario members and generation, but after a couple of restarts and re-imports it just started working).

But if your concern about working with OBIEE and Essbase is not being able to handle outline changes, and value-based hierarchies have other restrictions that you can’t otherwise work with, I’d say this is worth taking a look at. This was certainly an eye-opener for me, and I can’t help wondering why Oracle haven’t made more noise about this, apart from burying the news hidden away in the documentation. Strange, but good news nonetheless.

At the time I said:

“Fast forward to 2009, and if I was recruiting an Oracle Business Intelligence developer now, what would I be looking for? Ideally, most if not all of the following product skills:

• Oracle Business Intelligence Enterprise Edition, usually with a specialization in either the back-end (the BI Server) or front-end (Answers, Delivers, BI Publisher), would be mandatory
• Essbase skills would be very desirable, and would be required far more than Oracle OLAP skills in the past.
• Oracle BI Application experience is a definite bonus, with the real skill in being able to apply customizations to the ETL and RPD layers
• Another very desirable skill is being able to implement at least Hyperion Planning, and ideally some more of the Hyperion financial applications. It’s rare though to find people with both good OBIEE and Hyperion skills, those that do can virtually write their own salary cheque.
• In terms of ETL, OWB is now “mainstream” and is a mandatory skill. ODI is rising in importance, but I don’t see many “classic BI” projects that use it, at least at the moment.

Compared to five years ago, skills in the Oracle database, and in particular with the Oracle Application Server, are less of a priority, as security for example is usually handled now by the OBIEE BI Server, and fast access to data is typically done through Essbase. Also, now that Oracle’s BI tools (and Fusion Middleware) are designed to work across heterogeneous platforms, its more useful now for developers to have experience with Active Directory, for example, compared to Oracle Internet Directory, and it’s also useful to have knowledge of other database platforms such as Microsoft SQL Server, Microsoft Analysis Services and the like as Oracle’s BI tools are increasingly being sold to non-Oracle database customers.”

So how have things changed, now it’s 2011? Well, some things are much the same; OBIEE, if you’re an Oracle BI developer, is a mandatory skill, and as back in 2009, you tend to specialize in the backend (RPD development, possibly Informatica if you’re an Oracle BI Applications developer), or the frontend, creating the analyses and dashboards that are most visible to the end users. Take away all of the stuff about WebLogic, Enterprise Manager and Fusion Middleware, and the core of OBIEE hasn’t changed much since the days of Siebel Analytics, and we typically see the same old problems, misunderstandings and so on with RPD development that we’ve seen since first transitioning to the technology back in 2005. Expect things to change fairly rapidly in the next couple of years though, with a new Administration tool, possibly a new (or alternative) XML or database-based repository, and a lot of work going on within Oracle to make the whole process of developing and deploying repositories a lot more automated.

One skill that an Oracle BI developer will need, and that wasn’t such a requirement back in 2009 (even, in my view at the time, becoming a less relevant skill), is application server skills, specifically Oracle WebLogic Server. For the time being at least, if you’re deploying OBIEE 11g, you’re deploying WebLogic Server, and having skills in such areas as WebLogic security, WebLogic Scripting Tool, the WebLogic Admin Console, and technologies such as admin and managed servers, node managers and OPMN is pretty much a requirement if you’re going to be an end-to-end OBIEE 11g developer. You could pretty-much install OBIEE 10g and forget about it, but nowadays you need to have some systems administration skills if only to get the product installed and running.

Another skill I downplayed in 2009 but is increasingly relevant now, is security. OBIEE 10g took care of security and connectivity to LDAP servers from within the Administration tool, but now you need to know Oracle Platform Security Services, application roles and policies, and all the associated technologies around WebLogic’s implementation of security. In fact, one major skill you need now is the ability to read and digest reams of documentation, made all the worse by OBIEE’s content being folded in with the rest of the Fusion Middleware product line, making it tricky to pick out how, for example, you can connect OBIEE to third-party LDAP servers, or how to perform upgrades between releases of 11g. It’s all a lot more complicated now, and you can’t just charge in, ignore the documentation and not do your homework.

Essbase was something I highlighted back in 2009, and at the time, it seemed fairly imminent that all Essbase projects would use Oracle BI technology, and all Oracle BI projects would include an element of Essbase. Now, i’m not so sure; Essbase is going gangbusters and if you’re an Essbase specialist, you can more or less name your price, but for OBIEE projects, we see a bit of Essbase, but it’s not gone mainstream yet. Realistically, whllst OBIEE 11g solved a number of 10g-era problems with OBIEE/Essbase integration, it’s just introduced another set of new ones, and I don’t see many WebAnalysis, Financial Management or Interactive Reporting customers moving to OBIEE unless their application was pretty-much relational in form, in the first place. If you’ve got Essbase skills, it’s certainly a bonus, but unless you’re prepared to get to expert level, you’ll probably find most Essbase projects are handled by specialist Hyperion consultants, and your job is probably more likely to be raising SRs and working with the client to try and fix the OBIEE to Essbase connectivity issues.

I smile to myself when I look back at the comment on OWB. Certainly, in terms of new projects, OWB is dead now and every customers is moving towards ODI. Maybe it’s because it’s all that Oracle will sell on new projects, maybe it’s because customers don’t want to invest in products that have been sunsetted, or maybe it’s because ODI is better, but nowadays, you need to know ODI if you’re an OBIEE developer. This means knowing things like how to create knowledge modules, how to migrate projects from environment to environment, and how to use new features such as ODI OBIEE data lineage. But if you’re an OBIEE developer and you’ve got time to learn one more skill, make it ODI.

And so finally on to the Oracle BI Applications. My experience has been that most BI Applications projects have been handled by the big ERP integrators, occasionally calling on specialists if the OBIEE element goes wrong. Either you’re an RPD, DAC and Informatica developer, or you’re an OBIEE, RPD and OWB/ODI developer, and if you’re the former you’re probably a contractor or working for a large SI. The major bonus skill that you’d have as a BI Apps developer is knowledge of either Siebel or Oracle E-Business Suite as a data source, but all of this is going to change in the next couple of years as Oracle transition to the Fusion Applications, with the 11g release of the BI Apps focusing on the Fusion Applications first and then only with support for Apps Unlimited (basically, all the legacy ERP suites Oracle developed or acquired) coming later on. The big question though, at least in my mind, is whether you can still be a BI Apps developer in the future without also being a Fusion Apps developer, and a Fusion Development (ADF, JDeveloper, SOA Suite) developer; the 11g BI Apps will be embedded in the Fusion Apps, and whilst it’s possible to develop for the 7.9.x release of BI Apps as a standalone piece of technology, I wonder how much you’ll be able to achieve in isolation from the Fusion Apps.

So, to summarize, and to update my 2009 comments about what skills an Oracle BI developer needs, in 2011 I think you need:

• OBIEE 11g skills, both in terms of new functionality (mapping, analyses, KPIs and Scorecards etc) and new infrastructure (WebLogic, EM, OPSS etc)
• A smattering of Essbase skills, focused mainly on the integration with OBIEE and Essbase (and the many workarounds and gotchas)
• Good ODI skills, both in terms of the basics, but also being able to write knowledge modules, integrate with OBIEE, deployment and migration
• Solid database skills – OBIEE gave the illusion through aggregates etc that database tuning was redundant, but time has shown it’s by far the biggest success factor in a project – get the database design and optimisation wrong, and your project is toast. You need to know partitioning, materialized views, index types, and increasingly, you need to get yourself on an Exadata project as customers are buying the technology but you can’t teach it to yourself at home
• BI Apps skills, but watch out for everything changing when BI Apps 11g comes out, and be prepared to learn the Fusion Apps and JDeveloper if you want to stay in the game
• Looking to the future, keep an eye on technologies such as in-memory (TimesTen), mid-tier caching (Coherence), plus technologies such as Business Activity Monitoring (BAM), “big data” (Hadoop, large data sets, NoSQL), complex event processing and maybe products such as Qlikview, just in case Oracle buys them, or at least to know what the competition are up to, or more importantly pitching to your boss

The other thing to bear in mind of course, if you’re an Oracle BI developer, is that you need to have great business, communication and data modeling skills. But that’s another topic in itself, and maybe one of my colleagues will have some views on that.

As promised, I’ve uploaded the slides from the session here, and you can also read the blog post behind the session, here. I’ve also uploaded the slides to SlideShare, and you can browse through them here (just refresh the page if the area below is blank):

Keep an eye out on the blog soon for a follow-up posting, where we go into how you can leverage the OBIEE 11g Action Framework to perform write-back to an Essbase cube, and to trigger the execution of an Essbase calc script.

If your interest is with BI & DW topics, there’s tons of great content at this event. On the Sunday, there’s a special Oracle-led BI Symposium with content from Adam Bloom (security, and the action framework), Jacques Vigeant (Scorecard & Strategy Management)  and Nicky Sanger (Answers new features), plus a development keynote from Balaji Yelamanchili, Oracle’s Senior VP in charge of all OBIEE development. If you read my posting following Balaji’s presentation on the OBIEE technical roadmap at Collaborate’11, you’d certainly want to come to his session. For my part, I’ll be in there at 9am and there right through to the end, none of these sessions are to be missed.

During the rest of the week, there’s other great content including the return of the one-and-only David Allen, this time talking about ODI(EE), Mike Durran with a must-see presentation on OBIEE scripting and internals, Holger Freidrich on BI Server vs. Database aggregation, Kevin McGinley, winner of the US BI Forum Best Speaker prize on OBIEE and Mobile BI, plus a couple of sessions by myself and Stewart Bryson. In addition, if you’re interested in seeing what goes on in the world of Hyperion and Essbase, there’s a massive conference-within-a-conference going on led by such Essbase luminaries as Edward Roske and Tim Tow, with the combined EPM + OBIEE content area being led by Natalie Delamar.

I’ll be over in Long Beach from the Friday and then taking part in the conference from Sunday onwards. This is the last major BI event Stateside until Open World, so if you’re looking to dig deep into the OBIEE 1g platform, make sure you register now before the last discount ends on Thursday!

The example that we will be using here involves 2 reports. One report created in BI EE and another report created in Hyperion Financial Report. The idea is to use the Action Framework of BI EE 11g to pass the parameters from the BI EE report to the Hyperion Financial Report.

The BI EE report is shown below. This is a very simple report containing the Yearly sales data from the Demo > Basic Essbase cube. The idea is to navigate to a HFR report by clicking on any Sales number and at the same time passing the context to the HFR report.

The target FR report is given below. This report also goes against the Demo Basic Essbase cube with Year dimension acting as a User POV.

To enable the integration, we need to first setup the ability in BI EE to navigate the EPM catalog. This is done by adding a registry entry in the ActionFrameworkConfig.xml file. This file can be found under {Middleware_Home}\user_projects\domains\bifoundation_domain\config\fmwconfig\biinstances\coreapplication. Open this file in a text editor and add the following XML registry tags

<registry>
<id>HDPreg</id>
 <name>Hyperion Directory Provider</name>
 <content-type>epm</content-type>
 <provider-class>oracle.bi.action.registry.epm.HDPRegistry</provider-class>
 <description>Hyperion Financial Reports Registry</description>
 <location> <path>http://venkatepm:19000/raframework/browse/listxml</path> </location>
 <service-access>
 <account>EPM</account>
 <propagateIdentity>false</propagateIdentity>
 </service-access>
 </registry>

Remember the PropagateIdentity statement determines how the authentication/authorization is done. This is a must and without this tag, the EPM navigation will not work. Also note down the path of the EPM catalog listxml. In EPM 11.1.2, the listxml url has changed to point to the RA Framework. Ensure that while installing EPM, RA Framework is also chosen.

In the above registry XML entry, you will notice that we are using an account called EPM. This is nothing but a pointer to an actual account in the Weblogic Credential framework. So, we need to navigate to the Credentials page in the BI EE FMW Control (http://localhost:7001/em) and setup the EPM account.

In the Credentials page, create a new Credential Map called oracle.bi.actions (if it does not exist already).

Within this Credential Map, create a new key called EPM. In the username and password section of the EPM key, enter the admin username and password that will be used to connect to the EPM catalog(in my case hyperion/welcome123). This username and password will be used only for the catalog navigation.

Now restart the entire managed server so that BI EE can pick up the changes. As you see, the entire process of integration between EPM and BI EE (from BI EE) has been made simpler and also now we can add as many registry entries as we want, depending on the number of EPM targets that we want to navigate to.

After the restart, create a new BI EE report as shown below

Go to the column properties of the Sales measure and add an Action Link navigation.

Create a new Action Link and choose Navigate to EPM content as the target.

This will basically open up the EPM catalog. Lets choose our target FR report.

This will automatically open up a parameter screen where most of the workspace specific parameters will be filled in automatically

Add a new Parameter called Year which is what we will be passing from BI EE to HFR. Make the Year value to be passed from the Year column

In the URL field, enter the URL given below

http://venkatepm:19000/workspace/index.jsp?module=@{module}&repository_path=@{repository_path}&elementType=@{elementType}&repository_name=@{repository_name}&repository_format_id=@{repository_format_id}&run=@{run}&attribute=Year.id.@{Year}&attribute=Product.id.Product&attribute=Accounts.id.Accounts&attribute=Scenario.id.Scenario&attribute=Market.id.Market&mimetype=@{mimetype}

Basically the CONTEXT values can be made to be passed automatically from BI EE to HFR by setting up the dimension mapping in the FR.properties file(we shall see how this works in a future blog entry). But in our case above, we are passing the parameters manually.

Now we should have the action link enabled on the Sales column. By clicking on the FR Report link, the parameters will get passed from BI EE to HFR automatically. With the above approach, there is no SSO between BI EE and EPM. So, the username and passwords will have to be entered again.

If the SSO is enabled, SSO_TOKEN can be passed across BI EE and EPM so that the navigation is seamless.

As you see, the Qtr1 value that was passed from BI EE has now set the context in FR. The same can be done for Planning forms, Web Analysis reports etc. I will cover the Planning forms integration with BI EE in a future post.

“Using Oracle Essbase Release 11.1.2 Aggregate Storage Option Databases” takes a look at the new calculation capabilities in Essbase 11.1.2, in particular looking at how data can be aggregated, derived and allocated within 11.1.2 Essbase cubes and persisted in the database, opening up ASO databases to the kind of planning and budgeting applications previously restricted to BSO databases. If you’ve heard that aggregate storage is the strategic direction for Essbase but wondered how it could handle the sorts of calculations provided by block storage, you may well find this article interesting.

1. Dynamic References using CurrentMember, CurrentTupule etc are not supported currently (necessary if we want to run multiple allocations within the same dimension)
2. Hierarchical references like PARENT etc are not allowed in both calculations & allocations. These references are allowed for the parameters but not for the assignment scripts/allocation amounts.

The above 2 are significant drawbacks which i hope will be supported in future releases. Lets now look at a very simple example for allocation. I will be using the ASO version of the Demo->Basic cube for demonstrating this. The screenshot below shows the data that has been loaded into the cube.

If you notice, we have data for the Sales measure for all Regions except South. The idea is to allocate data into the South Region (both Sales & Cogs equally) based on the Total Expense values of the East Region. This is a very simple allocation example where our input data is at Level-0 but we are allocating from non Level-0. One good thing about this example is the fact that this shows that though ASO does not support input to non level-0 members, it supports allocation from them.

An Allocation in a ASO cube does not require a separate calculation script. All allocations are done through MaxLs directly. An allocation MaxL contains four main parts (in addition to other parameters which i shall not cover here).

1. POV – POV or a Point of View provides execution context to an allocation. A POV allows only reference to level-0 members. For every combination in the POV, the allocation will be executed once. POV is specified in the form of a MDX-Set.

2. Amount – This can contain a static member, Tupule or a constant. This does not support MDX expressions. In our example above, the amount will the (Total Expenses,East) Tupule. Ideally, Amount & POV should have all the dimensions in the cube.

3. Target – Target is specified in the form of a MDX Tupule. Basically it represents the region that will act as a target for the allocation. In our case, this will be empty.

4. Region – Region is a MDX set expression. This represents the target region where the data will be allocated. The dimensions specified in POV, Target and Region should be mutually exclusive as they all combine to form the target region. In the example above, Children(Total Expenses) & the Children (South) will form the Region.

5. Basis – This is commonly used when we are doing member based allocations. In our case, since we are using dividing the values equally (Spread allocation), this is optional.

So the final formula that i shall be using for achieving this is given below

execute allocation process on database DemoASO.Basic with
pov "Crossjoin(Descendants([Year],[Year].levels(0)),
Crossjoin(Descendants([Product],[Product].levels(0)),
Descendants([Scenario],[Scenario].levels(0))))"
amount    "([East],[Total_Expenses])"
target    ""
range        "CrossJoin({[Sales],[Cost_of_Goods_Sold]},
Descendants([South],[Market].levels(0)))"
spread;

As you see, Essbase has generated the necessary cells to accomplish this. As i mentioned last time in the Calculation blog post, the allocations/calculations are done separately and then the updated cells are pushed through external data load buffers. So, you would start noticing cells being loaded into incremental slices.

And if you look at the data, the South region would now have data with the values uniformly distributed.

Though this does work there are some inherent drawbacks. As ASO does not accept data at non level-0 members, the way allocation maxL’s are designed it makes it very tough(though possible) to do hierarchical allocations within the same dimension. Hopefully the introduction of dynamic member references and member functions should enable that in the future.