HPCM primarily has 3 main sweet spots

1. Every company will have indirect costs. For example, a Consulting company where Revenue is obtained through driving projects will have a lot of indirect costs like HR Costs, Admin Costs etc. HPCM provides an ability to allocate the costs back to the projects so that proper project profitability is derived. How the costs are allocated will be defined through HPCM itself. For example, if a company is running say 3 consulting projects with 20, 30 & 40 resources each, then the indirect costs are allocated back to the project based on the number of people(or time logged etc) in each project.

2. Every company will be storing the incoming Revenue & Costs in a Ledger. Due to various reasons, even the direct costs & revenue might not actually be tied back to a project (Consulting company example above). So, there might be a need to allocate the project based Direct Costs & Revenue as well to the Projects (allocation possibly by head count etc).

3. Allocation of Costs is pretty dynamic in nature depending on the type of business. It can vary quite frequently. So, the key is to ensure that the allocation logic can be changed frequently and easily. In addition, one more key point is to find the lineage back to the source on how the costs are obtained.

HPCM provides all the 3 above. If you are an Essbase or a Planning person, you could argue that we can do the same thing using these 2 products itself. Though true, in many cases Cost & Revenue allocation rules are defined by Functional Users. So, it is not possible for Functional users to create Business Rules/Calculation Scripts every time there is a change. In addition though Essbase is very good, it is very difficult to do a data lineage from a calculation script, to find out how the costs are allocated. Thats the main reason why, HPCM is a solution positioned primarily at Business/Functional users for providing that cost & Revenue Breakdown.

Though i have mentioned that HPCM is a functional tool, its underlying technology is very interesting. It has a relational metadata that stores the metadata related to HPCM. In addition each HPCM application will have 2 Essbase databases. One is Block Storage cube which will be used for the allocation & calculations. The other is a reporting Aggregate Storage cube which will be used for reporting. Data push from BSO and ASO is automatically available out of the box from HPCM. Also, one important point to note, change to dimensions, change to metadata, pushing data from BSO to ASO are all achieved within HPCM without writing any external code/scripts. Everything is done out of the box. This architecture is shown below.

In addition, the most interesting aspect of HPCM is the way it handles dimensions. This is what we will be covering in this article today. HPCM uses EPMA for managing its dimensions & attributes. HPCM as an application has 3 types of dimensions

1. System Dimensions – There are 2 System Dimensions – Measures & AllocationType. Generally while creating a HPCM application through the Wizard we can pre-create these 2 dimensions. AllocationType is used by HPCM internally for doing allocations. It is generally not needed to make any changes to this dimension. But Measures dimension is the most important dimension that HPCM uses for pushing costs & allocating them. We can create custom members in the Measure dimension if needed.

2. POV Dimensions – HPCM supports upto 4 point of view dimensions. These dimensions are generally for storing Year, Period, Scenario & Version. True to their names, they generally are used as POVs and are not used directly in any allocation (the POVs are always fixed in the calculation scripts).

3. Business Dimensions – Business Dimensions are those dimensions where allocations happen. These dimensions drive the allocation logic.

In addition HPCM also supports alias and attribute dimensions. For this article, i will use a simple case of demonstrating how to go about allocating HR Costs in a Consulting Company recognising its revenue through Projects. Lets make an assumption that on a monthly basis we record the HR Costs (including Salary paid to HR, other misc costs) etc. Lets also assume that we have 3 projects running in the company with the following break-up

a. Project A – 300 people full time
b. Project B – 500 people full time
c. Proejct C – 200 people full time

We will start off with creating the application through the Application Wizard (pre-create System Dimensions) and then we shall define the necessary dimensions.

We basically have 2 business dimensions – one for Accounts which will hold the HR Costs. Then we have the Project dimension which will record the revenue and costs specific to the project.

Lets deploy this application and then login to the application through Workspace.

In HPCM all allocation happens through stages. Stages is where allocation happen. HPCM supports uppto 9 stages. Each stage also supports intra stage allocation. Lets try to understand what this means from a Multi-Dimensional Essbase database Standpoint. In our example, we will have 2 stages. The first stage will have just the Accounts dimension – basically HRCosts in Accounts dimension will flow from Stage 1 to Stage 2 and will get allocated in Stage 2. So, Stage 2 will contain both the Accounts and Project dimensions.

After creating these 2 stages(ensure you also have a POV defined) lets go ahead and deploy this to Essbase from the Manage Database screen(both Calculation & Reporting Database). What you will notice is 3 things

1. There will be 2 essbase databases one suffixed with letter C and the other suffixed with letter R. C database is the Block Storage database that is used for allocation. R database is the Aggregated Storage database that will be used for reporting.

2. You will notice that for each stage there will be a corresponding set of dimensions prefixed by the Stage prefix given at the time of creation. So effectively, if there are 2 stages with 2 dimensions each, then Essbase will have 4 dimensions (though the 2 dimensions might be the same in EPMA).

3. You will also notice that each dimension will have a dummy member called NoMember. This is one of the most important members that controls the grain of the data. This member is the key in loading multi-grain data for allocation into HPCM.

Now that we have the Essbase cubes deployed as well, lets try to understand how the allocation logic works. To begin with lets assume for the month of Jan 2011, the HR Costs for the Consulting Company is say 1000 USD as shown below

This is the input data into Stage 1. So to load this in we will have be creating a text file and load it directly into Essbase. There are 3 options to load data into HPCM

1. Manual Data Entry – HPCM provides a screen where we can update data manually.

2. Staging Tables – We can load the data temporarily into a set of staging tables, and then from within HPCM we can push the data from Staging tables into Essbase.

3. Directly loading data into Essbase

In our case, we will load the data directly into Essbase as that will give more clarity on how HPCM works. For doing data load for Stage 1, remember we have a total of 9 dimensions in Essbase (Measures,AllocationType,Year, Scenario, Period, Version, ACAccount,PRAccount,PRProject). But our input data of HRCosts comes at a grain of only 7 dimensions (Measures,AllocationType,Year, Scenario, Period, Version, ACAccount). So, load this in we will have use the NoMember intersection of the remaining 2 dimensions (PRAccount & PRProject). The input data file to Essbase is shown below

CostInput,DirectAllocation,2011,Jan,Actual,Working,HRCosts,[PRAccounts].[NoMember],[PRProgram].[NoMember],1000
Our idea is to allocate the 1000 USD down to the 3 projects for the January Month. So, our end result should look like this

If you notice, the 1000 USD is split across the 3 projects based on the overall number of resources(Resource Count for each project/Total Resources*HRCost) in each project. So basically for the allocation to happen, we need to load the Resource Count data. Resource Count data for all projects and the individual projects have to be loaded as shown below

FixedDriverValue,DirectAllocation,2011,Jan,Working,Actual,HRCosts,HRCosts,Project A,300
FixedDriverValue,DirectAllocation,2011,Jan,Working,Actual,HRCosts,HRCosts,Project B,500
FixedDriverValue,DirectAllocation,2011,Jan,Working,Actual,HRCosts,HRCosts,Project C,200
Weight,DirectAllocation,2011,Jan,Working,Actual,HRCosts,[PRAccounts].[NoMember],[PRProgram].[NoMember],1000

In the above data file there are 2 things we can notice

1. We have used a measure called FixedDriverValue and Weight. We will see their significance a little bit later. For now think of them as intersections that will hold the Resource Count Data.

2. The first 3 records above have HRCosts repeated twice to load into both the Accounts dimension we have in the Essbase Cube. Again we will see the significance of why we are doing this below.

So far we have loaded all the necessary data into the Essbase Cubes and have also setup the stages. The next step is in defining the Allocation Logic. This is done through a concept called Drivers. From the Perspective of HPCM, drivers define how the allocation values get pushed from source to target. HPCM supports different allocations like Activity Based Costing etc through the concept of Drivers. In our case, the Driver for allocation is the Resource Count. Just to recap, within Essbase now we have dimensions catering to two stages – Stage 1 and Stage 2. This is illustrated below. So basically we have 3 separate sub-cubes each having its own intersection.

In the above diagram, whenever we want to do allocation there can be different ways of doing it. The 3 most common ways are

1. Allocation Based on Source – Here all the driver values are obtained from the source stage and data from source stage is then assigned to the Destination Stage based on the Source driver values.

2. Allocation Based on Destination – Here all the driver values are obtained from Destination stage and data from source stage is then assigned to the Destination Stage based on the Destination driver values.

3. Allocation Based on Source & Destination – Also known as Assignments – Here all the driver values are obtained from the intersection of Source & Destination and data from source stage is then assigned to the Destination Stage based on the Assignment driver values.

If you look at our Driver data above (3 records containing FixedDriverValue & 1 containing Weight), you can see that our driver (Resource Count) is loaded in 2 ways. First the FixedDriverValue is loaded at the intersection of Source & Destination stages. The second driver Weight is loaded at the Source Stage intersection. We have chosen these 2 measures (FixedDriverValue & Weight). We could have chosen any other measure like Rate, Volume etc. But each measure has a logical meaning and it is better to stick to the ones we think is logically close to what we are trying to do. If we are not able to use the existing measures and if they don’t relate to our driver names then we can create custom measures. So, this allocation handling through drivers is defined through the Driver Definition screen in HPCM.

As you see in our formula we are basically doing a division of each Resource Count from Stage 2 by the Total Resource Count in Stage 1. HPCM will automatically multiply the resulting values of the driver to the CostInput measure (input data) thereby allocating.

Once we have defined the driver the next step is to assign the driver to both the stages. HPCM allows us to have multiple drivers for each stage. And even within a single dimension in a stage we can define exceptions so that multiple drivers can be assigned within the same dimension. This way we can have allocation logic based not only on Resource Count but also on say Clocked Time of a resource in certain cases. Driver assignment is done through the Driver Selection screen in HPCM.

After the assignments, one other important feature of HPCM is its ability to assign allocations on a cell by cell basis. So, what we can do is, for each cell in the Source Stage we can assign the Destination cell(s) in the Destination Stage. That way we can clearly find out what intersections will get affected by what allocation logic. If you had to do them through say Essbase Calculation Scripts alone, it would have been such a laborious task not only while doing the allocations but also in maintaining them. In our example, lets assign the Data intersection of HRCosts in Stage 1 to the 3 Destination Projects in Stage 2 as shown below

Now lets run the Calculation & then immediately transfer the data to the ASO cube from the Manage Calculation screen.

If you now look at the data you can clearly see that each project now will have the corresponding costs allocated.

To validate how the costs have flown through the stages, HPCM provides an option to do a stage balancing report. This will show us how the costs have moved from stage to stage. In addition one biggest advantage of HPCM is, if there are any unassigned costs in the source, those will automatically move into Idle Costs (or we can configure it throw an error to ensure all costs are always allocated).

In terms of reporting, the ASO cube that HPCM provides will be more than sufficient. But based on what i have seen, we will have a lot of un-necessary dimensions especially when we are not concerned about the inter-stage essbase dimensions. In such cases, we can build our own ASO cube and extract only the necessary intersections that we need. But for now, i will showcase how BI EE 11g can be used for reporting again the native HPCM ASO cube. I will quickly show the lineage breakup reports as well

We start off with importing the ASO cube into BI EE which is pretty straight-forward as shown below.

Then lets start with building the report just for looking at our Stage 1 data. Remember Stage 1 input HRCosts data is loaded against NoMember intersections of the other dimensions. So we will have to explicitly filter to arrive at the right data.

Similarly to get the final allocated data we will have to apply filters as shown below

In the same way we can get the driver values directly from the cube using a simple BI EE report. All of these are shown in the form of a dashboard – showing how the costs from stage to stage using the drivers.

To illustrate what I’m talking about, take a simple analysis created against the Sample.Basic Essbase database, which I’ve imported into an OBIEE 11.1.1.5. repository using the default settings, and then created physical cube columns for the UDAs within the Market dimension, and the Default alias table within the Product dimension.

Notice how, at this stage, the measure hierarchy has been brought in as-is, with the full hierarchy preserved and a single measure, in a physical cube column called Basic – measure. To get access to a particular measure (for example, Profit), you’ll need to include this measure in an analysis criteria and filter on the Gen2, Measures, Gen3, Measures or Gen4, Measures logical column; or alternatively, access the measure dimension via a hierarchical column, and drill-down or use selection steps until you get to the one you want.

So let’s start then by creating an analysis where I have the Marketing and Payroll measures along the columns, and the East and West regions on the rows, and the numbers are for the Budget scenario. In MDX terms, using the essmsh command-line tool this would look something like:

MAXL> select {Measures.Marketing, Measures.Payroll} ON COLUMNS,
2>           {Market.East, Market.West, Market.South} ON ROWS
3> from Sample.Basic
4> where {Scenario.Budget};

Axis-1              (Marketing)         (Payroll)
+-------------------+-------------------+-------------------
(East)                            11210                7100
(West)                            16800               11660
(South)                            6120                5090

In OBIEE 11g, an equivalent analysis would look like this, using a pivot table view:

The criteria to create such an analysis would look like this:

Now this criteria is relatively simple, as we’ve selected measures that are within the same generation (Marketing and Payroll, both within the third generation in the Measure hierarchy). In reality, your measures are likely to come from different measure hierarchy generations, which is the reason you’ll often flatten the measures in the OBIEE physical model, but I’ll come on to the why and wherefores of that in a moment. For now though, the criteria is fairly simple and results in the following MDX sent to the Essbase server.

With
set [_Market2]  as '{Distinct({[Market].[East], [Market].[South], [Market].[West]})}'
set [_Measures4]  as '{Distinct({[Measures].[Marketing], [Measures].[Payroll]})}'
select
{ [Attribute Calculations]
} on columns,
NON EMPTY {crossjoin({[_Market2]},{[_Measures4]})} properties GEN_NUMBER, [Measures].[MEMBER_UNIQUE_NAME], [Measures].[Memnor], [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]
where ([Scenario].[Budget])

OK, not so bad. It’s a variation on what we coded by hand (using WITH blocks to define the set of rows and columns upfront, and a crossjoin to return the product of the two dimension member sets, but it’s recognizable as being more or less equivalent to our hand-coded MDX.

So what if we try a simple calculation? Let’s add the Marketing and Payroll measures together to create a new measure called Direct Costs. In MDX, I’d write this as:

MAXL> with member Measures.[Direct Costs] as
2>            'Measures.Marketing + Measures.Payroll'
3> select {Measures.Marketing, Measures.Payroll, Measures.[Direct Costs]} ON COLUMNS,
4>           {Market.East, Market.West, Market.South} ON ROWS
5> from Sample.Basic
6> where {Scenario.Budget};

Axis-1              (Marketing)         (Payroll)           (Direct Costs)
+-------------------+-------------------+-------------------+-------------------
(East)                            11210                7100               18310
(West)                            16800               11660               28460
(South)                            6120                5090               11210

So let’s try and create this now using OBIEE. Going back to the analysis criteria, I look for a way to add the two measures together.

OK, so this is interesting. There’s no Marketing or Payroll measure to select and use as part of an expression, as the measure name is actually a dimension member and there’s only one measure as-such. This is the drawback with bringing your measure dimension into OBIEE – you don’t then have individual measures you can select from to include in an analysis, create or include in calculations, and so on.

Now one way around this is to use the new “selection steps” feature in OBIEE 11g to calculate this value post-aggregation, after the initial filtering has happened and using the resulting result set. I therefore create a new selection step, like this:

and Bob is indeed your uncle.

So apart from figuring out the selection step move, this was actually quite straightforward. So how does the MDX look?

With
set [_Market2]  as '{Distinct({[Market].[East], [Market].[South], [Market].[West]})}'
set [_Measures4]  as '{Distinct({[Measures].[Marketing], [Measures].[Payroll]})}'
set [_Measures4FILTER1]  as '{Distinct({[Measures].[Payroll]})}'
set [_Measures4FILTER2]  as '{Distinct({[Measures].[Marketing]})}'
member [Attribute Calculations].[_MSCMFILTER1] as 'AGGREGATE(Intersect([_Measures4FILTER1], {[Measures].CurrentMember}))', SOLVE_ORDER = 102
member [Attribute Calculations].[_MSCMFILTER2] as 'AGGREGATE(Intersect([_Measures4FILTER2], {[Measures].CurrentMember}))', SOLVE_ORDER = 102
select
{ [Attribute Calculations],
[Attribute Calculations].[_MSCMFILTER1],
[Attribute Calculations].[_MSCMFILTER2]
} on columns,
NON EMPTY {crossjoin({[_Market2]},{[_Measures4]})} properties GEN_NUMBER, [Measures].[MEMBER_UNIQUE_NAME], [Measures].[Memnor], [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]
where ([Scenario].[Budget])

OK, so what’s going on here is the situation Venkat described back in some earlier blog posts, where selection steps involves the BI Server generating MDX that calculates lots of additional values which it then combines back together to give you your result set. Running this MDX statement through the MAXL command shell gives us this output:

Axis-1              Axis-1.properties   (Attribute Calculat (_MSCMFILTER1)      (_MSCMFILTER2)
+-------------------+-------------------+-------------------+-------------------+-------------------
(East, Marketing)   (GEN_NUMBER = 2, ty               11210            #Missing               11210
(East, Payroll)     (GEN_NUMBER = 2, ty                7100                7100            #Missing
(South, Marketing)  (GEN_NUMBER = 2, ty                6120            #Missing                6120
(South, Payroll)    (GEN_NUMBER = 2, ty                5090                5090            #Missing
(West, Marketing)   (GEN_NUMBER = 2, ty               16800            #Missing               16800
(West, Payroll)     (GEN_NUMBER = 2, ty               11660               11660            #Missing

The BI Server takes this result set, plus some additional member information it gets via another MDX query, and adds the results together as per the selection step calculation to give you your direct costs calculation. So whilst it’s very easy for the end-user to create their calculation (and in the case of using selection steps, just as easy to select measures from different measure hierarchy generations), but it’s not actually Essbase that’s doing the calculation, its the BI Server (and in fact Essbase is doing a lot of extra work to return all these subtotals, at least compared to my hand-written MDX).

An alternative to using selection steps though, would be to flatten the measure hierarchy into a flat list of measures, the same way that OBIEE 10g did by default when importing Essbase outlines. How does this affect the MDX that’s generated for a calculation like this?

I start then by duplicating the Essbase database in the OBIEE repository physical layer, and then select the option to flatten the measure hierarchy, giving me a physical model looking like this:

Now creating my calculation in the analysis criteria is really easy, as I’ve got individual measures for the Marketing and Payroll members, which I can just add together to create a new measure in the criteria.

So how do the numbers look? One advantage of working with a flattened list of measures, is that you can use a regular table view to display the results, rather than having to use a pivot table view as you do with a measure hierarchy. Taking a first look, the figures look correct:

What about the MDX? Well, it’s actually a lot closer to the MDX I wrote by hand; instead of adding the two measures together, it creates a set of the two (giving the same result, in this instance), but other than that, it looks pretty close to the MDX I wrote.

With
set [_Market2]  as '{Distinct({[Market].[East], [Market].[South], [Market].[West]})}'
select
{ [Measures].[Payroll],
[Measures].[Marketing]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]
where ([Scenario].[Budget])

Just as a quick check, I create another analysis that randomly adds up, divides and otherwise applies simple operators to some measures.

Now if I were to write an MDX statement to retrieve numbers like this, I’d probably write something along the lines of:

MAXL> with member Measures.[Sales Divided by Marketing]
2> as 'Measures.Sales / Measures.Marketing'
3> member Measures.[Sales Plus Total Expenses]
4> as 'Measures.Sales + Measures.[Total Expenses]'
5> member Measures.[Sales times Margin]
6> as 'Measures.Sales * Measures.Margin'
7> select {Measures.Sales, Measures.Marketing, Measures.[Sales Divided by Marketing], Measures.[Sales Plus Total Expenses], Measures.[Sales times Margin]} ON COLUMNS,
8> {Market.children} ON ROWS
9> from [Sample.Basic]
10> ;

Axis-1              (Sales)             (Marketing)         (Sales Divided by M (Sales Plus Total E (Sales times Margin
+-------------------+-------------------+-------------------+-------------------+-------------------+-------------------
(East)                            87398               14721    5.93696080429319              112708          4323666458
(West)                           132931               23529    5.64966636916146              173096          9308626206
(South)                           50846                8281    6.14007970051926               66504          1469246016
(Central)                        129680               19706     6.5807368314219              164544          9482979680

Looking at the MDX generated by the BI Server though, using flattened measures, gives us a bit of a surprise:

With
set [_Market2]  as '[Market].Generations(2).members'
select
{ [Measures].[Marketing],
[Measures].[Sales],
[Measures].[Total Expenses],
[Measures].[Margin]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]

All the BI Server does, then, is just request the basic values for measures from Essbase. The BI Server then performs the calculations, and you’ll find this happens all over the place. Where you would have assumed that the BI Server would ship-down calculations to the Essbase server, instead it just requests the basic data and performs the calculation in-memory, in the BI Server.

But if you particularly want to make use of an Essbase MDX function, there is actually a way you can do this, and it’s through the EVALUATE, EVALUATE_AGGR and EVALUATE_PREDICATE functions within OBIEE. These functions provide a way to pass through a request to the underlying datasource for an analysis to use a particular function, and you can use this approach to make use of Oracle database functions that either don’t have an equivalent within OBIEE, or perhaps you’ve written yourself. But you can also use this approach for working with Essbase sources, and it even allows you to do things that would be tricky to do in the ROLAP-world of OBIEE but are easy to do within a multi-dimensional database such as Essbase.

Take, for example, a situation were we want to list out, in an analysis, a list of markets together with sales amount AND the average sale amount for the states within each of the regions. In this case, getting the average sale amount for each region’s constituent states would actually be quite tricky with a regular relational source, as the dimension members your calculating the average for are at a different level of granularity to the one used in the query, but for Essbase this is easy. Let’s take a look at how we’d write this in MDX first.

MAXL> with
2>   set regions  as 'Market.children'
3>   member Measures.Children_AVG as 'AVG(Market.currentmember.children,Measures.Sales)'
4> select
5>   { Measures.Sales,
6>     Measures.Children_AVG
7>   } on columns,
8>   NON EMPTY {regions} on rows
9> from [Sample.Basic];

Axis-1              (Sales)             (Children_AVG)
+-------------------+-------------------+-------------------
(East)                            87398             17479.6
(West)                           132931             26586.2
(South)                           50846             12711.5
(Central)                        129680    21613.3333333333

Now moving over to OBIEE, what we want to use now is an EVALUATE_AGGR function, to leverage the AVG MDX function. This function takes two arguments:

AVG(set, numeric_value)

In other words, you pass it a set of members (or “tuples” in Essbase-speak), and a number to perform the average over. So now, if we want to use this in OBIEE to return an aggregated value, we need to substitute a dimension member reference and a measure reference which in turn will be passed down, along with the call to the AVG MDX function, to Essbase.

To do this I create a new logical column in the business model holding my Essbase data (the one with flattened measures), and edit the logical table source expression list to reference this function.

I also set the default aggregation type for this new column to AGGR_EXTERNAL, and then copy it across to the presentation layer subject area.

Now I can create an analysis that lists out the regions, their total sales, and then the average sale value for the states below each region, something that it’d actually be quite tricky to do with a relational source.

Taking a look at the MDX generated by the BI Server, it looks along the same lines as the one we wrote by hand:

With
set [_Market2]  as '[Market].Generations(2).members'
member [Measures].[_MSCM1] as 'AVG(Market.currentmember.children,[Measures].[Sales])', SOLVE_ORDER = 101
select
{ [Measures].[Sales],
[Measures].[_MSCM1]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]

Now what we’ve seen there is an example of an MDX function that returns an aggregated value, hence our use of EVALUATE_AGGR. But we can also return values that can be used as attributes for a dimension member, for example the count of immediate “children” under a dimension member. In this example, if we were to write a query in MDX that returned total sales for a region and also the number of states underneath it, the query would look like this:

MAXL> WITH
2> member Measures.Num_Children AS
3>   'count(Market.currentmember.children)'
4> select {measures.sales, measures.num_children} on columns,
5> {market.children} on rows
6> from Sample.Basic;

Axis-1              (Sales)             (Num_Children)
+-------------------+-------------------+-------------------
(East)                            87398                   5
(West)                           132931                   5
(South)                           50846                   4
(Central)                        129680                   6

To make use of this dimension member count in OBIEE, we’d need to use the EVALUATE function, and pass to it a reference to the currently selected dimension members’ children, and the name of the dimension that we’ll be calculating this for.

Taking a look at the MDX generated by the BI Server, it’s as you’d expect.

With
set [_Market2]  as '[Market].Generations(2).members'
member [Measures].[_MSCM1] as 'count([Market].Generations(1).dimension.currentmember.children)'
select
{ [Measures].[Sales],
[Measures].[_MSCM1]
} on columns,
NON EMPTY {{[_Market2]}} properties GEN_NUMBER, [Market].[MEMBER_UNIQUE_NAME], [Market].[Memnor] on rows
from [Sample.Basic]

So it’s a bit of a mixed picture with calculations with an Essbase source. On the one hand, OBIEE doesn’t really make much use at all of MDX functions, instead retrieving just the basic measure values from the Essbase database and performing the calculations itself, within the BI Server. In many cases this is fine, but it can trip you up if you’re expecting to offload processing to the Essbase server but instead, you find the BI Server is actually doing most of the work. In some cases there’s good reasons for the approach Oracle have taken – not all MDX functions work across all storage types, for example – but it’s worth being aware of this before you get too deep into working with Essbase.

The flip side of this though is being able to access MDX functions directly via the EVALUATE_AGGR and EVALUATE functions. These functions actually give you the ability to perform calculations, across levels and dimensions, that you would even be able to do with regular sources and OBIEE functions, and on most projects using Essbase sources you’re going to be using these functions alot, both to compensate for lack of push-down from the BI Server, but mostly to take advantage of the MDX functions and cross-hierarchy and level calculations that you can perform from Essbase.

There is one exception to all this though, and it’s an interesting one. The one type of OBIEE function that does get pushed down correctly to Essbase, is time-series functions. The AGO, TODATE and (new in 11g) PERIODROLLING do indeed get translated down to their equivalent MDX functions, which means that you can use the time-series calculations and wizard available in OBIEE to create all your time-based derived measures.

For example, consider a situation where you wish to calculate the value of sales for the previous quarter. In OBIEE, you’d create a time-series calculation looking like this:

Then your analysis would look like this:

Taking a look at the MDX generated by the BI Server, it does indeed use the PARALLELPERIOD MDX function.

With
set [_Year2]  as '[Year].Generations(2).members'
member [Measures].[_MSCM1] as '(ParallelPeriod([Year].[Quarter], 1,[Year].currentmember), [Measures].[Sales])'
select
{ [Measures].[Sales],
[Measures].[_MSCM1]
} on columns,
NON EMPTY {{[_Year2]}} properties GEN_NUMBER, [Year].[MEMBER_UNIQUE_NAME], [Year].[Memnor] on rows
from [Sample.Basic]

So, it’s been a long post, but what observations and rules-of-thumb can we draw therefore for Essbase sources and calculations?

• Measure hierarchies are great for end-users, particularly if they come from a traditional Essbase background. Coupled with selection steps, it’s easy to select measures from a hierarchy and perform calculations all from the Answers UI
• But measure hierarchies can often generate quite complex and costly MDX queries, and it’s often easier, as a developer, to work with flattened lists of measures. You can create these in a separate physical database model, or combine a measure hierarchy with a flattened list of measures in the same business model
• OBIEE out-of-the-box actually makes very little use of MDX functions in the Essbase database, instead choosing in most cases to just request measure values and then performing any calculations itself
• The exception to this is time-series queries, which do get pushed-down to their equivalent MDX function
• In some cases, you can compensate for this lack of push-down through the EVALUATE and EVALUATE_AGGR functions, but they don’t cover all situations and also limit you in terms of drill-down etc. EVALUATE and EVALUATE_AGGR also provide you with some extra functionality though, as they let you harness the ability of Essbase to calculate numbers across different hierarchy levels and dimensions.

That’s it for Essbase for a while now; keep an eye on the blog for a bit more on OBIEE 11g and Oracle OLAP, plus possibly some content on OBIEE 11g and Microsoft Analysis Services, to see how well support for Microsoft’s OLAP server compares to Essbase.

These two blog posts by Christian Berg back in 2010, based on OBIEE 10.1.3.4.1 and Essbase 9.3.1, describe the problem well and unless you were prepared to do some unsupported hacking with UDML and Notepad, working with an evolving Essbase outline was pretty much impossible with OBIEE 10g, and when 11g came out many customers re-defined their Essbase-derived physical model hierarchies as value-based to get around this problem.

But value-based hierarchies have their own issues, such as not being able to make use of UDAs or alias tables, or as outlined in this previous posting, not being suitable for vertical federation (drill-to-detail) with relational and file sources. So, it was with interest that I noticed a new section in the Essbase metadata modeling section of the OBIEE 11g documentation (11.1.1.3 and 11.1.1.5) entitled “About Incremental Import”, that says:

“You can choose to incrementally import Essbase metadata. In other words, you can perform an initial import, and then import again. You might want to import incrementally when information in the data source has changed, or when your first import only included a subset of the metadata. Note the following about incremental import:

• When you re-import metadata that already exists in the Physical layer, a message appears, warning you that your Physical objects will be overwritten.
• If you delete data in the source, re-importing the metadata does not automatically perform the deletion in the Physical layer. Instead, you must manually delete the corresponding Physical objects.
• If you rename an object in the source, the renamed object is imported as a new object. In this case, both the old object and the new (renamed) object are displayed in the Physical layer.
• In general, customizations that you have performed on the Physical layer data, such as determining the alias column to use for display, are retained after an incremental import. If you want to revert to the default imported view, you must delete the existing Physical layer objects and then re-import the metadata.”

So, how does this work, and how easy is it to implement? With this question in mind, I thought I’d give it a try.

To start off, I’ve imported a copy of the Sample.Basic Essbase database into the Physical layer of an Oracle BI Repository. Running this import creates a physical database named after the Essbase server (essdemo), a physical catalog named after the Essbase application (Sample), and a cube table named after the Essbase database (Basic). I performed the import using OBIEE 11.1.1.5 with all the import options left at their default value.

As you can see from the screenshot below, the Market physical dimension has three UDAs associated with it; Major Market, Small Market and New Market.

So, as my first test, what if I use Essbase Administration Services to add a new UDA value to the West Member called Development Market, a UDA value that’s new to the outline and therefor wouldn’t have been imported into the OBIEE repository in the past?

I save the updated outline and then go and re-run the Import Metadata wizard in the Oracle BI Administration tool. Selecting the same Essbase database brings up a warning asking me to confirm I want to re-import.

After running the re-import, I check the Market dimension, and yes, it contains the new UDA value.

But what if what the Administration tool has done, is just overwrite the old cube table definition with the new one? This could cause me problems if I’d fine-tuned the physical model previously by, for example, setting the hierarchy type for the Market dimension to Fully Balanced, or I’d taken the Measure hierarchy and flattened it into a flat list of measures.

So let’s perform these customizations on the physical model, by firstly setting the Market hierarchy type to Fully Balanced (I would choose Value, but you can’t access UDAs if you set your hierarchy to Value):

I then right-click on the cube table and select the Convert measure dimension to flat measures option.

So at this point, I’m looking to preserve the change I’ve made to the hierarchy type for the Market dimension, and also the new set of flattened measures in the cube table. Note that converting the measures to flat measures still leaves the Measure dimension in the cube table, and I would generally delete this from the business model after dragging and dropping the physical model into the Business Model and Mapping layer.

After saving the changed physical model, I then go back into Essbase Administration Services and add another UDA to one of the Market dimension members.

I then use the Import Metadata wizard in the Oracle BI Administration tool to bring in the updated outline. Checking the resulting updated cube table, the new UDA is there as expected.

The flattened measures are still there, but unfortunately, the Hierarchy Type for the Market hierarchy has been set back to Unbalanced.

So now, even thought I know that I can’t make use of UDAs when I set the Market hierarchy type to Value, I’ll set it to this anyway, and then go and add another UDA to the dimension, to see if it does preserve the Value hierarchy type setting after a re-import. I set the Hierarchy Type to Value:

I then add another UDA to the outline, re-import it using the Import Metadata wizard, and interestingly find that the Value hierarchy-type setting has been preserved. If I then switch the Hierarchy Type back to Unbalanced, Fully Balanced or any other level-based hierarchy type, I can then also access the new UDA that was added to the outline.

So the rule here seems to be: if you’ve added UDAs to your outline, and you re-import the outline using OBIEE 11.1.1.5 (I’ve also checked this for 11.1.1.3, it’s the same behaviour), existing changes to the outline are preserved, except Hierarchy Type settings where all level-based hierarchy types are reset back to Unbalanced. Any changes you’ve made to flatten the measure hierarchy, for example, are preserved OK.

So what about aliases? If we add a new alias table to the Essbase outline, and use it to add a new alias value to, for example, the East member, what happens?

At the moment, if I view the list of Alias tables imported into the cube table definition, I can see the standard Member_Name, Default and Long Names aliases.

Now again, as with UDAs, you can’t access alias tables when you set your Essbase-derived physical hierarchy type to Value, so I set it back to Fully Balanced and then import the updated outline.

As with UDAs, the new alias table has been brought in with the re-import, the hierarchy type has been reset back to Unbalanced, but other that that, the rest of the cube table definition seems unchanged.

So what about something a bit more drastic? What if one of the hierarchies in your Essbase outline suddenly grew an extra generation (which would correspond to a new hierarchy level and set of cube columns in the OBIEE physical layer), which in 10g would mean you were pretty-much out-of-luck. To test this out, first I add a new generation to the Scenario hierarchy, adding Initial Budget and Revised Budget members as children to the existing Budget member. I save the outline and restructure the cube, in preparation for importing into the OBIEE repository.

At the moment, the Scenario dimension and hierarchy within the cube table has two levels, corresponding to the two generations in the Essbase hierarchy.

I’ve also created a corresponding business mode, and subject area, off of this cube table, so that I can see how easy it’ll be to incorporate the additional levels into already-derived business models and subject areas.

So, here we go. I use the Import Metadata wizard to re-import the database definition into the physical layer, and check out the updated physical model.

And there we have it. The new child members in the Scenario dimension hierarchy have caused a new level to be created in the physical model, something that (unless i’m much mistaken) wasn’t possible in OBIEE 10g. And this is quite a big deal – for many of our customers, not being able to refresh their physical model off of a changed outline was a showstopper, and it’s been the major driver towards adoption of value-based hierarchies in OBIEE 11g. The fact that this is now possible is actually quite big news, and I’m surprised the dev team within Oracle responsible for OBIEE/Essbase integration hasn’t publicised this – I only came across it when trawling through the online docs whilst writing the Essbase chapter of the OBIEE 11g book.

So how easy it is to incorporate these changes into  your business model, which remember is automatically generated for you when you drag and drop your cube table into the business model and mapping layer of the Oracle BI Repository. To find out, I start by dragging three of the four cube columns that are contained within the new Gen3, Scenario physical level across to the correspondng business model logical table, like this:

Then I add this additional key column to the logical table key, like this:

Next, I add a new child level to the existing Scenario logical dimension, and add these three logical columns to the level.

Now I add the logical level keys, copying how they are set up for the other levels.

Then, to finish off the business model changes, I set the detail (content) level for the logical dimension and logical fact table sources, to reflect the additional level that’s in the Scenario dimension.

Finally, I update the subject area in the physical model, deleting the old presentation hierarchy within the Scenario presentation table and replacing it with the updated one from the business model and mapping layer.

Now I can verify the updated repository, and then create a sample analysis. First off, I create an analysis that just includes the updated Scenario dimension, to check that the new measure comes through as expected.

You can see the two new scenarios listed under the Budget member. Now I add a measure into the analysis, which should lead to these two new members being removed, as no data has been recorded against these scenarios.

Very good. The hierarchical column indicates that there are members under the Budget member, but if you try and click on the “expand” icon, nothing happens.

So we’ve shown that we can import new generations into the repository, and then feed these through to the business model and presentation layers, so that users can make use of the new members at this new level. Interestingly, if you then remove these two new members so that the hierarchy goes back to the original number of generations, and then re-import the outline back into the repository, the Administration tool removes the additional generation from the physical model, so you’re back to two generations again (contrary to what the manual says, actually). You’ve then got to go back and adjust your business model and subject area to reflect the new number of generations, but at least it’s possible to delete generations this way, as well as add them.

But what about something bigger – say, adding a dimension to an Essbase database? Given what we’ve seen so far, it’s likely that the incremental import from the outline will pick up the new dimension, but is it then possible to model it in the business model and subject area?

To test this out, I create another copy of the Sample.Basic Essbase database, delete the Population attribute dimension, and then import it into the Oracle BI Repository using the BI Administration tool. For good measure, I change the Market dimension hierarchy type to Fully Balanced, and convert the measure dimension to a flat list of measures.

I also create a corresponding business model and subject area for the database.

Then, I restore the missing attribute dimension to the source Essbase database (by dropping it, and then re-copying it from Sample.Basic using Essbase Administration Services), and then use the Administration tool to re-import the outline again. How does it work out?

Well, as you can see, the Population attribute dimension has been imported incrementally, and the measure dimension is kept as a flat list. Interestingly, the Market dimension’s hierarchy has been reset back to Unbalanced, which indicates that the re-import process sets all the level-based hierarchies back to Unbalanced regardless of whether the hierarchy has any other changes. So how easy is it to introduce this new attribute dimension to the business model?

Dragging and dropping the new attribute dimension from the Physical into the Business Model and Mapping layer adds two items to the business model; a logical table for the cube columns, and a logical dimension, for the physical hierarchy and physical levels. Notice how the logical table has the “fact’ indicator in it’s icon? To remove this, I create a logical join between this new logical table and the fact logical table, like this:

I also set the content level for the fact table logical table source, to reference the detail-level from the new Population logical dimension hierarchy.

Finally, I copy the new logical table over to the Presentation layer of the repository and add it to the existing subject area, along with it’s logical dimension. Now, to try it out in an analysis.

So there you go. It all appears to work fine. So to me, it looks like the limitation that we had with OBIEE 10g, where we couldn’t incrementally refresh a physical model in the repository based off of an Essbase database, is no longer the case. There’s a couple of gotchas – level-based hierarchies, even for those that haven’t been touched – get reset back to Unbalanced from any other level-based hierarchy setting, and the actual process of amending the business model and subject area are a bit tricky (at one point, I kept hitting a drilling error with the new scenario members and generation, but after a couple of restarts and re-imports it just started working).

But if your concern about working with OBIEE and Essbase is not being able to handle outline changes, and value-based hierarchies have other restrictions that you can’t otherwise work with, I’d say this is worth taking a look at. This was certainly an eye-opener for me, and I can’t help wondering why Oracle haven’t made more noise about this, apart from burying the news hidden away in the documentation. Strange, but good news nonetheless.

At the time I said:

“Fast forward to 2009, and if I was recruiting an Oracle Business Intelligence developer now, what would I be looking for? Ideally, most if not all of the following product skills:

• Oracle Business Intelligence Enterprise Edition, usually with a specialization in either the back-end (the BI Server) or front-end (Answers, Delivers, BI Publisher), would be mandatory
• Essbase skills would be very desirable, and would be required far more than Oracle OLAP skills in the past.
• Oracle BI Application experience is a definite bonus, with the real skill in being able to apply customizations to the ETL and RPD layers
• Another very desirable skill is being able to implement at least Hyperion Planning, and ideally some more of the Hyperion financial applications. It’s rare though to find people with both good OBIEE and Hyperion skills, those that do can virtually write their own salary cheque.
• In terms of ETL, OWB is now “mainstream” and is a mandatory skill. ODI is rising in importance, but I don’t see many “classic BI” projects that use it, at least at the moment.

Compared to five years ago, skills in the Oracle database, and in particular with the Oracle Application Server, are less of a priority, as security for example is usually handled now by the OBIEE BI Server, and fast access to data is typically done through Essbase. Also, now that Oracle’s BI tools (and Fusion Middleware) are designed to work across heterogeneous platforms, its more useful now for developers to have experience with Active Directory, for example, compared to Oracle Internet Directory, and it’s also useful to have knowledge of other database platforms such as Microsoft SQL Server, Microsoft Analysis Services and the like as Oracle’s BI tools are increasingly being sold to non-Oracle database customers.”

So how have things changed, now it’s 2011? Well, some things are much the same; OBIEE, if you’re an Oracle BI developer, is a mandatory skill, and as back in 2009, you tend to specialize in the backend (RPD development, possibly Informatica if you’re an Oracle BI Applications developer), or the frontend, creating the analyses and dashboards that are most visible to the end users. Take away all of the stuff about WebLogic, Enterprise Manager and Fusion Middleware, and the core of OBIEE hasn’t changed much since the days of Siebel Analytics, and we typically see the same old problems, misunderstandings and so on with RPD development that we’ve seen since first transitioning to the technology back in 2005. Expect things to change fairly rapidly in the next couple of years though, with a new Administration tool, possibly a new (or alternative) XML or database-based repository, and a lot of work going on within Oracle to make the whole process of developing and deploying repositories a lot more automated.

One skill that an Oracle BI developer will need, and that wasn’t such a requirement back in 2009 (even, in my view at the time, becoming a less relevant skill), is application server skills, specifically Oracle WebLogic Server. For the time being at least, if you’re deploying OBIEE 11g, you’re deploying WebLogic Server, and having skills in such areas as WebLogic security, WebLogic Scripting Tool, the WebLogic Admin Console, and technologies such as admin and managed servers, node managers and OPMN is pretty much a requirement if you’re going to be an end-to-end OBIEE 11g developer. You could pretty-much install OBIEE 10g and forget about it, but nowadays you need to have some systems administration skills if only to get the product installed and running.

Another skill I downplayed in 2009 but is increasingly relevant now, is security. OBIEE 10g took care of security and connectivity to LDAP servers from within the Administration tool, but now you need to know Oracle Platform Security Services, application roles and policies, and all the associated technologies around WebLogic’s implementation of security. In fact, one major skill you need now is the ability to read and digest reams of documentation, made all the worse by OBIEE’s content being folded in with the rest of the Fusion Middleware product line, making it tricky to pick out how, for example, you can connect OBIEE to third-party LDAP servers, or how to perform upgrades between releases of 11g. It’s all a lot more complicated now, and you can’t just charge in, ignore the documentation and not do your homework.

Essbase was something I highlighted back in 2009, and at the time, it seemed fairly imminent that all Essbase projects would use Oracle BI technology, and all Oracle BI projects would include an element of Essbase. Now, i’m not so sure; Essbase is going gangbusters and if you’re an Essbase specialist, you can more or less name your price, but for OBIEE projects, we see a bit of Essbase, but it’s not gone mainstream yet. Realistically, whllst OBIEE 11g solved a number of 10g-era problems with OBIEE/Essbase integration, it’s just introduced another set of new ones, and I don’t see many WebAnalysis, Financial Management or Interactive Reporting customers moving to OBIEE unless their application was pretty-much relational in form, in the first place. If you’ve got Essbase skills, it’s certainly a bonus, but unless you’re prepared to get to expert level, you’ll probably find most Essbase projects are handled by specialist Hyperion consultants, and your job is probably more likely to be raising SRs and working with the client to try and fix the OBIEE to Essbase connectivity issues.

I smile to myself when I look back at the comment on OWB. Certainly, in terms of new projects, OWB is dead now and every customers is moving towards ODI. Maybe it’s because it’s all that Oracle will sell on new projects, maybe it’s because customers don’t want to invest in products that have been sunsetted, or maybe it’s because ODI is better, but nowadays, you need to know ODI if you’re an OBIEE developer. This means knowing things like how to create knowledge modules, how to migrate projects from environment to environment, and how to use new features such as ODI OBIEE data lineage. But if you’re an OBIEE developer and you’ve got time to learn one more skill, make it ODI.

And so finally on to the Oracle BI Applications. My experience has been that most BI Applications projects have been handled by the big ERP integrators, occasionally calling on specialists if the OBIEE element goes wrong. Either you’re an RPD, DAC and Informatica developer, or you’re an OBIEE, RPD and OWB/ODI developer, and if you’re the former you’re probably a contractor or working for a large SI. The major bonus skill that you’d have as a BI Apps developer is knowledge of either Siebel or Oracle E-Business Suite as a data source, but all of this is going to change in the next couple of years as Oracle transition to the Fusion Applications, with the 11g release of the BI Apps focusing on the Fusion Applications first and then only with support for Apps Unlimited (basically, all the legacy ERP suites Oracle developed or acquired) coming later on. The big question though, at least in my mind, is whether you can still be a BI Apps developer in the future without also being a Fusion Apps developer, and a Fusion Development (ADF, JDeveloper, SOA Suite) developer; the 11g BI Apps will be embedded in the Fusion Apps, and whilst it’s possible to develop for the 7.9.x release of BI Apps as a standalone piece of technology, I wonder how much you’ll be able to achieve in isolation from the Fusion Apps.

So, to summarize, and to update my 2009 comments about what skills an Oracle BI developer needs, in 2011 I think you need:

• OBIEE 11g skills, both in terms of new functionality (mapping, analyses, KPIs and Scorecards etc) and new infrastructure (WebLogic, EM, OPSS etc)
• A smattering of Essbase skills, focused mainly on the integration with OBIEE and Essbase (and the many workarounds and gotchas)
• Good ODI skills, both in terms of the basics, but also being able to write knowledge modules, integrate with OBIEE, deployment and migration
• Solid database skills – OBIEE gave the illusion through aggregates etc that database tuning was redundant, but time has shown it’s by far the biggest success factor in a project – get the database design and optimisation wrong, and your project is toast. You need to know partitioning, materialized views, index types, and increasingly, you need to get yourself on an Exadata project as customers are buying the technology but you can’t teach it to yourself at home
• BI Apps skills, but watch out for everything changing when BI Apps 11g comes out, and be prepared to learn the Fusion Apps and JDeveloper if you want to stay in the game
• Looking to the future, keep an eye on technologies such as in-memory (TimesTen), mid-tier caching (Coherence), plus technologies such as Business Activity Monitoring (BAM), “big data” (Hadoop, large data sets, NoSQL), complex event processing and maybe products such as Qlikview, just in case Oracle buys them, or at least to know what the competition are up to, or more importantly pitching to your boss

The other thing to bear in mind of course, if you’re an Oracle BI developer, is that you need to have great business, communication and data modeling skills. But that’s another topic in itself, and maybe one of my colleagues will have some views on that.

As promised, I’ve uploaded the slides from the session here, and you can also read the blog post behind the session, here. I’ve also uploaded the slides to SlideShare, and you can browse through them here (just refresh the page if the area below is blank):

Keep an eye out on the blog soon for a follow-up posting, where we go into how you can leverage the OBIEE 11g Action Framework to perform write-back to an Essbase cube, and to trigger the execution of an Essbase calc script.

If your interest is with BI & DW topics, there’s tons of great content at this event. On the Sunday, there’s a special Oracle-led BI Symposium with content from Adam Bloom (security, and the action framework), Jacques Vigeant (Scorecard & Strategy Management)  and Nicky Sanger (Answers new features), plus a development keynote from Balaji Yelamanchili, Oracle’s Senior VP in charge of all OBIEE development. If you read my posting following Balaji’s presentation on the OBIEE technical roadmap at Collaborate’11, you’d certainly want to come to his session. For my part, I’ll be in there at 9am and there right through to the end, none of these sessions are to be missed.

During the rest of the week, there’s other great content including the return of the one-and-only David Allen, this time talking about ODI(EE), Mike Durran with a must-see presentation on OBIEE scripting and internals, Holger Freidrich on BI Server vs. Database aggregation, Kevin McGinley, winner of the US BI Forum Best Speaker prize on OBIEE and Mobile BI, plus a couple of sessions by myself and Stewart Bryson. In addition, if you’re interested in seeing what goes on in the world of Hyperion and Essbase, there’s a massive conference-within-a-conference going on led by such Essbase luminaries as Edward Roske and Tim Tow, with the combined EPM + OBIEE content area being led by Natalie Delamar.

I’ll be over in Long Beach from the Friday and then taking part in the conference from Sunday onwards. This is the last major BI event Stateside until Open World, so if you’re looking to dig deep into the OBIEE 1g platform, make sure you register now before the last discount ends on Thursday!

OBIEE 10g offered a lot of promise for Hyperion customers, in that you could connect Essbase to the Oracle BI Repository and create rich, interactive dashboards and analyses using Essbase data, relational data, or a combination of the two. The BI Server translated the incoming “logical SQL” queries from the Presentation Server into MDX queries that Essbase understood, allowing you to create reports, graphs and other visualizations that worked the same way whether your source data came from an Oracle Database, an Essbase cube, or any other sort of supported data source. You could also integrate the OBIEE dashboard with Hyperion Workspace, so that users signed on once through Shared Services and could then log in to OBIEE using the same credentials, inside a unified query environment, as detailed in this blog post from 2009.

But this integration came with limitations. The 10g version of OBIEE didn’t really have a proper OLAP-style query tool, such that when you drilled-down from one hierarchy level to another, it kept adding columns to the report and placing filters on the column you’ve drilled on, and there was no proper hierarchical selection tool, meaning that Essbase users moving from tools such as SmartView or WebAnalysis found OBIEE 10g to be a very “relational” query environment. In addition, only level-based hierarchies were supported within the Oracle BI Repository, and whilst OBIEE did a good job in “unraveling” the parent-child hierarchies in Essbase into corresponding level-based hierarchy when you imported an Essbase outline into the repository, changes to the hierarchy – particularly adding new levels or generations into the Essbase hierarchy – usually meant that you had to re-import the whole Essbase cube back into the repository, losing any integration work that you’d done beforehand.

So OBIEE 11g offered a bunch of improvements that were particularly aimed at Essbase, and other OLAP server, customers; the new hierarchical columns feature brought proper OLAP analysis to OBIEE, and coupled with OLAP-oriented features such as hierarchical prompts and selection steps, made OBIEE 11g a much more conducive environment for Essbase users.

There’s much better support for Essbase-native features such as value-based hierarchies, alternate hierarchies, multiple alias tables, UDAs and other Essbase artifacts, and the value-based hierarchy feature in particular is very welcome as it can avoid the need to completely re-import your Essbase database if your outline gains additional hierarchy levels.

So coupled with the new and improved visualization capabilities in OBIEE 11g, including maps, scorecards, improved dashboard controls, the whole new UI, improved BI Publisher and so on, there’s a lot to recommend OBIEE 11g to users of the “legacy” Hyperion tools looking for a bit of a refresh of their BI environment.

Some of the new capabilities in OBIEE 11g, such as the action framework, may not immediately obvious to Hyperion users as useful to them, but they actually bring the ability to create additional integration between the platforms. One thing that’s still lacking, for example, is the ability to write-back from the dashboard to Essbase (you can write back to relational sources, but not Essbase, yet), and you can also use it to call actions that, for example, run a calc script or perform an allocation routine. If you take a look at the new 11.1.1.5 version of the SampleApp (due out in a couple of weeks), there’s some working examples “from the field” that perform this function, and there’s a posting on our blog from last year that show how you can use the Action Framework, and the Credential Store within Oracle Fusion Middleware, to provide integration between the OBIEE toolset and the Hyperion toolset.

As you’ll also be on the OBIEE platform, Oracle’s “strategic” BI direction, you’ll also be able to take advantage of features such as Oracle Enterprise Manager, Oracle Platform Security Services, and all the other middleware features that come with 11g. And you’ll be able to also integrate with the Oracle Business Intelligence Applications, giving you the ability to reach through to an integrated data warehouse sourcing data from Oracle E-Business Suite, SAP, Siebel and Peoplesoft.

But with every new feature, you’re going to expect to hit some new issues. Some things haven’t changed in this release compared to 10g; if you do need to make changes to the Essbase database metadata in your Oracle BI Repository and you can’t get around it by switching to value-based hierarchies, you still need to re-import the whole outline as there’s no “sync” or “incremental import” feature in OBIEE 11g. The Oracle BI Server still function-ships very few OBIEE functions down to MDX functions, so you’ll either have to have the BI Server perform analytic-style functions for you, or you’ll need to use the EVALUATE function to pass them down yourself. And we’ve actually lost the integration between Hyperion Workspace and the OBIEE dashboard, so if you want both OBIEE and the Hyperion tools to use the same directory, you’ll need to connect them both to an external LDAP server, and use some other SSO solution over both products rather than the “point-to-point” solution that worked with OBIEE 10g.

Some new features, such as value-based hierarchies, come with one or more “gotchas” that may well catch you out by causing one problem when you’ve just fixed another. If, for example, you’ve moved your Essbase hierarchies in the Oracle BI Repository physical layer to value-based ones, your users benefit from more efficient searches across the hierarchy and you won’t need to re-import the outline if you add a new level or generation. But where you’ll come unstuck now is if you’re planning on setting up drill-to-detail reports, where Essbase provides the aggregated data but the detail comes from a relational database, such as Oracle. In this case, you’ll find it hard to set up the “vertical federation” that’s required as there are no “levels” to designate the granularity of the Essbase and relational sources. There are a number of workarounds (we covered two approaches, here and here, towards the end of 2010), but neither of them are pretty and you’ll have a “facepalm”-type moment when you realise this limitation of value-based hierarchies.

Other, possibly more serious issue, is around the way that OBIEE generates the logical SQL, and physical MDX, for pivot-table style queries involving hierarchical columns. Take for example the following simple pivot-table report with a single product hierarchy and revenue measure, where the user has drilled-down into the data to produce some totals and sub-totals.

Now if this analysis was being generated against a relational source, the BI Server would generate three or four logical SQL requests for each measure “grouping”, which in turn would normally be pushed-down into a single SQL statement that used sub-query factoring (the WITH clause). Again, not pretty, but databases such as Oracle 10gR2 onwards can handle this pretty well.

Where it gets tricky though is when you’ve got an equivalent Essbase source under the covers. In this case, these logical SQLs all get translated into one or more physical MDX statements, each one of which is returning the aggregated values for these sets of grouped measures. If we were using an “MDX-native” tool such as SmartView or WebAnalysis, we’d only need the one MDX query, but in this case our query has generated five of them.

Now where it gets really interesting is if we add a second, or third hierarchical column to the report, and in particular, when we pivot the dimensions around so that one of them runs along the columns, rather than just the rows, of the pivot table, like this:

In this case, what would appear to be a simple pivot table query with a couple of dimensions drilled-in to asymmetrically, ends up generating around fifty-eight separate MDX queries, each of which is relatively simple, but multiply that by a number of concurrent users, and you’ve got a seriously big load on your Essbase server.

Now in practice, there are workarounds, and the new subquery-caching feature in OBIEE 11g means that these MDX sub-queries are often retrieved from cache rather than sent on a round-trip to the Essbase server. You can also limit the effect by keeping all of the dimensions on the rows, rather than rows and columns, of your pivot table, and in practice most Essbase systems aren’t all that heavily loaded, so you may well get away with it – but it’s something to bear in mind.

Another new feature that can also add to this problem is selection steps. Selection steps are very interesting to Essbase customers as they allow you to build up queries that use just certain members from the dimension, combine them together, use them to filter data and do other “OLAP”-style filtering and selections. To take the example below, we’ve used the Sample.Basic cube to initially return the grand total figure, then used selection steps to keep just the Q1 and Q2 figures, add a calculated member for H1, then do something simular for the regions.

All good so far, and light-years ahead of what we could do with OBIEE 10g. The problem comes though with the way that the selection step sub-queries are translated from the logical SQL that the BI Server uses down to the physical MDX queries that Essbase uses. From the tests that Venkat has carried out, the number of logical SQLs that selection steps generates comes to the product of all the individual selection steps, so if you have three dimensions that contain 2, 3 and 3 selection steps each, you’re going to end up with 2 x 3 x 3 = 12 logical SQLs, each one of which translates into 1 or 2 physical MDX queries. Once you add in the usual grand total at the top that brings back the entire cube, and any hierarchical column drills you might add in, you can see how you might end up with a very large number of MDX statements being sent to the Essbase server for what appears to be a very straightforward report.

Now in practice you tend to have workarounds for these issues; we tend to suggest keeping the number of hierarchical columns to a minimum in an analysis, and when you do use them, limit how far down you drill into them. If you use selection steps, try and keep them to one or two per dimension maximum, and overall, size your system to expect large numbers of relatively simple MDX statements.

Another aspect to consider is where your users current make use of the Excel Add-in, or SmartView, to do their analysis. Typically, they’ll use Excel and the “query-by-example” feature of the Excel Add-in to write-in the names of the dimension members they want in the X and Y axis, and then do a retrieval based on the selected members. Using this approach, end-users can create pretty complex analyses where there’s very little limitation on the combination of dimension members they use (selecting budget profit for this year, alongside the same figure for last year, together with actual inventory and forecast margin, for example). Creating analyses using these sorts of member selections is certainly possible using OBIEE 11g, but you’ll need to learn a new set of skills to do it properly.

For example, if you want to create a dynamic time-series reference such as the value of a measure as at this time last year, or over the past three months, or year-to-date, you can use one of the OBIEE time-series functions (AGO, PERIODROLLING, TODATE) to provide an offset value for a measure.

Similarly, OBIEE 11g now by default creates a single measure within your logical fact table which is then dimensioned by the measure (accounts) and scenario dimension, if present. If you want to display on the screen your “budget profit” figure using this approach, you can use the FILTER function in OBIEE to filter the measure by these two dimension member names, an approach you can use to display any arbitrary measure and scenario combination in an analysis.

Another technique that’s useful is being able to make use of native MDX function names through EVALUATE and EVALUATE_AGGR. As I mentioned earlier, one limitation with this current release is that very few OBIEE functions are function-shipped down to MDX, but if there’s an MDX function you want to use, you can easily access it through EVALUATE (for scalar functions, such as TOPCOUNT) or EVALUATE_AGGR (for functions that return an aggregated value, such as STDDEV).

So there’s two things to take away from this last point; firstly, don’t think of OBIEE 11g as a like-for-like replacement for Excel and the Excel-Addin, think of it instead as a complementary tool that covers, say, 60% of its functionality but then provides a whole raft of features that Excel alone can’t provide. Also, be prepared to roll your sleeves up and learn how OBIEE analyses are created, and expect your non-trivial reports to require a bit of Essbase knowledge an understanding of the more exotic features of Answers and OBIEE.

There’s more to the story that I’ll go into in the session, including what’s now possible for Hyperion Planning and Hyperion Financial Management users, and I’ll also run through some detail on how the actions integration works between the OBIEE dashboard and Essbase writeback and calc scripts, and I’ll also go into more detail on how you set up vertical and horizonal federation between Essbase and relational sources, and why you might want to use the new LOOKUP function to do some of this instead. So without jumping the gun too much, it’s a bit of a “qualified Yes”, but I’ll go into more detail on the day.

My session at the OUG Conference Series EPM & Hyperion 2011 is at 9.40am – 11.10am on Day 2 (15th June 2011), and full details on the event, including how UKOUG members can get in free, are on the event website. I’ll also post the slides on our Articles page once the event is over. Hopefully I’ll see you there.

1. Data Level Filters

2. Meta Read Filters

Data Level filters hide the data values of the members. But the members themselves are visible. On the other hand Meta Read Filters provide data and member level security.

Data Level Filters:

To test this we start off with creating a user in shared services called as secuser. This user is provisioned to access the Demo > Basic cube. Also, a filter is created to provide access only to the IDescendants of the West Market member.

I believe there is a bug in the Admin Console UI of Essbase 11.1.2. It does not allow the admin user to assign filters to other users. Thanks to Glenn, just realized that the filter assignments have moved over to SS. So, lets assign the created filter to the user through MaxL instead.

grant filter ‘Demo’.'Basic’,'MarkFilt’ to user secuser;

Let’s now update the connection pool settings of the repository to connect to Essbase using the above created user.

Let’s now create a report containing the hierarchical Market column and a measure as shown below.

This works as expected. Typically EAS will generate #No Access values for all the cells for which the user does not have access.

Meta Read Filters:

Now that we have tested the data level filters, lets test the metaread filters. To test this, let’s create one more user called metareaduser who will have access to all members under the West member of the Market dimension. Very similar to the data level filter use case above but this one will provide dimension member level filtering capability.

Let’s now update the connection pool of the RPD with the new user and then test whether the above security works.

Again this works as expected. Let’s now find out what happens during the drill if we assign a metaread filter to the lowermost level i.e. on the Seattle member instead of West.

This is very good. As you see whenever we assign a metaread filter to the lowermost level, we still see the branch from root leading up to that member (with data level security applied). That way we can have security applied at any level in a hierarchy and we still will have a seamless drill experience.

1. BI EE generates the most optimal MDX as long as we have all the attribute or Hierarchical columns in the Row-Edge of a Pivot.

2. While using Selection Steps, the number of MDX generated can increase proportional to N^M where N is the number of dimensions and M is the average number of selection steps in each dimension.

3. While moving hierarchical columns to the Pivot Column edge and while drilling on the column edge, the number of MDX generated can increase quite a lot

4. Every MDX generated will have a corresponding Sub-Logical SQL combined to the main Logical SQL through UNION ALL

The last point above is very important as the performance of a given BI EE report going against Essbase is determined by that. We start off with decoding how and why BI EE generates so many Sub-Logical SQLs (for the same Case 2 we discussed in the last post)

This generates a single Logical SQL with only one Sub-Logical SQL. Now let’s move the Market Dimension to the Column Edge of the Pivot and then look at the SQL generated.

Interestingly this generates a logical SQL with 3 Sub-Logical SQLs. Let’s look at the Logical SQL’s closely. In the first Logical SQL above, we had a statement like this

But after we moved one of the columns to the column edge, we notice 3 logical SQLs with the following filters

As you see, the Pivot Operation introduces more SQLs. Let’s try to understand why this is needed. So, to make it simple we apply a selection step on both the dimensions to basically start with predefined members as shown below.

This will also produce 3 Sub-Logical SQLs (as we are applying only one selection step per dimension i.e. 1^1 = 1 – no extra logical SQL because of selection Step). If we now look at each Sub-Logical SQL you can notice the following

The last SQL alone would have been sufficient to generate our report. But BI EE generates 2 additional Logical SQLs which are not necessary, atleast for this report. Why does BI EE generate these 2 additional queries. The 2 additional queries look like total queries on the row-edge and column-edge. Maybe these are for the totals. To validate this, let’s try adding a row-wise and column-wise total in the pivot report.

The total numbers exactly match with the numbers that we got from the first 2 logical SQLs shown above. But again, if we look at the logical SQL, there will be a total of 6 Sub-Logical SQLs. The 3 additional ones are shown below (for the totals)

One interesting thing to note here is, each total fires a separate query which is a new feature in BI EE 11g for making sure that the totals generated are correct and as expected. But still we are getting new queries generated for totals. And We haven’t accounted still for the need of the first 2 sub-logical sqls. Let’s investigate further by removing the sub-totals and then drilling down on the East member. You will notice 5 sub-logical SQLs fired to generate this query.

Rather than taking each and every sub-logical sql, let’s take the actual logical sql and fire it.

As you notice, the Queries 4 and 5 are the only ones needed for generating the actual report. But the first 3 queries seem to be doing something in addition to what was actually asked for in the report. If you notice closely, the first 3 queries seem to have an integer which controls the sort order. Now, is it possible that these three additional queries are there just to enable the outline sort order.  To test this, lets move the Market dimension back to the Row-edge with the East member exploded to its children as shown below

Let’s now take the complete Logical SQL and fire that against the BI Server.

Very interesting. We now have the SORTKEY pushed into the grain of the query itself. So, what essentially is happening is, if we have the hierarchical columns in the row-edge, there is no-need for BI EE to fire another logical sql to get the sort order as that can be obtained as part of the query grain itself. But in the case of hierarchical columns on the column edge and row-edge, there will be additional queries fired. Now to find out how many such queries will get fired, lets move all the columns to the column edge as shown below.

If we now look at the Logical SQL, there will be only one that too with the SortKey pushed to the grain of the query itself as shown below.

So, only when we have hierarchical columns in both the row-edge and the column edge, we have separate queries fired to extract the sort order. So, given below is a matrix that shows the number of additional Sub-Logical SQLs

In the next & the last post of this series, i will summarise the findings so far and perhaps try to come out with a set of high-level pointers that should hopefully help people trying to implement BI EE 11g on Essbase.

Basically this is a pivot report where one of the attribute columns (Accounts dimension Generation 3) has been moved to the column edge. Let’s start with looking at the MDX generated.

Interestingly, this is a very efficient MDX and this is what other tools would generate as well. Now, why did we see a completely different MDX in the last blog post. To understand that let’s try to bring a new hierarchical column (with Unbalanced Hierarchy) Market. And drill to the second generation and then look at the MDX generated.

Interestingly there are 3 MDX that are generated. They are given below

We can clearly see that the last 2 queries are very efficient and are actually needed to produce the right results. But what i am not sure of is the need for the first MDX query. To understand why there are 3 MDX queries, lets take a look at the Logical SQL.

If you notice, there are 2 UNION ALL clauses combining 3 separate Logical SQLs. The first logical SQL and the second one are different only in the filters (i.e a check where the hierarchical column is not null). Lets take each one of these Logical SQLs and fire them separately.

As you see there is no difference at all between these 2 queries apart from the Market member appearing in the 2nd query. Somehow first query seems to be an unwanted query as the 2nd one more or less achieves the same result. Not sure whether this can be classified as a bug but this definitely will add increased overhead on Essbase Servers to fire an extra query. Let’s now start making the pivot report a bit more complex. First let’s move the Market dimension from the Row edge to the column edge above the Accounts. Then lets add the Product Value hierarchy in the row edge.

A very simple report. This generates in total 6 Logical SQLs combined through UNION ALL. Out of this 6, 4 are expected as seen in the first part (if both the hierarchical columns are in the row-edge we will get only 4 Logical SQLs and in turn 4 MDX). The 4 logical sqls are represented below

a. Generation 1 of Product with Generation 1 of Market
b. Generation 1 of Product with Generation 2 of Market
c. Generation 2 of Product with Generation 1 of Market
d. Generation 2 of Product with Generation 2 of Market

In addition to these 4, the above pivot table generates 2 extra Logical SQLs which to me is somewhat un-necessary. Let’s look at the corresponding MDX statements of the additional Logical SQLs

The above 2 MDXs produce results for the same cells already covered by the other 4 MDX statements. This is totally un-necessary and this happens when we include a Value-Based hierarchy. Let’s see what happens when we include 2 level based hierarchies instead of value based ones.

Again another massive Logical SQL with 8 sub-logical SQLs this time, each producing its own MDX. There are a couple of things that we can infer from this.

1. BI EE 11g MDX generation is basically an extension of BI EE 10g MDX generation when broken down into individual sub-logical SQLs.

2. The BI Server itself hasn’t changed much apart from some additional functions. The core that has changed is the presentation service which translates every interaction into a corresponding logical SQL(each combined through UNION ALLs).

3. The biggest change when it comes to BI EE 11g is the dynamic logical SQL for every drill. The more you drill bigger will be the logical SQL and in turn larger will be the number of MDX fired. There is pure mathematics into this and in the next post let’s look at how we can arrive at the number of Logical SQLs generated by a report and also the potential number of MDX generated.