Rittman Mead - Delivered Intelligence

As mentioned in the blog entry here, BI Server 11g now supports more functions which were added to support the new UI features. Each of these functions can be used as part of the Logical SQL or can even be used directly in the Answers UI. In this blog entry we shall be looking at how some of these functions can be used.

1. SELECT_PHYSICAL: Though this is not a function, this is a new feature supported by BI Server that can bypass the BMM layer in the SQL queries. So, any SQL that uses SELECT_PHYSICAL will be more closer to the ANSI SQL standard. There are 2 main advantages of this

a. Any external reporting tools that generate SQL automatically can now leverage the functionality of BI Server. For example, in the blog entry here i had shown the difficulty in actually using Logical SQL with Cognos. But introduction of SELECT_PHYSICAL should alleviate this issue thereby making BI Server open for better integration externally.

b. Using SELECT_PHYSICAL one can now make direct joins across data sources without actually having the need to model them separately in the Business Model and Mapping layer.

With the addition of more in-memory operations in the BI Server, this should act as a point of integration for external tools that have the need to leverage the metadata framework of BI EE.

To illustrate how this works, lets look at the repository shown below,

As you see, the above repository contains all the relevant tables in the SH schema. Now, to use the SELECT_PHYSICAL command, lets go to the Issue SQL section of the BI EE and fire the SQL given below

SELECT_PHYSICAL
0 s_0,
CHANNEL_CLASS s_1,
SUM(AMOUNT_SOLD) s_2
FROM
"ORCL - SH".."SH"."CHANNELS"
A,
"ORCL - SH".."SH"."SALES" B
WHERE
A.CHANNEL_ID = B.CHANNEL_ID
GROUP BY CHANNEL_CLASS

Remember, though this is a SELECT_PHYSICAL SQL statement, BI Server will still do the parsing as the objects will have to reside inside the repository.

This option can be enabled at the ODBC/JDBC level as well. That way the DSN can be configured to just accept SELECT_PHYSICAL or SELECT commands.

I will cover more details of how this can be leveraged from external tools like QlikView, Cognos etc in a future blog entry.

2. AGGREGATE AT:

This function adds more multi-level analysis capabilities within the BI Server. This function provides the same functionality as the Level Based Measures. What this does is, it provides an ability to extract the values of a parent level in a query without having to SUM up the values in the lower level. To illustrate, lets look at the simple report below

Now, what we need is an ability to display the value of yearly sales alongside the quarterly sales. To do that, we need to enter the formula as shown below

AGGREGATE("Sales"."AMOUNT_SOLD" AT "Times"."TimesDim - Calendar Year"."Calendar Year")

If we look at the report now, we should see that we can compare the Yearly sales along the Quarterly sales i.e. the AGGREGATE AT function will not respect the filter on that dimension. If you want the filters to be honored as well then the older SUM BY functions can still be used.

If you look at the Physical SQL generated, you would actually notice 2 separate WITH clauses (minimum). One for the Sales figure. And the other for the Yearly Sales figure. The Yearly Sales figure will not have the filter applied.

WITH
SAWITH0 AS (select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3,
     D1.c4 as c4,
     D1.c5 as c5
from
     (select sum(T44515.AMOUNT_SOLD) as c1,
               T44523.CALENDAR_QUARTER_DESC as c2,
               T44523.CALENDAR_QUARTER_ID as c3,
               T44523.CALENDAR_YEAR as c4,
               T44523.CALENDAR_YEAR_ID as c5,
               ROW_NUMBER() OVER
(PARTITION BY T44523.CALENDAR_QUARTER_ID
ORDER BY T44523.CALENDAR_QUARTER_ID ASC) as c6
          from
               TIMES T44523,
               SALES T44515
          where  ( T44515.TIME_ID = T44523.TIME_ID and
(T44523.CALENDAR_QUARTER_NUMBER in (1, 2)) )
          group by T44523.CALENDAR_QUARTER_DESC,
T44523.CALENDAR_QUARTER_ID, T44523.CALENDAR_YEAR,
T44523.CALENDAR_YEAR_ID
     ) D1
where  ( D1.c6 = 1 ) ),
SAWITH1 AS (select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3
from
     (select sum(T44515.AMOUNT_SOLD) as c1,
               T44523.CALENDAR_YEAR as c2,
               T44523.CALENDAR_YEAR_ID as c3,
               ROW_NUMBER() OVER (PARTITION BY
T44523.CALENDAR_YEAR_ID ORDER BY T44523.CALENDAR_YEAR_ID ASC) as c4
          from
               TIMES T44523,
               SALES T44515
          where  ( T44515.TIME_ID = T44523.TIME_ID )
          group by T44523.CALENDAR_YEAR, T44523.CALENDAR_YEAR_ID
     ) D1
where  ( D1.c4 = 1 ) ),
SAWITH2 AS (select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3,
     D1.c4 as c4,
     D1.c5 as c5,
     D1.c6 as c6
from
     (select 0 as c1,
               D1.c2 as c2,
               case  when D2.c2 is not null then D2.c2
when D1.c4 is not null then D1.c4 end  as c3,
               D1.c1 as c4,
               D2.c1 as c5,
               D1.c3 as c6,
               ROW_NUMBER() OVER (PARTITION BY D1.c2, D1.c3,
case  when D2.c2 is not null then D2.c2 when D1.c4 is not null
then D1.c4 end  ORDER BY D1.c2 ASC, D1.c3 ASC,
case  when D2.c2 is not null then D2.c2
when D1.c4 is not null then D1.c4 end  ASC) as c7
          from
               SAWITH0 D1 left outer join SAWITH1 D2 On D1.c5 = D2.c3
     ) D1
where  ( D1.c7 = 1 ) )
select D1.c1 as c1,
     D1.c2 as c2,
     D1.c3 as c3,
     D1.c4 as c4,
     D1.c5 as c5
from
     SAWITH2 D1
order by c1, c3 NULLS FIRST, c2

3. CALCULATEDMEMBER:

This function is used to derive calculated members using members at any level of a hierarchy. In 10g, we can do this from a pivot table. In 11g, this function has been introduced at the BI Server level itself so that we can leverage these directly from the repository.

To illustrate this, consider the hierarchy shown below

In this hierarchy, we want to find out the difference between Steven King’s Salary and his 2 direct reportees (Neena Kochhar and Lex De Haan). To do this, we need to use a function given below

CALCULATEDMEMBER("HR"."Employee"."EmployeeDim",'Test', MEMBER(100) - MEMBER(101) -MEMBER(102))

This would push the function directly on the parent-child hierarchy and give us the correct values. This function cannot be used if there are other attributes of the same dimension in the report. This is generally used by external application tools. But within BI EE Answers, there is no need for using this as we can achieve the same using Custom Groups and Calculations.

4. EVALUATE_ANALYTIC:

This is another significant new feature that can be useful to push down the oracle database analytic functions. I have covered EVALUATE number of times before for function shipping database functions. EVALUATE_ANALYTIC extends that further and provides us an ability to function ship the analytic functions as well. The syntax of this function remains the same. I will cover this in detail in another blog post as there are quite a few use cases where EVALUATE_ANALYTIC can prove to be very useful.

5. HIERARCHICAL FUNCTIONS:

BI Server now supports hierarchical functions like ISCHILD, ISROOT, ISANCESTOR, ISDESCENDANT & ISLEAF that can be used to traverse a Parent-Child hierarchy. These functions generally return a boolean data type (true/false) and hence can be used in filters to restrict data. They can also be used in column formula case statements.

For example, in a hierarchy if we want to know whether a specific member is a leaf node or not, we can use the below function to display Yes/No as an attribute of the parent-child hierarchy

CASE WHEN ISLEAF(Employee.EmployeeDim) THEN 'Yes' ELSE 'No' END

Next up is some interesting posts on the BI EE integration with Fusion Middleware & Essbase. I will be doing these posts towards the end of next week.

As Mark had pointed out earlier here, one of the significant changes in the BI EE 11g release is the migration of security from the BI EE repository to Weblogic. Users and Groups will no more be maintained within the rpd. Even in an external authentication like LDAP, Groups will not be maintained within the repository. This is good in many ways as it can significantly reduce the size of the repository. The downside though is, in offline mode unless there is a change in any security object (like custom filters to a group etc), the users & groups will not become visible. The use of Web Catalog Groups is no more recommended. It still exists primarily for supporting upgrades. In this blog post, lets see how the new weblogic based security works.

There is a terminology change that we need to get used to while moving to BI EE 11g. Groups now refer to Weblogic groups. BI EE specific groups are called as Application Roles. The figure shown below explains the relationship between Users, Groups & Application Roles.

Weblogic security is much more comprehensive and it can support multiple external directories like LDAP, OID etc. It also supports table authentication. So, if you are planning on setting up external authentication, its preferred to set it up at the Weblogic layer. We can also setup external directories within BI EE (like in 10g).

To start with lets create a simple user bieeuser1 in the Weblogic Console Security Realm (http://localhost:7001/console).

While creating a user we can assign it to a provider. A provider is the authentication directory against which users will be authenticated. In our case, we have created the user in the default weblogic directory. In the same way lets create 2 more users bieerpdadmin & bieewebcatadmin.

The idea is to provide the following privileges to each of the users that we have created

bieeuser1 – Casual user with no access to the repository but with read only access to dashboards (no answers)
bieerpdadmin – Repository Administrator with full control of the Repository objects but with no control on the web catalog.
bieewebcatadmin – Web Catalog Administrator with full control on the web catalog but with no access to the repository

Remember, its not always necessary to assign users to weblogic groups. Users can be directly assigned to application roles. Once the users are created, lets login to the Weblogic enterprise manager and navigate to the coreapplication under the Business Intelligence folder.

Now, right click on the coreapplication and choose the Security->Application Roles option

This should list all the default application roles available within BI EE.

Let’s neglect the existing roles(ideally we will be reusing/reassigning the default roles as much as possible) and create 3 new roles.

1. BIEERoleUser
2. BIEERoleRPDAdmin
3. BIEERoleWebCatAdmin

For now lets not add any users/roles to these roles.

Lets now navigate to the Application Policies and start granting specific rights/grants to the roles that we created. Remember in any ADF application we need to define application policies so that they can be granted to individual users. We start with assigning the following privileges to the BIEERoleUser

1. resourceType=oracle.bi.server.permission,resourceName=oracle.bi.server.queryUserPopulation

This basically provides an ability to query the BI Server repository.

Lets assign this privilege to the BIEERoleUser application role.

In the same way lets assign the following privileges to the BIEERoleRPDAdmin and BIEERoleWebCatAdmin

For BIEERoleRPDAdmin

1. resourceType=oracle.bi.server.permission,resourceName=oracle.bi.server.ManageRepositories

For BIEERoleWebCatAdmin

1. resourceType=oracle.bi.server.permission,resourceName=oracle.bi.server.queryUserPopulation

Now that the roles have the right privileges, lets go back to the Application Role screen and start assigning the 3 users to each of these roles.

After assigning these, if you try to login to the repository in online mode, only the user bieerpdadmin can login. We have satisfied one requirement, that is we have created a user who is the repository admin. For presentation services, by default during install the privileges in presentation services get assigned to only 3 default groups BIAdministrator, BIAuthor and BIConsumer. So, to make bieewebcatadmin a true webcatalog administrator we have to go to the Application Roles and add the BIAuthor & BIAdministrator role to BIEERoleWebCatAdmin

This will automatically make the user bieewebcatadmin user to be the administrator of the webcatalog. Initially it is mandatory to assign a new role to the BIAdministrator role as that is pre-configured to be the only role having access to Privileges in Presentation Services. Granting BIAdministrator to a role also opens up the access to the repository. This can be disabled as well. It is not necessary to always create new roles and assign policies to them. Existing roles can be re-used and re-assigned. The entire security model is now more aligned with Oracle Platform Security framework so that all the fusion applications can integrate with each other seamlessly.

One of the important points to consider when moving from BI EE 10g to 11g is in understanding the BI Server behavior changes. Though there are quite a lot of changes in the underlying code for performance improvement etc, introduction of new features in 11g can potentially alter your generated queries in 11g. In this blog entry i will basically cover 2 main important changes that have been introduced in 11g. There are more actually, but that is something we will try to cover in our training days event.

1. Level Based Measures at Detail Level:

One significant change in 11g is the handling of level based measures at the detail level. For example, lets consider a measure AMOUNT_SOLD that is assigned to the detail day level of the date dimension.

In 10g, whenever we bring this column for reporting, we would basically see something as shown below i.e. the level will always be enforced.

In the case of 11g, there are 2 types of behavior

a. If the report contains normal attribute columns and when the query is at a grain higher than the measure grain (say the report is at the Year level), then normal aggregation will be applied and the level will not be enforced like in 10g. An example screenshot is given below

b. If the report contains hierarchical columns, then if the query is at higher grain than the measure, the measure values will become null. The values will appear at that grain or below. An example screenshot is given below

2. Priority Groups & Order of LTS:

In 11g, there is a new concept called Priority Groups. In 10g, the LTS that determines the final query is dependent on the ordering of the LTS in the dimension logical table(in most cases – but there are cases where we cannot be always be sure to guarantee this behavior). But in 11g, this is determined by the priority order set in each and every LTS.

For example, lets consider a case where there are 2 aliases for the same dimension CHANNELS as shown below

In the business model and mapping layer, lets assume that both the LTS are mapped to all the columns. Both the LTS are at the same detail level. Now, lets assign a priority group of 0 for the Channel Alias – B LTS and a priority group of 1 for the Channel Alias – A LTS. Also the SALES LTS of the Fact table will also be assigned to the priority group 0. In 11g, the LTS with the least number is considered as the one that has to be chosen for querying (highest priority)

Now, lets create a report containing columns from Channels dimension and the Sales fact.

If you look at the query of the report, you will notice that Channels Alias – B will be chosen as it is the one having the highest priority.

Lets now go back to the repository and reassign the highest the priority to Channels Alias – A as shown below

For the same report, if you look at the query now, you will notice that Channels Alias – A will be chosen instead of the B alias.

Mark has already explained the new Parent-Child hierarchy new feature in 11g here. Parent Child hierarchies are different from normal level based hierarchies not only in the nature of their structure but also in the way aggregations are done. For example, in a level based hierarchy, the most common type of aggregation that is done involves rolling up the numbers from the lower most level. But in the case of parent-child hierarchy, an aggregation by default enforces the roll-up across all descendant members. To understand this lets try to take this through an example.

Lets consider a sales department with the employee hierarchy shown below

Its a simple parent child employee hierarchy found in the oracle HR schema. Now the requirement for us is to model the Salary attribute of an employee in 3 ways

1. As an attribute – Show the salary of each employee as an attribute in the parent child hierarchy.
2. As a measure (All intermediary Roll-ups) – Roll-up the salary figures of all the employees who roll into the employee. For example, Scott King will have a Salary equal to sum of salaries of all the employees who report to him, including the ones who are his direct & in-direct reporting employees.
3. As a measure (Level-0 Roll-up) – Roll-up the salary figures from the lower most level till the top node For example, in the above figure, Scott King will have a Salary Roll-up figure of all the employees who are the lowermost level (not including the intermediary nodes – very similar to normal level based hierarchy roll-ups)

The above requirement is quite straightforward, but to achieve similar such aggregations in a normal level-based hierarchy, we will have introduce fragmentation & other techniques. But with parent-child hierarchy modeling now available in BI EE, all the above 3 requirements can quite easily be achieved.

Following is the employees parent child table that stores all the attributes of an employee

We start off with modeling this as a normal dimension table. As shown in Mark’s blog entry, we then create the Parent-Child table that stores all the relationships within the employee hierarchy (if you look at the script used in creating this, it will be containing a Connect By statement that will connect a root hierarchical node to every member that is part of its hierarchy).

Once this is created, we start with including the Salary as an attribute within the Employee dimension itself

This will serve our first requirement i.e show salary as an attribute of every employee. Now, to model the 2nd requirement i.e. to model salary as a measure and then make it to rollup from all the intermediary hierarchical members, we start with defining an alias on the main Employee table. Then join this table to the custom parent child employee table. The join structure of all the 3 tables is given below

We would then be using the salary column from the new employee alias table in our logical fact table as shown below

Lets now look at the report now and see what BI EE generates.

As you see, BI EE has generated the correct results with the roll-up of intermediary members as well. This is something that is quite difficult to model in BI EE 10g. If we look at the query generated (just the key one that generates the measure as BI EE now generates lots of intermediary sub-queries to support contextual drilling) you would notice that the default Parent-Child behavior is to achieve a roll-up across a hierarchy

To achieve the 3rd requirement i.e. to model a level based hierarchy kind of rollups from level-0, we create another alias in the physical layer and then model it the same way as the 2nd requirement. In the fact logical table, we then create a new measure that will map to the new physical alias as shown below

To ensure that we roll-up only from level-0 members, we then apply a filter in the fact logical table source as shown below

Lets now take a look at what BI EE generates

As you see, BI EE now generates the roll-ups only from level-0 members as against every descendant. We can have even more granular roll-ups based on custom defined attributes.

As well as the three Rittman Mead OBIEE 11g Training Days events we’re running in London, Atlanta and Bangalore later this year, I’ve also agreed to do a number of Enterprise Business Intelligence Masterclasses for Oracle University, including a visit down to Australia and Singapore. These events are seminar-style, cover OBIEE 11g as well as OWB11gR2/ODI 11gR1 and a bit of Essbase 11.1.2, and are an ideal high-level introduction to the technology behind the Oracle 11g BI&DW technology stack.

More dates will be announced over the next few months, but for now, here’s the current set of confirmed dates and venue:

• De Meern (Utrecht), Netherlands, 10-11 November 2010
• Bratislava, Slovakia, 25-26 November 2010
• Melbourne, Australia, 6-7 December 2010
• Sydney, Australia, 9-10 December 2010
• Singapore, Singapore, 13-14 December 2010
• Oslo, Norway, 14-15 February 2011

I’ll post details of any future events on our Events website page.