I found Oracle support document 1258925.1, which has a handle on the problem. It describes a deficiency OLAP has with indexing attributes that contain newline characters. However… the note references version 11.2.0.1 of Analytic Workspace Manager (AWM), where indexing of all attributes is a binary choice. However, in 11.2.0.2 of AWM, we can choose to index on an attribute-by-attribute basis, as seen in the attribute details pane:

So here’s what I know: I have an indexed attribute with newline characters. ETL processing and data quality notwithstanding (that seems like a strange thing to survive an ETL process), I need to find out which attribute to un-index. Problem is… there is nothing in the maintenance logs that tell me which attribute is the problem. (By the way: if this information exists and you know where it is, then please comment and I’ll update the blog post.)

So I wrote this little piece of PL/SQL that did the trick for me, and I wanted to share it. CHR(10) is the construct we use in PL/SQL to denote a newline character, so I construct a query against each column to determine whether it contains any:

SQL> DECLARE
  2     l_table      VARCHAR2(30)    := 'DIM_FUND';
  3     l_results    NUMBER;
  4  BEGIN
  5     FOR x IN ( select 'select count(*) from '
  6                       ||l_table
  7                       ||' where regexp_like('
  8                       ||column_name
  9                       ||', chr(10))' stmt,
 10                      column_name
 11                  from dba_tab_columns
 12                 where table_name=l_table
 13              )
 14     LOOP
 15  --      dbms_output.put_line( x.stmt );
 16
 17        EXECUTE IMMEDIATE x.stmt
 18        INTO l_results;
 19
 20        IF l_results > 0
 21        THEN
 22           dbms_output.put_line
 23           ( x.column_name
 24             ||': '
 25             || l_results
 26           );
 27
 28        END IF;
 29
 30     END LOOP;
 31
 32  END;
 33  /
FUND_FIN_MANAGER_TITLE: 3

PL/SQL procedure successfully completed.

Elapsed: 00:00:01.82
SQL>

This did the trick! I un-indexed the attribute in AWM and the dimension maintenance procedure was successful.

According to the Oracle BI 11.1.1.5 Certification Matrix, Oracle OLAP 10.2, 11.1 and 11.2 are all supported as data sources. No specific patch levels are mentioned, but in general with Oracle OLAP you should always be on the latest patch release as critical updates are often made available through metalink even between major database releases. That said, for this example I’ve used the standard, unpatched version of Oracle OLAP that comes with Oracle Database 11.2.0.1, along with Analytic Workspace Manager 11.2.0.1.0 downloaded from OTN.

For the example I’ll be working with the Global sample OLAP dataset that’s also available from OTN, which in Analytic Workspace Manager looks like this:

This dataset contains two cubes, which join to either all or a subset of the dimensions within the analytic workspace. There’s also some derived measures in the cubes, and all of the data has been processed and pre-aggregated, so that when viewed in Analytic Workspace Manager it looks like this:

To import the metadata for this Oracle OLAP analytic workspace into an Oracle BI Repository, make sure you are using OBIEE 11.1.1.5 and start the Oracle BI Administration tool from the Windows Start Menu. Either open an existing repository online, or create a new one offline, and select Oracle OLAP as the data source. Then, enter the following details to connect to the database and view the list of analytic workspaces:

Connection Type : Oracle OLAP
Data Source Name : host:port:sid, e.g. localhost:1521:orcl
User Name : Username of account containing the analytic workspace, e.g. global
Password : Password for the above account
Target Database : Name for imported physical database in RPD physical layer, or name of existing physical database

There’s a potential couple of “gotchas” here. Firstly, don’t type in a TNSNAMES entry (for example, “orcl”) as the Data Source Name, as you’ll get the following error if you do:

This error happens because the Administration Tool uses a new AWImportService java class to access the Analytic Workspace metadata, and it only takes connection details in the JDBC format (you can switch back to a TNSNAMES entry later in the process). If you hit this error, re-enter the connection details in the host:port:sid format and it’ll work.

The other error you might hit is if Javahost isn’t running when you try to make the connection.

This is because the Administration Tool uses Javahost (the system component that provides the ability for the C-based OBIEE components to run Java processes) to access AWImportService, even if you’re opening the repository offline. If this happens, make sure Javahost is running and it’ll then work, but this does then suggest that you need to perform this import on a server or workstation where OBIEE 11.1.1.5 is installed and running, otherwise the Administration Tool won’t be able to access a Javahost instance.

Once you connect, you’re presented with a list of all the analytic workspaces accessible to the account you connected with, which you can then select for import into the repository. Where this differs to importing Analytic Workspace-based views in OBIEE 10g is that you are also importing the hierarchies into the repository, as you do with an Essbase, SAP B/W or Microsoft Analysis Services metadata import.

Once the import completes, you can see the physical database, Analytic Workspace, Oracle OLAP dimensions, hierarchies, cubes and measures that you’ve imported, in the Physical layer of your repository.

Before you can access data from the analytic workspace through the repository, you’ve got to make a quick change in the connection pool connection details to go back to a TNSNAMES entry. Double-click on Connection Pool within the new database and edit the Data source name entry so that it uses a TNSNAMES entry rather than the host:port:sid that’s there because of the previous step.

Looking inside the Physical layer metadata that’s been created using the Import Wizard, you can see that hierarchies within an Oracle OLAP dimension can be either level-based, or parent-child. This is the same as with Essbase physical hierarchies, although from an initial test it doesn’t look like you can alter an already-imported hierarchy from one type to the other, meaning that you’ll only be able to switch from level-based to value-based hierarchies if you change the underlying hierarchy type in the analytic workspace itself.

This could potentially be an issue as, like with Essbase sources, it doesn’t appear you can “re-import” an analytic workspace definition back into the repository to pick up new hierarchy levels. It’s early days yet though but, as with Essbase sources, if you’re testing this out, bear in mind that the analytic workspace definition may change over time, and you’ll need to work out how to reflect those changes in your repository without breaking all the existing mappings and analyses.

So once you’ve imported the analytic workspace metadata into the physical layer of the repository, as with an Essbase database, you can then drag and drop the cube definition first into the Business Model and Mapping Layer, and then the Presentation Layer, of your repository, so that it looks like this:

Now the various layers of the repository metadata are complete, I upload the RPD file to the Oracle BI Instance ready to run my first analysis. Before I do this though, I enable query logging at level 5, so that I can see the physical SQL, and the logical execution plan, for the analyses I create. With the Oracle BI repository online now, I select Manage > Identity from the BI Administration application menu, and notice that, even though I’m connected to an online repository, whilst my application roles are listed, there are no users.

So what’s going on here then? Well, in 11.1.1.5, there’s an enhancement with the Identity Manager dialog where, by default, users from the LDAP server are filtered so that WebLogic doesn’t try and return all LDAP users every time you open this dialog. To change this filter (which by default is set to blank, and has to be set to “*” to return all users), select Action > Set Online User Filter from the Security Manager dialog and then enter the search filter.

So now with logging set to level 5 for the weblogic user, I use my web browser to bring up the Oracle BI homepage and run a simple query to return the total for costs across all dimensions.

So what does the SQL look like, that the BI Server has generated to return this result? I take a look at the NQQuery.log file and find the entry for this analysis.

WITH
SAWITH0 AS (select distinct case when count(*) > 1 then null else max(units_cube_cost) end as c1
from
(select *
from table(olap_table('GLOBAL.GLOBAL duration session', '', '',
'measure units_cube_cost from UNITS_CUBE_COST ' ||
'dimension channel_id as varchar2(100) from CHANNEL with ' ||
'  attribute channel_level as varchar2(100) from CHANNEL_LEVELREL ' ||
'  hierarchy CHANNEL_PARENTREL(CHANNEL_HIERLIST ''PRIMARY'') ' ||
'    inhierarchy CHANNEL_INHIER ' ||
'dimension customer_id as varchar2(100) from CUSTOMER with ' ||
'  attribute customer_level as varchar2(100) from CUSTOMER_LEVELREL ' ||
'  hierarchy CUSTOMER_PARENTREL(CUSTOMER_HIERLIST ''SHIPMENTS'') ' ||
'    inhierarchy CUSTOMER_INHIER ' ||
'dimension product_id as varchar2(100) from PRODUCT with ' ||
'  attribute product_level as varchar2(100) from PRODUCT_LEVELREL ' ||
'  hierarchy PRODUCT_PARENTREL(PRODUCT_HIERLIST ''PRIMARY'') ' ||
'    inhierarchy PRODUCT_INHIER ' ||
'dimension time_id as varchar2(100) from TIME with ' ||
'  attribute time_level as varchar2(100) from TIME_LEVELREL ' ||
'  hierarchy TIME_PARENTREL(TIME_HIERLIST ''CALENDAR'') ' ||
'    inhierarchy TIME_INHIER ' ||
'loop optimized ' ||
'row2cell r2c '
))
where channel_level = 'TOTAL' and customer_level = 'TOTAL' and product_level = 'TOTAL' and time_level = 'TOTAL' and (units_cube_cost is not null)
))
select distinct 0 as c1,
D1.c1 as c2
from
SAWITH0 D1

So what’s different here, compared to previous support for Oracle OLAP, is that the BI Server is writing the query using the OLAP_TABLE function, rather than accessing AW objects via SQL views, and it’s using 11g features such as “loop optimized” that Stewart covered in this earlier blog post on new capabilities in Oracle OLAP 11g. Notice also that the OLAP_TABLE function references all dimensions associated with the measure (albeit at “TOTAL” level) even though they’re not included in the analysis, as that’s the way that OLAP_TABLE works.

Next I run another simple query, this time just accessing an attribute from the product dimension.

This time, as I’m only accessing a dimension and note a measure, the OLAP_TABLE function just references the dimension in the function clause.

WITH
SAWITH0 AS (select distinct family_product_long_descripti1 as c1
from
(select *
from table(olap_table('GLOBAL.GLOBAL duration session', '', '',
'dimension product_id as varchar2(100) from PRODUCT with ' ||
'  attribute product_level as varchar2(100) from PRODUCT_LEVELREL ' ||
'  hierarchy PRODUCT_PARENTREL(PRODUCT_HIERLIST ''PRIMARY'') ' ||
'    inhierarchy PRODUCT_INHIER ' ||
'    familyrel family_product_long_descripti1 as varchar2(100) from ' ||
'      PRODUCT_FAMILYREL(PRODUCT_LEVELLIST ''FAMILY'') ' ||
'      label PRODUCT_LONG_DESCRIPTION ' ||
'row2cell r2c '
))
where product_level = 'FAMILY'
))
select distinct 0 as c1,
D1.c1 as c2
from
SAWITH0 D1
order by c2

So what about using a hierarchical column? As a quick reminder, when you use a a hierarchical column within OBIEE 11g, the Presentation Server sends a request to the BI Server, which then generates logical requests for each subtotal within the hierarchy. For relational sources, the BI Server then joins these logical queries together into a single (rather large) physical SQL request using subquery factoring (subqueries and WITH clauses). With Essbase (and other MDX-based) sources though the BI Server can’t do this, and instead fires of lots of individual MDX queries to the Essbase server, and then joins the resultsets together again before passing the results back to the Presentation Server. This can cause quite an excessive load on the Essbase server, so I’m keen to see how this works with Oracle OLAP.

To start off, I create a simple analysis where I drill into the product dimension, and display it along with one of the measures in the analytic workspace, like this:

Taking a look at the logical execution plan and resulting physical SQL that this generates, the BI Server has indeed generated multiple logical SQL queries (as it would with relational sources), but unlike an Essbase source the resulting physical SQL is all in a single query. So in some senses OBIEE 11g queries against Oracle OLAP sources are more efficient than Essbase ones, as the BI Server is able to keep the hierarchical column query as a single SQL statement rather than generating lots of MDX statements as it does with MDX-based sources.

Another nice feature of Oracle OLAP is that it’s very easy, using Analytic Workspace Manager, to derive time-series measures based off of a set of base measures. In the Oracle OLAP cube that we’ve imported in, you can see these under the presentation fact table, and these can be selected like any other measure to add to your analysis criteria.

So one last test I’d like to do, because this is a weak area of the way that Essbase is integrated into OBIEE, is around function push-down. For Essbase and other MDX-based OLAP sources, very few OBIEE functions are shipped-down to their corresponding MDX functions, which can affect performance as the BI Server has to provide ranking and other such calculations for Essbase sources, even though they’re more than capable of doing this themselves. So let’s try a rank function in an analysis and see what happens.

So how does the SQL look?

WITH
SAWITH0 AS (select case when count(*) > 1 then null else max(units_cube_profit) end as c1,
account_customer_long_descrip1 as c2,
account_customer as c3
from
(select *
from table(olap_table('GLOBAL.GLOBAL duration session', '', '',
'measure units_cube_profit from UNITS_CUBE_PROFIT ' ||
'dimension channel_id as varchar2(100) from CHANNEL with ' ||
'  attribute channel_level as varchar2(100) from CHANNEL_LEVELREL ' ||
'  hierarchy CHANNEL_PARENTREL(CHANNEL_HIERLIST ''PRIMARY'') ' ||
'    inhierarchy CHANNEL_INHIER ' ||
'dimension customer_id as varchar2(100) from CUSTOMER with ' ||
'  attribute customer_level as varchar2(100) from CUSTOMER_LEVELREL ' ||
'  hierarchy CUSTOMER_PARENTREL(CUSTOMER_HIERLIST ''MARKET'') ' ||
'    inhierarchy CUSTOMER_INHIER ' ||
'    familyrel account_customer as varchar2(100) from ' ||
'      CUSTOMER_FAMILYREL(CUSTOMER_LEVELLIST ''ACCOUNT'') ' ||
'      label CUSTOMER ' ||
'    familyrel account_customer_long_descrip1 as varchar2(100) from ' ||
'      CUSTOMER_FAMILYREL(CUSTOMER_LEVELLIST ''ACCOUNT'') ' ||
'      label CUSTOMER_LONG_DESCRIPTION ' ||
'dimension product_id as varchar2(100) from PRODUCT with ' ||
'  attribute product_level as varchar2(100) from PRODUCT_LEVELREL ' ||
'  hierarchy PRODUCT_PARENTREL(PRODUCT_HIERLIST ''PRIMARY'') ' ||
'    inhierarchy PRODUCT_INHIER ' ||
'dimension time_id as varchar2(100) from TIME with ' ||
'  attribute time_level as varchar2(100) from TIME_LEVELREL ' ||
'  hierarchy TIME_PARENTREL(TIME_HIERLIST ''CALENDAR'') ' ||
'    inhierarchy TIME_INHIER ' ||
'loop optimized ' ||
'row2cell r2c '
))
where channel_level = 'TOTAL' and customer_level = 'ACCOUNT' and product_level = 'TOTAL' and time_level = 'TOTAL' and (units_cube_profit is not null)
)
where  ( 1 = 1 )
group by account_customer_long_descrip1, account_customer),
SAWITH1 AS (select distinct 0 as c1,
D1.c2 as c2,
D1.c1 as c3,
Case when D1.c1 is not null then Rank() OVER ( ORDER BY D1.c1 DESC NULLS LAST ) end as c4,
D1.c3 as c5
from
SAWITH0 D1)
select D1.c1 as c1,
D1.c2 as c2,
D1.c3 as c3,
D1.c4 as c4
from
SAWITH1 D1
order by c1, c2

Well, whilst the RANK() calculation hasn’t been converted to OLAP DML and performed by the analytic workspace, it has been performed by the database rather than having to be done by the BI Server. In a way it’s academic as by the time the numbers have been summed up, ranking them is a much simpler job, but it looks like aggregate and other analytic functions pushed-down by the BI Server will get added to the Oracle SQL statement that summarizes the results of the OLAP_TABLE query.

So, dare I say it, it looks like proper Oracle OLAP support in OBIEE 11.1.1.5 works pretty well, at least based on first impressions. It doesn’t suffer from the same “square peg in a round hole” syndrome that you get when the BI Server tries to convert from relational SQL to multidimensional MDX, because it’s the Oracle Database that does this conversion, through the OLAP_TABLE function. Of course, with all new features and in particular, new data source support, the “devil is in the detail”, but it’s certainly an encouraging start. If anyone implements Oracle OLAP through this new approach, I’d be interested to hear how you get on.

1. It required an Essbase License
2. Any change to the OLAP metadata required an XOLAP cube rebuild within Essbase

Some time last year, Simba Technologies announced an MDX OLE-DB provider for Oracle OLAP. So far i did not get an opportunity to test this though it looked promising. Couple of weeks back we got an evaluation copy from Simba to test the driver (i will have to thank Simba and their Oracle OLAP – MDX provider team for providing us with an evaluation copy). This driver basically provides an ability for Excel users to leverage the power of Oracle OLAP using the Excel Pivot Tables/Charts etc. At a high level this driver does the following

1. End users can use the native Excel functionality to create charts/pivot tables etc
2. The charts/Pivot tables generate MDX (standard OLE-DB based microsoft MDX)
3. Simba driver then converts the MDX to one or more SQL calls to the Oracle OLAP

In this blog entry we will basically see how this driver works. The install process is quite straightforward where we are taken through a set of steps that will setup the OLE-DB driver. This driver will work only for Oracle OLAP versions 11.1.0.7 or above. Then we start off with setting up a DSN to connect to the Oracle OLAP database. Ensure that the client driver of Oracle used in the DSN is atleast of the 11.1.0.7 version.

Once this is setup, from Excel use the Data Connection Wizard to setup a OLE-DB connection through the Simba MDX driver.

This should automatically connect us to the Oracle OLAP schemas.

As you see, we can save a connection to a cube and then the same connection can be reused later for creating more reports. Lets start with creating a simple pivot table report (using native Microsoft Excel Pivot tables)

After this if you notice, we will now be having metadata of Oracle OLAP exposed within the Pivot Table member selection panels. This is very similar to Hyperion Visual Explorer where we are shown all the levels in a dimension and all the hierarchies as well. We can pick and choose either a specific level or we can choose members from multiple levels by applying proper filters. Lets create a very simple report as shown below

As you see, we now have the ability to drill using the native MS pivot table functionality. Lets look at the MDX fired to generate the above query.

SELECT
{[MEASURES].[SALES],[MEASURES].[SALES_YTD]}
DIMENSION PROPERTIES
PARENT_UNIQUE_NAME ON COLUMNS ,
NON EMPTY
CrossJoin(Hierarchize({DrilldownLevel({[TIME].[CALENDAR].[ALL_YEARS].[ALL_YEARS]})}),
Hierarchize({DrilldownLevel({[PRODUCT].[STANDARD].[ALL_PRODUCTS].[ALL_PRODUCTS]})})) DIMENSION PROPERTIES PARENT_UNIQUE_NAME,
[TIME].[CALENDAR].[CALENDAR_YEAR].[CALENDAR_YEAR_END_DATE],
[TIME].[CALENDAR].[CALENDAR_YEAR].[CALENDAR_YEAR_TIME_SPAN],
[TIME].[CALENDAR].[CALENDAR_YEAR].[CALENDAR_YEAR_LONG_DESCR],
[TIME].[CALENDAR].[CALENDAR_YEAR].[CALENDAR_YEAR_SHORT_DESC],
[TIME].[CALENDAR].[CALENDAR_YEAR].[END_DATE],
[TIME].[CALENDAR].[CALENDAR_YEAR].[TIME_SPAN],
[TIME].[CALENDAR].[CALENDAR_YEAR].[LONG_DESCRIPTION],
[PRODUCT].[STANDARD].[DEPARTMENT].[DEPARTMENT_LONG_DESCRIPT],
[PRODUCT].[STANDARD].[DEPARTMENT].[DEPARTMENT_SHORT_DESCRIP],
[PRODUCT].[STANDARD].[DEPARTMENT].[LONG_DESCRIPTION] ON ROWS
FROM
[SALES_CUBE] CELL PROPERTIES VALUE, FORMAT_STRING,
LANGUAGE, BACK_COLOR, FORE_COLOR, FONT_FLAGS,2

As you see the MDX retrieves all the necessary Oracle OLAP level properties as MDX intrinsic properties. This is very interesting. The MDX driver basically does a metadata level mapping between MDX and Oracle OLAP. I am not sure how much of this is documented(in terms of MDX to SQL conversion calls) but again this looks very promising. Now lets look at the SQL that is fired back to Oracle OLAP. The driver can generate multiple SQL Queries for a single MDX call. This is very similar to BI EE 11g (which i shall be covering later once BI EE 11g is GA) where while doing a drill to multiple levels we will see multiple SQL calls being generated.

The first 2 SQL’s generated(for this report) will be for constructing the metadata or the member list for all the dimensions that are part of the query

SELECT
'OLAPTRAIN' AS CATALOG_NAME,
'SALES_CUBE' AS CUBE_NAME,
members.DEPTH AS LEVEL_NUMBER,
members.HIER_ORDER AS MEMBER_ORDINAL,
members.DIM_KEY AS MEMBER_NAME,
1 AS MEMBER_TYPE,
SHORT_DESCRIPTION AS MEMBER_CAPTION,
1 AS CHILDREN_CARDINALITY,
CASE
WHEN (members.PARENT IS NULL) THEN NULL
ELSE members.DEPTH-1
END AS PARENT_LEVEL,
CASE
WHEN members."CALENDAR_QUARTER" IS NOT NULL
AND members.LEVEL_NAME != 'CALENDAR_QUARTER'
THEN '[TIME].[CALENDAR].[CALENDAR_QUARTER].[' || members.PARENT || ']'
WHEN members."CALENDAR_YEAR" IS NOT NULL
AND members.LEVEL_NAME != 'CALENDAR_YEAR'
THEN '[TIME].[CALENDAR].[CALENDAR_YEAR].[' || members.PARENT || ']'
WHEN members."ALL_YEARS" IS NOT NULL
AND members.LEVEL_NAME != 'ALL_YEARS' THEN '[TIME].[CALENDAR].[ALL_YEARS].[' || members.PARENT || ']'
ELSE (CAST (NULL AS VARCHAR2(1)))
END AS PARENT_UNIQUE_NAME,
(CAST (NULL AS VARCHAR2(1))) AS DESCRIPTION
, members.CALENDAR_YEAR_END_DATE AS PROPERTY_4
, members.CALENDAR_YEAR_TIME_SPAN AS PROPERTY_5
, members.CALENDAR_YEAR_LONG_DESCR AS PROPERTY_6
, members.CALENDAR_YEAR_SHORT_DESC AS PROPERTY_7
, members.END_DATE AS PROPERTY_20
, members.TIME_SPAN AS PROPERTY_21
, members.LONG_DESCRIPTION AS PROPERTY_22
, 'TIME' AS DIMENSION_NAME
, 'CALENDAR' AS HIERARCHY_NAME
, members.LEVEL_NAME AS LEVEL_NAME
FROM
"OLAPTRAIN".TIME_CALENDAR_VIEW members
WHERE
members.LEVEL_NAME = 'CALENDAR_YEAR'
ORDER BY MEMBER_ORDINAL, PARENT_UNIQUE_NAME, MEMBER_NAME

SELECT
'OLAPTRAIN' AS CATALOG_NAME,
'SALES_CUBE' AS CUBE_NAME,
members.DEPTH AS LEVEL_NUMBER,
members.HIER_ORDER AS MEMBER_ORDINAL,
members.DIM_KEY AS MEMBER_NAME,
1 AS MEMBER_TYPE,
SHORT_DESCRIPTION AS MEMBER_CAPTION,
1 AS CHILDREN_CARDINALITY,
CASE
WHEN (members.PARENT IS NULL) THEN NULL
ELSE members.DEPTH-1
END AS PARENT_LEVEL,
CASE
WHEN members."COUNTRY" IS NOT NULL AND members.LEVEL_NAME != 'COUNTRY'
THEN '[GEOGRAPHY].[REGIONAL].[COUNTRY].[' || members.PARENT || ']'
WHEN members."REGION" IS NOT NULL AND members.LEVEL_NAME != 'REGION'
THEN '[GEOGRAPHY].[REGIONAL].[REGION].[' || members.PARENT || ']'
WHEN members."ALL_REGIONS" IS NOT NULL AND members.LEVEL_NAME != 'ALL_REGIONS'
THEN '[GEOGRAPHY].[REGIONAL].[ALL_REGIONS].[' || members.PARENT || ']'
ELSE (CAST (NULL AS VARCHAR2(1)))
END AS PARENT_UNIQUE_NAME,
(CAST (NULL AS VARCHAR2(1))) AS DESCRIPTION
, members.ALL_REGIONS_SHORT_DESCRI AS PROPERTY_9
, members.ALL_REGIONS_LONG_DESCRIP AS PROPERTY_10
, members.LONG_DESCRIPTION AS PROPERTY_12
, 'GEOGRAPHY' AS DIMENSION_NAME
, 'REGIONAL' AS HIERARCHY_NAME
, members.LEVEL_NAME AS LEVEL_NAME
FROM
"OLAPTRAIN".GEOGRAPHY_REGIONAL_VIEW members
WHERE
members.LEVEL_NAME = 'ALL_REGIONS'
ORDER BY MEMBER_ORDINAL, PARENT_UNIQUE_NAME, MEMBER_NAME

Then the final query will be for generating the measure values.

SELECT
SALES_CUBE_VIEW.SALES, SALES_CUBE_VIEW.TIME,
SALES_CUBE_VIEW.PRODUCT
FROM
"OLAPTRAIN".SALES_CUBE_VIEW SALES_CUBE_VIEW
WHERE
SALES_CUBE_VIEW.TIME IN ('ALL_YEARS', 'CY2008', 'CY2010','CY2007','CY2009' )
AND SALES_CUBE_VIEW.PRODUCT IN ('ALL_PRODUCTS', '-518', '-519', '-520' )
AND SALES_CUBE_VIEW.CHANNEL = 'ALL_CHANNELS'
AND SALES_CUBE_VIEW.GEOGRAPHY = 'ALL_REGIONS'

If you look at all the queries, they are all OLAP aware i.e. default member filters are applied properly and there is no additional aggregation that is pushed through SQL. This is very interesting and if there are customers using Oracle OLAP, this is one driver that can potentially be put to good use for Excel based reporting.

Currently looks like there is no way to fire custom MDX queries through the Excel 2007 that i have. So, i am not sure how the driver will behave/work when we push custom MDX aggregations like AGGREGATE, SUM etc. Also, i am not sure whether a mapping for all MDX functions(like intersect, union etc) to corresponding OLAP SQL calls exist. But I was told that custom MDX functions should also work well. It is just a case of Excel 2007 not supporting custom MDX queries for the native Pivot Tables.

To bridge this, Simba Technologies came out with a MDX-ODBC bridge which can basically convert MDX to SQL. Oracle did endorse this as can be inferred from this press release here. This is really good as one can easily access Oracle OLAP metadata using MDX without rewriting any of the front-end applications that use MDX. At about the same time, when EPM 11 release came out, Oracle released a new technology called XOLAP which essentially did the same thing i.e exposing Essbase Cubes using the inherent outline structure but by leveraging the SQL at runtime for data. I am not sure whether there is any public XSD based mappings to convert MDX to SQL, but the XOLAP does the conversion extremely well. In fact, the same methodology that i will be showing below can be used for BI EE as well.

What i will be showcasing today is quite simple and can be understood by the high level architecture diagram below

I have used Oracle OLAP 11g R2 since a significant number of bugs have been fixed in this release. The hardest part is in actually locating the 11g version of the Analytic Workspace Manager. The link in OTN is broken and it took me a while in getting the latest version. The latest version of AWM compatible with 11gR2 of Oracle database can be downloaded here. I shall be using the GLOBAL schema for demonstrating this. I start with first building a simple cube from AWM as shown below

There are 2 main things to be aware of before actually starting to the build the XOLAP cube from this Oracle OLAP cube.

1. Each cube built using Oracle OLAP 11g, will have at-least one dimension view per dimension and one cube view. So, the joins between the dimension views and cube views will be similar to joins between a dimension table and fact table. A sample screenshot of a dimension and cube view is given below.

In 11g, there is no necessity for creating the views separately (unlike 10g). And in 11g, the joins across the views perform very efficiently (like a normal table join) whereas in 10g, it is not recommended to make joins between the dimension and cube views.

2. Oracle OLAP is similar to an ASO cube in some respects. The Oracle OLAP 11g SQL views exposed are similar to a write-back partitioned ASO cube. For example, lets consider a simple hierarchy shown below

When we load data and aggregate the cube, the hierarchy stored within Oracle OLAP will be as shown below

Similar to Essbase ASO cube(to enable write-back partitions), this structure for Oracle OLAP provides it a flexibility to store upper level values and also a flexibility to join with the main cube view to get at the data. For example, a normal SQL query on relational data to get the value of member E in the above hierarchy will look as shown below (Assuming each column for each level is named as LEVEL_N)

SELECT LEVEL_2, SUM(MEASURE) FROM DIMENSION
WHERE LEVEL_2 = 'E'
GROUP BY LEVEL_3

But in OLAP since the value of member E is already stored, we should not be doing a SUM aggregation at run-time. Rather we should fire a SQL query to get at the stored value itself. That is done using the SQL query shown below

SELECT LEVEL_2, MEASURE FROM OLAP_FACTDIMENSION_VIEW
 WHERE LEVEL_2 = 'E'

Understanding this is the key in knowing why we need a transparent partition even within Essbase. So, once the Oracle OLAP Cube is built, we log into Essbase Studio and then start building our XOLAP cube. The structure of the XOLAP cube will consist of all the dimensions but with a flat member list. All of these will be built using the OLAP 11g Views.

For example, lets take the Channel and Time Dimension for the XOLAP cube. Both these dimensions will have a flat hierarchy list as shown below

Once the hierarchies and measure hierarchies have been created within Essbase Studio, we need to build an Essbase Cube Schema and then deploy that as a XOLAP cube.

Open the XOLAP cube outline from within EAS. You should basically see a flat outline containing all dimension members.

It is not necessary that the XOLAP cube should have a flat structure. We just need a XOLAP cube where every dimension member in the Oracle OLAP 11g can be mapped to a level-0 member in the target XOLAP cube. In order to get the hierarchical structure for reporting, we again should build another Block Storage cube with no data but with just the hierarchical structure. I will build that here again using Essbase Studio as shown below

While building the cube we need to enable Duplicate member support as the XOLAP cube by default has that enabled. So for transparent partition to work the target Block Storage cube should also have this enabled. After enabling this deploy this block storage cube. This is the cube that will be used for reporting.

The final step in the process will be is to create the transparent partition between both the cubes. Since the member names match for all the dimensions, every time the cube is rebuilt, we can easily rebuild the partition using MaxL.

Now we can easily use the Hyperion Smart View or the classic Excel-add in to report on the OLAP data. Every time a report is run, Smart view will fire an MDX query against the Block Storage cube. Essbase will internally source it from the XOLAP cube through the transparent partition. XOLAP will then convert the MDX to SQL and generate the data. This will all be fast as direct reporting again Oracle OLAP, since Essbase will go only against pre-summarized OLAP intersections. No aggregations will be performed on Oracle OLAP unless necessary.

The same concept can be used using BI EE as well for reporting on Oracle OLAP data. Even in Oracle OLAP 11g, though the integration between BI EE and Oracle OLAP is much easier now, BI EE still does not fire the desired queries unless we model the repository using a complex multi-level assignment technique to go at the desired Oracle OLAP intersections. In the case of using XOLAP, that is not necessary.

Recently, Nigel and Mark Handford contacted me to let me know that both the BI Survey and the OLAP Report have now been brought together into a new publication called The BI Verdict, which takes their existing content and extends it with additional coverage. Having regularly read and consulted both the OLAP Report over the past ten years, and helped publicise the BI Survey, I’d thoroughly recommend the BI Verdict and urge you to take a look, particularly if you are looking to start a new project or make a tool selection.

Anyway, with all this happening, it reminded me that I’d been meaning for some time to approach Nigel for an interview for this blog. I’ve got a particular respect for Nigel as he always seems to champion the BI customer and isn’t afraid to call the various vendors on their strategy, and he’s been in the business long enough to have seen it all before and have some perspective on current developments. Anyway, here’s my questions and Nigel’s answers, and I’d be interested to hear any feedback or comments on what’s been said. Thanks again to Nigel and to Mark Handford for making this possible:

[Mark Rittman] : “There’s been a lot of innovation in the BI tools market over the past few years, plus a large amount of consolidation. What do you see as the major trends and opportunities in the BI tools market at the moment?”

[Nigel Pensde] : “Actually, I’m disappointed by the level of innovation in the BI tools market, even before the industry consolidation, which will certainly make it even worse.

Many of the claimed ‘innovations’ are actually rehashed versions of ideas that have been around for ages. For example, in-memory was the original BI architecture from more than 40 years ago (with APL), and column-oriented databases are almost as old. Dashboards, too, date back more than 25 years, to the brief EIS wave.

But there’s certainly been a lot of consolidation, which is probably bad news for customers, as they’ll be paying higher prices for products that don’t progress as much as they would otherwise have done (or which may be discontinued altogether). The new owners of the BI products are much more interested in integration between their many products, than innovation in non-core products.

The products are also likely to get more technical to install and implement, which may be good for consultants, but not for users.

I suppose the best opportunities will come for smaller, creative new vendors who take advantage of the likely mismanagement of the products acquired by large vendors. Many of the BI products now owned by the large non-BI vendors will fail to move forward or will not be promoted aggressively, thus leaving gaps in the market for nimbler vendors.”

[MR] : “There has been a lot of talk around in-memory analysis, desktop analysis and column-store analysis over the past two or three years, particularly with the release of products such as Microsoft PowerPivot and Qliktech’s Qlikview. What is your opinion of these products, how much impact will they have on the market, and are they solutions suitable for enterprise customers?”

[NP] : “Of course, QlikView is hardly a new product – it’s been around since the mid 1990s. The product sells well because it’s aggressively marketed, easy to use (at least for simple applications) and very fast. But it’s less successful for large, complex apps, and organisations that try to use it for such apps are less likely to be happy.

Of course, QlikTech will undoubtedly beef up the product’s functionality, but it has to be careful not to do this at the expense of ease of use. For example, products like Essbase and Microsoft OLAP Services (renamed to Analysis Services in 2000) started out as being aimed at business users. Now, both are far more capable and scalable, but also a lot more complex, and no business users could even think of developing new apps themselves (again, good news for consultants!).

PowerPivot is interesting. On the one hand, it’s just a way of making Excel 2010 bigger and faster, but not functionally richer. In this guise, I’m not sure how big the uptake will be, even if it’s free – after all, just how many people regularly need to download and analyse tens of millions of rows of data on their desktops? I think this may be more of a way for Microsoft to persuade its customers to upgrade to Office 2010 much earlier than they might otherwise have done, and for some of them to then use the latest versions of Microsoft server products like SharePoint and SQL Server.

But PowerPivot uses the very impressive new in-memory multidimensional VertiPaq engine, which will also feature in various Microsoft server products, such as SharePoint and Analysis Services (in fact, VertiPaq was developed by the Analysis Services team, and it will be an important new engine for Analysis Services). In this guise, it will certainly attract customers to these server products, by being easy to deploy, very scalable and extremely fast (but it currently can’t handle the more complex cubes that the older Analysis Services engines manage).”

[MR] : “Oracle have made big investments in business intelligence and enterprise performance management in the last five years. What do you think of Oracle’s strategy in the market, and how does it compare to IBM and SAP, two other large consolidators in the market? Do you see OBIEE having an impact in the market, and has the Oracle takeover of Essbase affected take-up and usage of the tool?”

[NP] : “Oracle has acquired many BI and PM products (partly as a by-product of other non-BI acquisitions, such as Siebel), rather than investing in building successful new products of its own. As part of this process, Oracle has inherited Siebel’s philosophy of providing pre-configured BI applications for its many OLTP applications.

This is probably a smart strategy, as customers who have invested many millions in shiny new transaction apps are probably not averse to spending a smaller amount to have better reporting and analysis of their OLTP data. OBIEE obviously plays a big part in this strategy. However, I don’t think OBIEE will have any impact in the larger (ie, non-Oracle) BI market.

SAP may attempt a similar strategy, but is in a weaker position to do so, as Business Objects did not bring with it a set of successful analytical applications that plugged straight into OLTP apps. And this isn’t even an option for IBM, which is not in the ERP business. To a lesser extent, Microsoft already does this with its Dynamics range.

Essbase sales into the Oracle base will certainly rise, but will probably largely disappear elsewhere as the product’s development tapers down. One irony is that Essbase had some very good dedicated OLAP front-ends when Hyperion was independent, but not for much longer. This means Essbase users will be expected to access it from OBIEE, which is not at all optimised for Essbase’s rich multidimensional structure. And this certainly won’t be an attractive option for non-Oracle sites.”

[MR] : “A recent article on The BI Verdict website mentioned a possible “OLAP Revival” following recent initiatives from the big vendors. Why do you think this is the case, and do not tools such as QlikView and PowerPivot negate the need for big investments in OLAP technologies?”

[NP] : “I didn’t write it, but it was a commentary on the increased investment being made in OLAP by all the major vendors, which remains true. For example, PowerPivot is the first deliverable from Analysis Services’ new VertiPaq engine. And one could argue that QlikView is a simple OLAP tool that fulfils exactly the same role today that Cognos PowerPlay did in the mid 1990s (ie, simple, business-oriented, in-memory analysis of multidimensional data).

IBM is also promoting TM1 much more actively than its previous owners did, and it’s to be the primary engine in Cognos Planning (which has always had its own proprietary engines until now).”

[MR] : “From your work with the BI Survey and BI Verdict, what do customers tell you are the critical success factors, and most common cause of failure, on business intelligence and OLAP projects?”

[NP] : “You probably expected me to cite the usual problems: lack of executive sponsorship and data access/quality. But, perhaps surprisingly, slow query performance is actually the most frequently-reported problem. Also, organisations that choose BI products taking into account their performance derive more business benefits than those who choose on the basis of vendor factors, like corporate standards.

This should be a salutary warning when people get over-excited about BI in the Cloud – that’s probably just about the worst way possible to deliver fast query performance.

Another key factor is involving business users in the actual implementation of BI and PM projects (ie, directly involved, not just sponsoring them). Projects that include business users in the team are notably more successful than those implemented just by IT people. And specialist BI consulting forms are more successful than either in-house IT or other types of non-specialist consultants.”

[MR] : “According to your Wikipedia entry, you’ve been working in the BI and OLAP industries since 1973. What do you think has been the most important innovation in the industry whilst you’ve been working in it, and what innovation do you think is over the horizon that will have the most impact on BI customers and developers?”

[NP] : “Not for the first time, Wikipedia is misleading on this. Back in 1973, I was a graduate trainee in an engineering company, and not using any BI tools. That came a couple of years later, and I didn’t actually join what we’d now call the BI industry until 1977.

Although I wasn’t using Express, the tools I was using back then had similar UIs (ie, a noisy teletype), and were aimed at OR specialists, rather than what we’d now think of as business users. Ironically, it means that modern tools are in some ways less sophisticated than their predecessors were 30+ years ago.

For example, FCS, the 1970s predecessor of Essbase, included Monte Carlo analysis, but Hyperion had to acquire a separate product (Crystal Ball) to get this capability into its 21st century product line. I’m also pretty sure that Express in the 1970s had statistical features not available in today’s Oracle OLAP Option and Essbase.

One thing that’s gone full circle is the architecture: in the 1970s, BI software was typically accessed using what we’d now call cloud-based, multi-tenant SaaS, using thin clients — in other words, dumb terminals connecting to a timesharing network’s remote mainframes (often in another country). You didn’t buy or rent the software, but just paid for usage. And it worked, so I don’t think anyone can claim that recent developments are ‘innovative’.

For example, I was responsible for a successful oil taxation model called Petrofisc some 30 years ago. This model was created using open source, collaborative techniques, and was widely used by oil companies planning their North Sea investments. It ran on a SaaS thin-client architecture and was even used simultaneously by competing oil companies negotiating against each other.

Of course, today’s BI software is far easier to use. It has a decent GUI (at least for the end users – techies usually still have to write code), even if it doesn’t do much more than its ancestors. Many things are now done automatically (and so they should be, given the huge increase in computer power with relatively modest performance improvements), and of course the volumes of data and numbers of users are much larger.

Getting hold of that data is now much easier, thanks to the move from very proprietary (often home-grown) transaction apps to modern ERP apps running on relational databases. Standard ETL tools and data warehouses mean that modern BI products don’t need to include such features themselves, as older products like Express once had to do.

However, many apparently promising innovations haven’t taken off, including:

• Advanced visualisation – for many years, it has been expected that innovative new ways of visualising business data would be the Next Big Thing, but most have failed to catch on. Instead, we have stupid gimmicks like animated speedometers and artistic 3D data presentations that just obscure important information. But we do finally seem to be getting some genuinely useful techniques, like spark lines, micro and bullet charts in mainstream products now. Needless to say, I made heavy use of these visualisation options in The BI Survey 8, with not a 3D chart or speedo in sight.

• Expert systems – these were all the rage in the 1980s and early 1990s, and it was hoped that they would help managers interpret the results from their BI applications, and suggest actions.

• Self-learning systems – again, this seemed a good idea 25 years ago, but it failed to take off. The idea was that BI applications would learn what metrics you spent most time analysing, what you then did next and how you liked your data presented. They would change the way they worked accordingly (much like modern automatic gearboxes adapt to your driving style). Thus, your ad hoc, interactive browsing habits would form the basis for automatic, adaptive dashboards and exception reports.

• Integrated data mining for end users – in the 1990s, BI vendors enthusiastically promoted simplified data mining for business users. But this was a big flop, as no-one but statistical specialists felt confident with such techniques.

• Integration of non-numeric information with BI – this has always seemed like an overdue idea, but it’s still not deployed widely.

• BI for mobile devices – OK, this hasn’t been a complete flop, but it turns out that BI users on the move would probably rather use a laptop than a smart phone or PDA for viewing BI reports and alerts.

• BI extranets – greedy BI vendors saw this is a brilliant way of selling even more licences to customers who already had all the licences they could possibly use for their own staff. But it seems that most companies don’t feel any great need to expose their internal BI information to outsiders. Nor is it clear who should pay for a BI extranet. So, yes, there are indeed a few BI extranets, but they are rare.

And what innovations are just over the horizon? By definition, I haven’t a clue, but the chances are that they will prove to be yet more recycled old ideas.”

Thanks Nigel, and you can find more information from Nigel (including some free analysis on trends in the BI Market) at the BI Verdict website

As such it’s not an introduction to basic Oracle OLAP via the Analytic Workspace Manager, it’s more something that could have been written in the days of Express and takes a more low-level, philosophical approach to OLAP with a particular emphasis on multi-dimensional concepts and the multi-dimensional building blocks that make up an Analytic Workspace. Whilst illustrated using recent front-end tools such as the Data Viewer within AWM, it eschews the approach taken by Oracle to make OLAP data behave as much like relational data as possible, an instead takes you straight into OLAP DML and the building blocks of a mult-dimensional database. The book does start off with instructions on how to build a cube using AWM, but even in this chapter its not long before we’re into the OLAP Worksheet (the OLAP DML front-end that’s hidden in AWM) where we’re using OLAP DML to add members to dimensions and query the metadata.

As such, it’s a brilliant book for anyone who’s used Analytic Workspace Manager and tools like Discoverer for OLAP and even OBIEE, and has wondered just how the engine within Oracle OLAP actually works. I’ve waited years for a book on OLAP DML and Jon Paredes has produced something that must surely be a labour of love, particularly as the market for Oracle OLAP books in total must be pretty small let alone one that focuses on OLAP DML, something that Oracle are in fact de-emphasizing with the move towards the OLAP API and cube organized materialized views in Oracle OLAP 11g.

One word of warning that I would give though is that with this move towards SQL and PL/SQL-based cube definitions in Oracle 11g, it’s now impossible to create OLAP objects using OLAP DML that are then registered in the OLAP metadata used increasingly by Oracle’s development and query tools, as this metadata has from 11g moved out of the analytic workspace itself and into the standard Oracle relational data dictionary tools. You can still use this manual approach though if working with OLAP data through the OLAP_TABLE function, and I know I’ll be working through John’s book over the next few months to fill out my knowledge of OLAP DML.

So, overall, a great book and it’s great to see something that’s been produced by someone for their love of the technology, and based on Oracle OLAP DML programming which is a bit of a specialist subject these days (I doubt most Oracle developers would even have heard of it). If you’re a developer using Oracle OLAP, particularly if you’ve migrated from another technology such as Holos or TM/1 and are wondering how to get under the controls hidden by Analytic Workspace Manager, or if you’re looking to use Oracle OLAP for high-end statistical or financial analysis, this is a very interesting read and I’d encourage you to get hold of a copy. If you’ve ever wondered just how to do an allocation, a forecast or create a calculation using more than two operators and been frustrated by the simplicity of Analytic Workspace Manager, this book may turn out to be an excellent investment.

Recently I completed a project for a client involving an Oracle OLAP implementation in version 10gR2 of the database. The client was developing a reporting solution to plug-in to their proprietary client-server architecture for delivering a content subscription to their customers. Reporting in their application had always been difficult for them, and they decided to deploy Oracle OLAP to ease the maintenance and increase the performance of the reports. The data model and subsequent OLAP model was simple enough… and it seemed like a real match for Oracle OLAP as I looked through the defined dimensions and measures.

One of the reports was especially problematic: the desire to pull back aggregated data according to a custom date range… sometimes that date range was for a particular set of months, other times it was from one individual day to another. To complicate the issue, the report needed to pull back these custom date ranges and display the results across the lowest level of a hierarchy in another dimension. So it wasn’t necessarily the custom date range itself that killed the reports, but the fact that this range required data to be pulled back at either the DAY or MONTH level. This coupled with needing to report at the lowest level in an additional hierarchy caused the application to pull back a substantial amount of rows, and Oracle OLAP seemed incapable of doing it in a reasonable amount of time.

In the end, I convinced the client to give version 11g a try to see if some of their performance issues would be alleviated with that release. When it was all done, the performance of 11g was “exponentially faster” to use the client’s own words. As a matter of fact, the performance gains were so dramatic that 10g and 11g seemed less like different versions and more like different products all-together.

So I wanted to take a moment to demonstrate a comparable scenario using the SH schema, and view the performance of the same query on the two different database versions: in my case, 10gR2 and 11gR2. When examining some of the timings below, please keep in mind that the Oracle database is running on a virtual machine on my Mac Book Pro, with only 1 virtual CPU and 1 GB of RAM. So it’s not the absolute runtimes that concern us, but instead the comparison timings.

First, I crank up AWM and define the CUSTOMER dimension using surrogate keys, and then define the GEOGRAPHY hierarchy containing the TOTAL, COUNTRY, STATE, CITY and CUSTOMER levels:

Then I use the Mapping Editor to map the relational source, which is the SH.CUSTOMERS table, to the appropriate levels in the CUSTOMER MARKETING hierarchy:

I go on and define the TIME and PRODUCT dimensions in a similar manner. I then build a SALES cube using all three dimensions:

To get the best possible performance in 10gR2, I’ve enabled compression, and also, to speed up my load times, I’ve logically partitioned the cube according to the MONTH level in the CALENDAR hierarchy of the TIMES dimension:

Furthermore, I use level-based pre-summarization… the only option in 10gR2. For each hierarchy in the cube, I go through the levels and determine which are to be pre-aggregated and which should be calculated on the fly. The default is to use skip-level aggregation, meaning that every other level level is pre-aggregated, which is a decent trade-off between load performance and query performance. I keep the defaults and move along:

I then map the relational sources to my two measures: QUANTITY_SOLD and AMOUNT_SOLD. These values come from the SALES table in the SH schema:

After I MAINTAIN the cube using AWM, the cube is built, populated and aggregated. I also go on to create the SQL Relational views using the OLAP View Generator plug-in. This allows me to query the SALES cube using simple SQL without having to navigate the complexity of OLAP_TABLE for every single query.

First I query the SALES cube pulling back a small number of rows at pre-aggregated levels. This is the typical use case for Oracle OLAP, and in 10gR2, performs decently. Notice that I’m pulling back CUSTOMER and PRODUCT data at the highest level in the hierarchy, while also pulling data from the TIME dimension at the MONTH level.

SQL> select *
  2    from sh_olap.sales_cubeview
  3   where time_level='MONTH'
  4     and product_level='TOTAL'
  5     and customer_level='TOTAL'
  6     and quantity_sold is not NULL;

48 rows selected.

Elapsed: 00:00:21.96

Execution Plan
----------------------------------------------------------
Plan hash value: 3653525965

------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |                |     1 | 56092 |    22   (5)| 00:00:01 |
|   1 |  VIEW                               | SALES_CUBEVIEW |     1 | 56092 |    22   (5)| 00:00:01 |
|   2 |   SQL MODEL AW HASH                 |                |     1 |     2 |            |          |
|*  3 |    COLLECTION ITERATOR PICKLER FETCH| OLAP_TABLE     |       |       |            |          |
------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter(SYS_OP_ATG(VALUE(KOKBF$),30,31,2) IS NOT NULL AND
              SYS_OP_ATG(VALUE(KOKBF$),28,29,2)='MONTH' AND SYS_OP_ATG(VALUE(KOKBF$),19,20,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),10,11,2)='TOTAL')

Statistics
----------------------------------------------------------
      12264  recursive calls
       1806  db block gets
      23749  consistent gets
       3422  physical reads
        688  redo size
       5082  bytes sent via SQL*Net to client
        392  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
        295  sorts (memory)
          0  sorts (disk)
         48  rows processed

SQL>

So we pulled 48 rows in 21.96 seconds… not bad for the hardware. But suppose I decide to pull data from CUSTOMER at the individual CUSTOMER level, returning significantly more rows from the cube. Can Oracle OLAP handle it?

SQL> select *
  2    from sh_olap.sales_cubeview
  3   where time_level='MONTH'
  4     and product_level='TOTAL'
  5     and customer_level='CUSTOMER'
  6     and quantity_sold is not NULL;

67034 rows selected.

Elapsed: 00:26:46.94

Execution Plan
----------------------------------------------------------
Plan hash value: 3653525965

------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |                |     1 | 56092 |    22   (5)| 00:00:01 |
|   1 |  VIEW                               | SALES_CUBEVIEW |     1 | 56092 |    22   (5)| 00:00:01 |
|   2 |   SQL MODEL AW HASH                 |                |     1 |     2 |            |          |
|*  3 |    COLLECTION ITERATOR PICKLER FETCH| OLAP_TABLE     |       |       |            |          |
------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter(SYS_OP_ATG(VALUE(KOKBF$),30,31,2) IS NOT NULL AND
              SYS_OP_ATG(VALUE(KOKBF$),28,29,2)='MONTH' AND SYS_OP_ATG(VALUE(KOKBF$),19,20,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),10,11,2)='CUSTOMER')

Statistics
----------------------------------------------------------
        387  recursive calls
     651249  db block gets
   14255252  consistent gets
      12707  physical reads
          0  redo size
    3988723  bytes sent via SQL*Net to client
        535  bytes received via SQL*Net from client
         15  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      67034  rows processed

SQL>

So returning 67034 rows takes us from a sub-minute query all the way to 26 minutes. Now granted… returning tens of thousands of rows is not typically in the wheelhouse of OLAP. But shouldn’t it be? When considering the investment in building and maintaining cubes, it’s a real shame if they are incapable of answering all the questions needed for targeted reporting environments.

So can 11g do any better? I start up my 11gR2 VM and build the same cube. Significant differences exist between the 10gR2 and 11gR2 versions of AWM, but for the sake of this posting, I’ll only point out one: cost-based aggregation, which is only available for compressed cubes. For a more detailed analysis of other differences, see this post and this post by Mark.

Instead of level-based aggregation as we saw above in 10g, we are now able to specify a percentage, which tells the database roughly how many possible cell values at all the different levels in the cube will be pre-calculated. What’s more… we are able to dictate different percentages between the top and bottom level partitions.

So let’s take a look at the performance we get in 11gR2. The first query is our high-level query, where we are only pulling back values from PRODUCT and CUSTOMER at the TOTAL level, and from TIME at the MONTH level. Notice that the query is slightly different: I actually have to perform a JOIN in the SQL syntax. That’s because 11gR2 manages SQL relational views automatically as part of the cube-building process, and they differ somewhat from the ones generated by the AWM plug-in in 10gR2.

SQL> select *
  2    from sh_olap.sales_view c1
  3    JOIN sh_olap.product_view p1 ON product=p1.dim_key
  4    JOIN sh_olap.customer_view c1 ON customer=c1.dim_key
  5    JOIN sh_olap.time_view t1 ON TIME=t1.dim_key
  6   where t1.level_name='MONTH'
  7     and p1.level_name='TOTAL'
  8     and c1.level_name='TOTAL'
  9     and quantity_sold is not null;

48 rows selected.

Elapsed: 00:00:00.11

Execution Plan
----------------------------------------------------------
Plan hash value: 2133067731

------------------------------------------------------------------------------
| Id  | Operation         | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |          |     1 |   100 |    29   (0)| 00:00:01 |
|   1 |  JOINED CUBE SCAN |          |       |       |            |          |
|   2 |   CUBE ACCESS     | TIME     |       |       |            |          |
|   3 |   CUBE ACCESS     | SALES    |       |       |            |          |
|   4 |   CUBE ACCESS     | CUSTOMER |       |       |            |          |
|*  5 |   CUBE ACCESS     | PRODUCT  |     1 |   100 |    29   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   5 - filter(SYS_OP_ATG(VALUE(KOKBF$),2,3,2)='MONTH' AND
              SYS_OP_ATG(VALUE(KOKBF$),8,9,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),12,13,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),5,6,2) IS NOT NULL)

Statistics
----------------------------------------------------------
         10  recursive calls
          0  db block gets
         30  consistent gets
         22  physical reads
          0  redo size
       3526  bytes sent via SQL*Net to client
        349  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
         48  rows processed

SQL>

WOW! We just pulled the same 48 rows in only 0.11 seconds. Compared with 22 seconds in 10gR2… 11gR2 is 200 times faster at the high-level query. Now, let’s take a look at the more detailed query, where we want to pull values from the CUSTOMER dimension at the CUSTOMER level:

SQL> select *
  2    from sh_olap.sales_view c1
  3    JOIN sh_olap.product_view p1 ON product=p1.dim_key
  4    JOIN sh_olap.customer_view c1 ON customer=c1.dim_key
  5    JOIN sh_olap.time_view t1 ON TIME=t1.dim_key
  6   where t1.level_name='MONTH'
  7     and p1.level_name='TOTAL'
  8     and c1.level_name='CUSTOMER'
  9     and quantity_sold is not null;

67034 rows selected.

Elapsed: 00:00:05.61

Execution Plan
----------------------------------------------------------
Plan hash value: 2133067731

------------------------------------------------------------------------------
| Id  | Operation         | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |          |     1 |   100 |    29   (0)| 00:00:01 |
|   1 |  JOINED CUBE SCAN |          |       |       |            |          |
|   2 |   CUBE ACCESS     | TIME     |       |       |            |          |
|   3 |   CUBE ACCESS     | SALES    |       |       |            |          |
|   4 |   CUBE ACCESS     | CUSTOMER |       |       |            |          |
|*  5 |   CUBE ACCESS     | PRODUCT  |     1 |   100 |    29   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   5 - filter(SYS_OP_ATG(VALUE(KOKBF$),2,3,2)='MONTH' AND
              SYS_OP_ATG(VALUE(KOKBF$),8,9,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),12,13,2)='CUSTOMER' AND
              SYS_OP_ATG(VALUE(KOKBF$),5,6,2) IS NOT NULL)

Statistics
----------------------------------------------------------
       2194  recursive calls
       1771  db block gets
       9287  consistent gets
       8183  physical reads
     103152  redo size
    3201497  bytes sent via SQL*Net to client
        492  bytes received via SQL*Net from client
         15  SQL*Net roundtrips to/from client
          9  sorts (memory)
          0  sorts (disk)
      67034  rows processed

SQL>

This query took 26 minutes and 47 seconds on 10gR2, but only 5.61 seconds on 11gR2. The math is staggering… roughly 286 times faster in the new version.

This is not just a newer, faster database (though there is plenty of that as well), but a drastic change in the way 11gR2 retrieves the data from the cube. According to A.A Hopeman from the OLAP product development team at Oracle, the term for this paradigm shift is “LOOP OPTIMIZED”:

For 11gR1 we initiated a large project called LOOP OPTIMIZED. This was a coordinated project between the OLAP Engine and the API to have the API produce meta data that lets the engine figure out how to loop as close to “sparsely” (think looping the composite) as possible. This is complex stuff. In 11gR2, the feature is more robust, handles more calc types and is just plain faster. 11gR2 also contains many other features such as improved compression to improve query performance.

David Greenfield, also from the OLAP product team at Oracle, explains what makes LOOP OPTIMIZED performance so much better:

“LOOP OPTIMIZED” improves the way that the AW loops the cube to get results. Suppose, for example, that you have a cube with three dimensions:

TIME: 10 members
PRODUCT: 20 members
CUSTOMER: 30 members

There are 6000 logical cells in this cube (10 * 20 * 30). But your fact will typically have many fewer rows, say 60 for instance. A ‘dense loop’ of the cube will literally loop all 6000 combinations looking for those 60 cells that have data. A ‘sparse loop’ of the cube will loop exactly those 60 cells.

The difference in performance between a sparse and dense loop is dramatic.

Dramatic indeed.

“Oracle Essbase & Oracle OLAP: The Guide to Oracle’s Multidimensional Solution (Osborne Oracle Press Series)” is a joint effort written by Dan Vlamis and Chris Claterbos, who covered the Oracle OLAP side, and Mike Nader, Michael Schrader, Dave Collins, Floyd Conrad and Mitch Campbell covering the Essbase angle. What the book tries to do is to cover OLAP on the Oracle platform in general, initially going through some general OLAP concepts, then going through the two product offers and how they complement each other, and then diving deeper into the architecture, cube creation process and maintenance of each of the two OLAP platforms.

I’ve read a few Essbase books recently, most notably Edward Roske and Tracey McMullen’s “Look Smarter Than You Are With Essbase 11″, and there’s a fair bit of difference between how Essbase is handled in this book and the one by Edward and Tracey. The “Look Smarter” book is a bit more of a denser read, starts off with manual building of Essbase cubes and spends a lot of time on the internals, storage types, creation of rules files and so on. This book takes a bit more of a “product managers” approach to Essbase at the start, spending quite a bit of time positioning it vis a vis Oracle OLAP and going through some deployment scenarios and case studies. The “hands-on” part starts off by covering Essbase Studio first (the “Look Smarter” book only really covers this in passing at the end) which makes it probably a better book for beginners new to Essbase. Coverage of Essbase development is pretty comprehensive, and there’s chapters on usage with OBIEE (though more the front-end, not the importation and modeling of cubes in the semantic model) and some content on Planning, HFM and so on. This is a great intro to Essbase, and particularly useful if you’re considering the technology and not quite sure how it fits in with the rest of Oracle’s Fusion Middleware and BI/DW stack.

I was particularly keen to read the Oracle OLAP chapters though, as even thought I’ve worked with the product for many years, there are many Express-era concepts such as sparsity, dimension ordering, OLAP DML and so on that I could benefit from some of Dan and Chris’ thoughts on. These concepts are covered pretty well in chapter 3 of the book, with updates for 11g, though some of the newer concepts in 11g such as cube scripts, the new expression syntax, SQL access and partitioning could do with a bit more expansion – I guess this is the downside of sharing a book with Essbase, but it does mean there’s plenty of scope for an advanced, Oracle OLAP specific follow-on book if Dan and Chris have the time. One thing that is particularly useful though is coverage of the cube organized materialized view feature new to 11g – although there was a flurry of publicity around this around the launch of 11g, I haven’t seen a huge amount of uptake and the coverage and explanation of this should hopefully bring this to more people’s attention.

As usual for Oracle Press, the book is well edited and laid out and will be a great resource for customers new to OLAP who are wondering which of Oracle’s OLAP solutions to implement. From the Oracle OLAP side, you couldn’t get two more experienced developers and consultants to write a book on it, and I certainly picked up lots of useful tips and clarifications whilst working on the review. From the Essbase side, there’s some useful positioning stuff and a good introduction to cube building using Essbase Studio, and I’d say if you bought this book plus Edward and Tracey’s one as well, you’d have Essbase development pretty well covered off. Dan tells me that the book should be ready in time for Oracle Open World in two weeks time, so if you’re in SF and can get a chance to pop into the bookstore in Moscone West, you should be able to pick up a copy “hot off the presses”.

1. Oracle Business Intelligence Enterprise Edition. 2008 was the year that OBIEE went from early-adopter technology (at least for general Oracle customers) to a mainstream BI tool. A good proportion of these implementations were accompanied by the Oracle BI Applications, showing that the message of “buy vs. build” was resonating, typically with organizations who had tried building their own “first generation” DW themselves. Of course we’re still waiting for OBIEE 11g, with Answers+ being typically the 11g component (along with the web-based template builder for BI Publisher) typically being the bits that most customers are waiting for. That said, there were two major releases in the 10g codestream in 2008, with the 10.1.3.3.3 release providing Essbase support and the 10.1.3.3.4 release providing integration with EPM Workspace, as well as the new Sample Sales application that shows off development best practices. Certainly from my perspective I’m finding that customers are now implementing Oracle BI in the form of OBIEE, rather than Business Objects or Cognos, and for previous users of Discoverer there’s a lot of customer satisfaction with this new bit of technology. 2008 was a good year for Oracle BI and it’s good to have a product that finally holds it own in the marketplace.

2. Essbase. Speaking of Essbase, who would have thought that this OLAP server would be the hot new technology in the Oracle world in 2008? My article on Essbase and OBIEE was in the top 10 tech articles for OTN last year, and everyone I know is either learning Essbase or dusting down their skills. Most customers I speak to who are looking to implement OBIEE also want to talk about EPM (Planning, Financial Consoidation etc) at the same time, and Essbase is the common factor that brings all of these together. Conference-wise, the biggest new area at ODTUG Kaleidoscope last year was the Essbase track, and look out for further Essbase content, together with crossover Essbase / OBIEE content, at next year’s Kaleidoscope. Now’s the time indeed to know your ASO from your BSO, your MDX from your SQL.

3. Oracle OLAP. Whilst Essbase has been grabbing the limelight, Oracle OLAP has finally delivered, six years on from the initial 9i releases, on the promise of integrated OLAP in the database. Cube Organized Materialized views have provided SELECT-level integration with SQL and query rewrite, whilst the new analytic expressions language in 11g looks likely to make it’s way into regular Oracle SQL as a way of simplifying analytic expressions. Just like Essbase and MDX will be must-have skills for Oracle BI implementors, Oracle OLAP, like materialized views, partitioning and compression, will be must-have skills for Oracle DW implementors. It’ll also be interesting to see whether Oracle OLAP gets parity with Essbase as a multi-dimensional data source in OBIEE 11g and Oracle BI Administrator 11g, if that’s the case (in terms of importing data sources) then it’ll be simple choice between in-database OLAP and external server OLAP when it comes to adding analytic horsepower to your OBIEE implementation.

4. Oracle Data Integrator. Whilst OBIEE and Essbase have been getting all the news, 2008 has been curiously quiet for Oracle Data Integrator. A rather low-key Data Quality add-in came along with the 10.1.3.4 release, but whilst I hear about the tool talked about by customers occaisionally, I don’t see many implementations. 2009 should be different though as the 10.1.3.5 release is the first to be certified with the Oracle BI Applications; I suspect judgement will be reserved until customers see how well it works as an alternative to Informatica in this scenario, a lot of which won’t be down to ODI but more down to how will it’s integrated with the DAC and how well (first time) the migrated Informatica mappings work.

5. Oracle Warehouse Builder. Whilst ODI has been pointed to as being the future of Oracle ETL, Oracle Warehouse Builder has quietly gone about becoming the default ETL tool (including the option of hand-coding) for Oracle shops. Ironically, of course, everyone now says they like it once they hear it’s being phased out or integrated with Oracle Data Integrator, and a few frustrations remain particularly around the Byzantine rules around control centers, locations, configurations and local/remote databases, but the 10.2.0.4 release was a solid one and most of the functionality new to 10.2 (Paris) now works well. I remember going into Oracle customer sites and having to persuade them to use OWB, now it’s the default, and there’s a good product management team who take the time to blog, get out to conferences and engage with the community. Truly a product success story.

6. Exadata and HP Oracle Database Machine. Of course the big news in 2008, at least from the Oracle product marketing machine, was Exadata and the Oracle Database Machine. Realistically it’s not something you’re going to buy for yourself, plug into your laptop and give it a whirl, but it addresses a market segment that Oracle clearly felt they were losing, and the geek inside all of us loves a new bit of hardware to go with our software. I think, like RAC and so on, we’ll reserve judgement until non-Oracle database specialists get their hands on it, but it was good to hear about databases, and data warehousing, at Open World in September and I suspect 2009 will bring similar groundbreaking news from the DW front.

7. Alternative Analytic DBMS Technologies. One side-effect of the Exadata announcement, at least from my perspective, was that it opened up the question as to whether new hardware types are required when working with very large data warehouse databases. At the same time, vendors such as Vertica, Netezza and ParAccel raised their profile through hiring ex-Oracle product development people, reaching out to the blogger community and advocating non-traditional hardware architectures. At this point it’s difficult to say whether they are just a flash in the pan (I wouldn’t want to predicate my business on one hardware innovation) or whether we’re experiencing a Cambrian explosion of new technologies and vendors, but it’s certainly raised the provide of VLDB and data warehouses and their particular hardware needs.

8. The Economy. Outside of Oracle of course, the big news was the economy. Will customers pull in their horns and cancel all non-critical IT projects, or will the need to innovate despite cuts in budgets mean that BI&DW is seen as the only way to work out what’s important and what’s not? Certainly (fingers crossed) we’ve found that demand for BI&DW skills has if anything gone up, as an industry we’re able to deliver value fast and of course we focus on ROI and business benefits. I suspect that unless we experience a kind of IT spending “nuclear winter” then demand for BI that “delivers” will in reality be stronger than ever in 2009, like all recessions it’s a time for innovation and rapid business change, and organizations with a good view of their business going forward as well as going back will be the ones that emerge into 2010 stronger than ever.

9. Rittman Mead. 2008 was a transformative year for us, with our team going from Jon and I up to six full-timers and a number of associates. We’ve focused on the quality of our process, delivery and methodology, and as always we share whatever we learn and try and put back into the community as much as possible. As a small and agile organization we think we’re well placed to handle whatever the economy throws at us in 2009, and we’re looking forward to delivering on our book, developing further our methodology and delivering business-changing Oracle BI&DW systems for our customers.

10. Looking to 2009. Technology-wise, 2009 will be the most significant year for product releases for many years. OBIEE 11g should be out at some point (the first half of 2009 looks most likely), with Answers+, a new scorecarding application, a metadata API for the semantic layer and the Action Frameworks amongst the new features. OWB and ODI will become further fused together, we may get some new Essbase point releases and of course the 11gR2 release of the database should at least be announced in 2009. To my mind, 2010 is likely to be the year of convergence, but 2009 will have enough new releases to keep us busy and with the betas out soon, we’ll be writing the “how-tos” and seminar material in time for the official releases.

So that’s it for 2008; thanks to everyone who’s read the blog, good luck for 2009, and see you all back here in January.

This is of course pretty significant, as Oracle have been more or less the only major OLAP vendor to not support MDX as a query language, at least in their in-database OLAP product (Essbase of course supports MDX for Aggregate Storage Option cubes). The rumour about Oracle extending MDX support to Oracle OLAP has been circulating the various conferences for the past year or so, I’d heard that it might be through a third-party spreadsheet add-in rather than direct support in the database and obviously this is it.

Of course this falls short of direct support for MDX, I guess with Oracle’s clear backing of SQL as their preferred OLAP query language that would have been a step too far, but this does give customers who prefer Excel as their OLAP query tool (i.e. most of them) and want to create calculations and selections in MDX the ability to choose Oracle OLAP, with all it’s clear security, scalability and managability benefits, rather than having to use a separate OLAP server. I don’t know what pricing is like, and how well the bridge performs, but it’s certainly interesting and gives Oracle OLAP 11g customers a means to analyze their cubes in a proper multi-dimensional environment, at least until Answers+ arrives.

More details at the Vlamis blog and on the Simba Technologies website.