Recently, Nigel and Mark Handford contacted me to let me know that both the BI Survey and the OLAP Report have now been brought together into a new publication called The BI Verdict, which takes their existing content and extends it with additional coverage. Having regularly read and consulted both the OLAP Report over the past ten years, and helped publicise the BI Survey, I’d thoroughly recommend the BI Verdict and urge you to take a look, particularly if you are looking to start a new project or make a tool selection.

Anyway, with all this happening, it reminded me that I’d been meaning for some time to approach Nigel for an interview for this blog. I’ve got a particular respect for Nigel as he always seems to champion the BI customer and isn’t afraid to call the various vendors on their strategy, and he’s been in the business long enough to have seen it all before and have some perspective on current developments. Anyway, here’s my questions and Nigel’s answers, and I’d be interested to hear any feedback or comments on what’s been said. Thanks again to Nigel and to Mark Handford for making this possible:

[Mark Rittman] : “There’s been a lot of innovation in the BI tools market over the past few years, plus a large amount of consolidation. What do you see as the major trends and opportunities in the BI tools market at the moment?”

[Nigel Pensde] : “Actually, I’m disappointed by the level of innovation in the BI tools market, even before the industry consolidation, which will certainly make it even worse.

Many of the claimed ‘innovations’ are actually rehashed versions of ideas that have been around for ages. For example, in-memory was the original BI architecture from more than 40 years ago (with APL), and column-oriented databases are almost as old. Dashboards, too, date back more than 25 years, to the brief EIS wave.

But there’s certainly been a lot of consolidation, which is probably bad news for customers, as they’ll be paying higher prices for products that don’t progress as much as they would otherwise have done (or which may be discontinued altogether). The new owners of the BI products are much more interested in integration between their many products, than innovation in non-core products.

The products are also likely to get more technical to install and implement, which may be good for consultants, but not for users.

I suppose the best opportunities will come for smaller, creative new vendors who take advantage of the likely mismanagement of the products acquired by large vendors. Many of the BI products now owned by the large non-BI vendors will fail to move forward or will not be promoted aggressively, thus leaving gaps in the market for nimbler vendors.”

[MR] : “There has been a lot of talk around in-memory analysis, desktop analysis and column-store analysis over the past two or three years, particularly with the release of products such as Microsoft PowerPivot and Qliktech’s Qlikview. What is your opinion of these products, how much impact will they have on the market, and are they solutions suitable for enterprise customers?”

[NP] : “Of course, QlikView is hardly a new product – it’s been around since the mid 1990s. The product sells well because it’s aggressively marketed, easy to use (at least for simple applications) and very fast. But it’s less successful for large, complex apps, and organisations that try to use it for such apps are less likely to be happy.

Of course, QlikTech will undoubtedly beef up the product’s functionality, but it has to be careful not to do this at the expense of ease of use. For example, products like Essbase and Microsoft OLAP Services (renamed to Analysis Services in 2000) started out as being aimed at business users. Now, both are far more capable and scalable, but also a lot more complex, and no business users could even think of developing new apps themselves (again, good news for consultants!).

PowerPivot is interesting. On the one hand, it’s just a way of making Excel 2010 bigger and faster, but not functionally richer. In this guise, I’m not sure how big the uptake will be, even if it’s free – after all, just how many people regularly need to download and analyse tens of millions of rows of data on their desktops? I think this may be more of a way for Microsoft to persuade its customers to upgrade to Office 2010 much earlier than they might otherwise have done, and for some of them to then use the latest versions of Microsoft server products like SharePoint and SQL Server.

But PowerPivot uses the very impressive new in-memory multidimensional VertiPaq engine, which will also feature in various Microsoft server products, such as SharePoint and Analysis Services (in fact, VertiPaq was developed by the Analysis Services team, and it will be an important new engine for Analysis Services). In this guise, it will certainly attract customers to these server products, by being easy to deploy, very scalable and extremely fast (but it currently can’t handle the more complex cubes that the older Analysis Services engines manage).”

[MR] : “Oracle have made big investments in business intelligence and enterprise performance management in the last five years. What do you think of Oracle’s strategy in the market, and how does it compare to IBM and SAP, two other large consolidators in the market? Do you see OBIEE having an impact in the market, and has the Oracle takeover of Essbase affected take-up and usage of the tool?”

[NP] : “Oracle has acquired many BI and PM products (partly as a by-product of other non-BI acquisitions, such as Siebel), rather than investing in building successful new products of its own. As part of this process, Oracle has inherited Siebel’s philosophy of providing pre-configured BI applications for its many OLTP applications.

This is probably a smart strategy, as customers who have invested many millions in shiny new transaction apps are probably not averse to spending a smaller amount to have better reporting and analysis of their OLTP data. OBIEE obviously plays a big part in this strategy. However, I don’t think OBIEE will have any impact in the larger (ie, non-Oracle) BI market.

SAP may attempt a similar strategy, but is in a weaker position to do so, as Business Objects did not bring with it a set of successful analytical applications that plugged straight into OLTP apps. And this isn’t even an option for IBM, which is not in the ERP business. To a lesser extent, Microsoft already does this with its Dynamics range.

Essbase sales into the Oracle base will certainly rise, but will probably largely disappear elsewhere as the product’s development tapers down. One irony is that Essbase had some very good dedicated OLAP front-ends when Hyperion was independent, but not for much longer. This means Essbase users will be expected to access it from OBIEE, which is not at all optimised for Essbase’s rich multidimensional structure. And this certainly won’t be an attractive option for non-Oracle sites.”

[MR] : “A recent article on The BI Verdict website mentioned a possible “OLAP Revival” following recent initiatives from the big vendors. Why do you think this is the case, and do not tools such as QlikView and PowerPivot negate the need for big investments in OLAP technologies?”

[NP] : “I didn’t write it, but it was a commentary on the increased investment being made in OLAP by all the major vendors, which remains true. For example, PowerPivot is the first deliverable from Analysis Services’ new VertiPaq engine. And one could argue that QlikView is a simple OLAP tool that fulfils exactly the same role today that Cognos PowerPlay did in the mid 1990s (ie, simple, business-oriented, in-memory analysis of multidimensional data).

IBM is also promoting TM1 much more actively than its previous owners did, and it’s to be the primary engine in Cognos Planning (which has always had its own proprietary engines until now).”

[MR] : “From your work with the BI Survey and BI Verdict, what do customers tell you are the critical success factors, and most common cause of failure, on business intelligence and OLAP projects?”

[NP] : “You probably expected me to cite the usual problems: lack of executive sponsorship and data access/quality. But, perhaps surprisingly, slow query performance is actually the most frequently-reported problem. Also, organisations that choose BI products taking into account their performance derive more business benefits than those who choose on the basis of vendor factors, like corporate standards.

This should be a salutary warning when people get over-excited about BI in the Cloud – that’s probably just about the worst way possible to deliver fast query performance.

Another key factor is involving business users in the actual implementation of BI and PM projects (ie, directly involved, not just sponsoring them). Projects that include business users in the team are notably more successful than those implemented just by IT people. And specialist BI consulting forms are more successful than either in-house IT or other types of non-specialist consultants.”

[MR] : “According to your Wikipedia entry, you’ve been working in the BI and OLAP industries since 1973. What do you think has been the most important innovation in the industry whilst you’ve been working in it, and what innovation do you think is over the horizon that will have the most impact on BI customers and developers?”

[NP] : “Not for the first time, Wikipedia is misleading on this. Back in 1973, I was a graduate trainee in an engineering company, and not using any BI tools. That came a couple of years later, and I didn’t actually join what we’d now call the BI industry until 1977.

Although I wasn’t using Express, the tools I was using back then had similar UIs (ie, a noisy teletype), and were aimed at OR specialists, rather than what we’d now think of as business users. Ironically, it means that modern tools are in some ways less sophisticated than their predecessors were 30+ years ago.

For example, FCS, the 1970s predecessor of Essbase, included Monte Carlo analysis, but Hyperion had to acquire a separate product (Crystal Ball) to get this capability into its 21st century product line. I’m also pretty sure that Express in the 1970s had statistical features not available in today’s Oracle OLAP Option and Essbase.

One thing that’s gone full circle is the architecture: in the 1970s, BI software was typically accessed using what we’d now call cloud-based, multi-tenant SaaS, using thin clients — in other words, dumb terminals connecting to a timesharing network’s remote mainframes (often in another country). You didn’t buy or rent the software, but just paid for usage. And it worked, so I don’t think anyone can claim that recent developments are ‘innovative’.

For example, I was responsible for a successful oil taxation model called Petrofisc some 30 years ago. This model was created using open source, collaborative techniques, and was widely used by oil companies planning their North Sea investments. It ran on a SaaS thin-client architecture and was even used simultaneously by competing oil companies negotiating against each other.

Of course, today’s BI software is far easier to use. It has a decent GUI (at least for the end users – techies usually still have to write code), even if it doesn’t do much more than its ancestors. Many things are now done automatically (and so they should be, given the huge increase in computer power with relatively modest performance improvements), and of course the volumes of data and numbers of users are much larger.

Getting hold of that data is now much easier, thanks to the move from very proprietary (often home-grown) transaction apps to modern ERP apps running on relational databases. Standard ETL tools and data warehouses mean that modern BI products don’t need to include such features themselves, as older products like Express once had to do.

However, many apparently promising innovations haven’t taken off, including:

• Advanced visualisation – for many years, it has been expected that innovative new ways of visualising business data would be the Next Big Thing, but most have failed to catch on. Instead, we have stupid gimmicks like animated speedometers and artistic 3D data presentations that just obscure important information. But we do finally seem to be getting some genuinely useful techniques, like spark lines, micro and bullet charts in mainstream products now. Needless to say, I made heavy use of these visualisation options in The BI Survey 8, with not a 3D chart or speedo in sight.

• Expert systems – these were all the rage in the 1980s and early 1990s, and it was hoped that they would help managers interpret the results from their BI applications, and suggest actions.

• Self-learning systems – again, this seemed a good idea 25 years ago, but it failed to take off. The idea was that BI applications would learn what metrics you spent most time analysing, what you then did next and how you liked your data presented. They would change the way they worked accordingly (much like modern automatic gearboxes adapt to your driving style). Thus, your ad hoc, interactive browsing habits would form the basis for automatic, adaptive dashboards and exception reports.

• Integrated data mining for end users – in the 1990s, BI vendors enthusiastically promoted simplified data mining for business users. But this was a big flop, as no-one but statistical specialists felt confident with such techniques.

• Integration of non-numeric information with BI – this has always seemed like an overdue idea, but it’s still not deployed widely.

• BI for mobile devices – OK, this hasn’t been a complete flop, but it turns out that BI users on the move would probably rather use a laptop than a smart phone or PDA for viewing BI reports and alerts.

• BI extranets – greedy BI vendors saw this is a brilliant way of selling even more licences to customers who already had all the licences they could possibly use for their own staff. But it seems that most companies don’t feel any great need to expose their internal BI information to outsiders. Nor is it clear who should pay for a BI extranet. So, yes, there are indeed a few BI extranets, but they are rare.

And what innovations are just over the horizon? By definition, I haven’t a clue, but the chances are that they will prove to be yet more recycled old ideas.”

Thanks Nigel, and you can find more information from Nigel (including some free analysis on trends in the BI Market) at the BI Verdict website

As such it’s not an introduction to basic Oracle OLAP via the Analytic Workspace Manager, it’s more something that could have been written in the days of Express and takes a more low-level, philosophical approach to OLAP with a particular emphasis on multi-dimensional concepts and the multi-dimensional building blocks that make up an Analytic Workspace. Whilst illustrated using recent front-end tools such as the Data Viewer within AWM, it eschews the approach taken by Oracle to make OLAP data behave as much like relational data as possible, an instead takes you straight into OLAP DML and the building blocks of a mult-dimensional database. The book does start off with instructions on how to build a cube using AWM, but even in this chapter its not long before we’re into the OLAP Worksheet (the OLAP DML front-end that’s hidden in AWM) where we’re using OLAP DML to add members to dimensions and query the metadata.

As such, it’s a brilliant book for anyone who’s used Analytic Workspace Manager and tools like Discoverer for OLAP and even OBIEE, and has wondered just how the engine within Oracle OLAP actually works. I’ve waited years for a book on OLAP DML and Jon Paredes has produced something that must surely be a labour of love, particularly as the market for Oracle OLAP books in total must be pretty small let alone one that focuses on OLAP DML, something that Oracle are in fact de-emphasizing with the move towards the OLAP API and cube organized materialized views in Oracle OLAP 11g.

One word of warning that I would give though is that with this move towards SQL and PL/SQL-based cube definitions in Oracle 11g, it’s now impossible to create OLAP objects using OLAP DML that are then registered in the OLAP metadata used increasingly by Oracle’s development and query tools, as this metadata has from 11g moved out of the analytic workspace itself and into the standard Oracle relational data dictionary tools. You can still use this manual approach though if working with OLAP data through the OLAP_TABLE function, and I know I’ll be working through John’s book over the next few months to fill out my knowledge of OLAP DML.

So, overall, a great book and it’s great to see something that’s been produced by someone for their love of the technology, and based on Oracle OLAP DML programming which is a bit of a specialist subject these days (I doubt most Oracle developers would even have heard of it). If you’re a developer using Oracle OLAP, particularly if you’ve migrated from another technology such as Holos or TM/1 and are wondering how to get under the controls hidden by Analytic Workspace Manager, or if you’re looking to use Oracle OLAP for high-end statistical or financial analysis, this is a very interesting read and I’d encourage you to get hold of a copy. If you’ve ever wondered just how to do an allocation, a forecast or create a calculation using more than two operators and been frustrated by the simplicity of Analytic Workspace Manager, this book may turn out to be an excellent investment.

Recently I completed a project for a client involving an Oracle OLAP implementation in version 10gR2 of the database. The client was developing a reporting solution to plug-in to their proprietary client-server architecture for delivering a content subscription to their customers. Reporting in their application had always been difficult for them, and they decided to deploy Oracle OLAP to ease the maintenance and increase the performance of the reports. The data model and subsequent OLAP model was simple enough… and it seemed like a real match for Oracle OLAP as I looked through the defined dimensions and measures.

One of the reports was especially problematic: the desire to pull back aggregated data according to a custom date range… sometimes that date range was for a particular set of months, other times it was from one individual day to another. To complicate the issue, the report needed to pull back these custom date ranges and display the results across the lowest level of a hierarchy in another dimension. So it wasn’t necessarily the custom date range itself that killed the reports, but the fact that this range required data to be pulled back at either the DAY or MONTH level. This coupled with needing to report at the lowest level in an additional hierarchy caused the application to pull back a substantial amount of rows, and Oracle OLAP seemed incapable of doing it in a reasonable amount of time.

In the end, I convinced the client to give version 11g a try to see if some of their performance issues would be alleviated with that release. When it was all done, the performance of 11g was “exponentially faster” to use the client’s own words. As a matter of fact, the performance gains were so dramatic that 10g and 11g seemed less like different versions and more like different products all-together.

So I wanted to take a moment to demonstrate a comparable scenario using the SH schema, and view the performance of the same query on the two different database versions: in my case, 10gR2 and 11gR2. When examining some of the timings below, please keep in mind that the Oracle database is running on a virtual machine on my Mac Book Pro, with only 1 virtual CPU and 1 GB of RAM. So it’s not the absolute runtimes that concern us, but instead the comparison timings.

First, I crank up AWM and define the CUSTOMER dimension using surrogate keys, and then define the GEOGRAPHY hierarchy containing the TOTAL, COUNTRY, STATE, CITY and CUSTOMER levels:

Then I use the Mapping Editor to map the relational source, which is the SH.CUSTOMERS table, to the appropriate levels in the CUSTOMER MARKETING hierarchy:

I go on and define the TIME and PRODUCT dimensions in a similar manner. I then build a SALES cube using all three dimensions:

To get the best possible performance in 10gR2, I’ve enabled compression, and also, to speed up my load times, I’ve logically partitioned the cube according to the MONTH level in the CALENDAR hierarchy of the TIMES dimension:

Furthermore, I use level-based pre-summarization… the only option in 10gR2. For each hierarchy in the cube, I go through the levels and determine which are to be pre-aggregated and which should be calculated on the fly. The default is to use skip-level aggregation, meaning that every other level level is pre-aggregated, which is a decent trade-off between load performance and query performance. I keep the defaults and move along:

I then map the relational sources to my two measures: QUANTITY_SOLD and AMOUNT_SOLD. These values come from the SALES table in the SH schema:

After I MAINTAIN the cube using AWM, the cube is built, populated and aggregated. I also go on to create the SQL Relational views using the OLAP View Generator plug-in. This allows me to query the SALES cube using simple SQL without having to navigate the complexity of OLAP_TABLE for every single query.

First I query the SALES cube pulling back a small number of rows at pre-aggregated levels. This is the typical use case for Oracle OLAP, and in 10gR2, performs decently. Notice that I’m pulling back CUSTOMER and PRODUCT data at the highest level in the hierarchy, while also pulling data from the TIME dimension at the MONTH level.

SQL> select *
  2    from sh_olap.sales_cubeview
  3   where time_level='MONTH'
  4     and product_level='TOTAL'
  5     and customer_level='TOTAL'
  6     and quantity_sold is not NULL;

48 rows selected.

Elapsed: 00:00:21.96

Execution Plan
----------------------------------------------------------
Plan hash value: 3653525965

------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |                |     1 | 56092 |    22   (5)| 00:00:01 |
|   1 |  VIEW                               | SALES_CUBEVIEW |     1 | 56092 |    22   (5)| 00:00:01 |
|   2 |   SQL MODEL AW HASH                 |                |     1 |     2 |            |          |
|*  3 |    COLLECTION ITERATOR PICKLER FETCH| OLAP_TABLE     |       |       |            |          |
------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter(SYS_OP_ATG(VALUE(KOKBF$),30,31,2) IS NOT NULL AND
              SYS_OP_ATG(VALUE(KOKBF$),28,29,2)='MONTH' AND SYS_OP_ATG(VALUE(KOKBF$),19,20,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),10,11,2)='TOTAL')

Statistics
----------------------------------------------------------
      12264  recursive calls
       1806  db block gets
      23749  consistent gets
       3422  physical reads
        688  redo size
       5082  bytes sent via SQL*Net to client
        392  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
        295  sorts (memory)
          0  sorts (disk)
         48  rows processed

SQL>

So we pulled 48 rows in 21.96 seconds… not bad for the hardware. But suppose I decide to pull data from CUSTOMER at the individual CUSTOMER level, returning significantly more rows from the cube. Can Oracle OLAP handle it?

SQL> select *
  2    from sh_olap.sales_cubeview
  3   where time_level='MONTH'
  4     and product_level='TOTAL'
  5     and customer_level='CUSTOMER'
  6     and quantity_sold is not NULL;

67034 rows selected.

Elapsed: 00:26:46.94

Execution Plan
----------------------------------------------------------
Plan hash value: 3653525965

------------------------------------------------------------------------------------------------------
| Id  | Operation                           | Name           | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT                    |                |     1 | 56092 |    22   (5)| 00:00:01 |
|   1 |  VIEW                               | SALES_CUBEVIEW |     1 | 56092 |    22   (5)| 00:00:01 |
|   2 |   SQL MODEL AW HASH                 |                |     1 |     2 |            |          |
|*  3 |    COLLECTION ITERATOR PICKLER FETCH| OLAP_TABLE     |       |       |            |          |
------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   3 - filter(SYS_OP_ATG(VALUE(KOKBF$),30,31,2) IS NOT NULL AND
              SYS_OP_ATG(VALUE(KOKBF$),28,29,2)='MONTH' AND SYS_OP_ATG(VALUE(KOKBF$),19,20,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),10,11,2)='CUSTOMER')

Statistics
----------------------------------------------------------
        387  recursive calls
     651249  db block gets
   14255252  consistent gets
      12707  physical reads
          0  redo size
    3988723  bytes sent via SQL*Net to client
        535  bytes received via SQL*Net from client
         15  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
      67034  rows processed

SQL>

So returning 67034 rows takes us from a sub-minute query all the way to 26 minutes. Now granted… returning tens of thousands of rows is not typically in the wheelhouse of OLAP. But shouldn’t it be? When considering the investment in building and maintaining cubes, it’s a real shame if they are incapable of answering all the questions needed for targeted reporting environments.

So can 11g do any better? I start up my 11gR2 VM and build the same cube. Significant differences exist between the 10gR2 and 11gR2 versions of AWM, but for the sake of this posting, I’ll only point out one: cost-based aggregation, which is only available for compressed cubes. For a more detailed analysis of other differences, see this post and this post by Mark.

Instead of level-based aggregation as we saw above in 10g, we are now able to specify a percentage, which tells the database roughly how many possible cell values at all the different levels in the cube will be pre-calculated. What’s more… we are able to dictate different percentages between the top and bottom level partitions.

So let’s take a look at the performance we get in 11gR2. The first query is our high-level query, where we are only pulling back values from PRODUCT and CUSTOMER at the TOTAL level, and from TIME at the MONTH level. Notice that the query is slightly different: I actually have to perform a JOIN in the SQL syntax. That’s because 11gR2 manages SQL relational views automatically as part of the cube-building process, and they differ somewhat from the ones generated by the AWM plug-in in 10gR2.

SQL> select *
  2    from sh_olap.sales_view c1
  3    JOIN sh_olap.product_view p1 ON product=p1.dim_key
  4    JOIN sh_olap.customer_view c1 ON customer=c1.dim_key
  5    JOIN sh_olap.time_view t1 ON TIME=t1.dim_key
  6   where t1.level_name='MONTH'
  7     and p1.level_name='TOTAL'
  8     and c1.level_name='TOTAL'
  9     and quantity_sold is not null;

48 rows selected.

Elapsed: 00:00:00.11

Execution Plan
----------------------------------------------------------
Plan hash value: 2133067731

------------------------------------------------------------------------------
| Id  | Operation         | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |          |     1 |   100 |    29   (0)| 00:00:01 |
|   1 |  JOINED CUBE SCAN |          |       |       |            |          |
|   2 |   CUBE ACCESS     | TIME     |       |       |            |          |
|   3 |   CUBE ACCESS     | SALES    |       |       |            |          |
|   4 |   CUBE ACCESS     | CUSTOMER |       |       |            |          |
|*  5 |   CUBE ACCESS     | PRODUCT  |     1 |   100 |    29   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   5 - filter(SYS_OP_ATG(VALUE(KOKBF$),2,3,2)='MONTH' AND
              SYS_OP_ATG(VALUE(KOKBF$),8,9,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),12,13,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),5,6,2) IS NOT NULL)

Statistics
----------------------------------------------------------
         10  recursive calls
          0  db block gets
         30  consistent gets
         22  physical reads
          0  redo size
       3526  bytes sent via SQL*Net to client
        349  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          1  sorts (memory)
          0  sorts (disk)
         48  rows processed

SQL>

WOW! We just pulled the same 48 rows in only 0.11 seconds. Compared with 22 seconds in 10gR2… 11gR2 is 200 times faster at the high-level query. Now, let’s take a look at the more detailed query, where we want to pull values from the CUSTOMER dimension at the CUSTOMER level:

SQL> select *
  2    from sh_olap.sales_view c1
  3    JOIN sh_olap.product_view p1 ON product=p1.dim_key
  4    JOIN sh_olap.customer_view c1 ON customer=c1.dim_key
  5    JOIN sh_olap.time_view t1 ON TIME=t1.dim_key
  6   where t1.level_name='MONTH'
  7     and p1.level_name='TOTAL'
  8     and c1.level_name='CUSTOMER'
  9     and quantity_sold is not null;

67034 rows selected.

Elapsed: 00:00:05.61

Execution Plan
----------------------------------------------------------
Plan hash value: 2133067731

------------------------------------------------------------------------------
| Id  | Operation         | Name     | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------
|   0 | SELECT STATEMENT  |          |     1 |   100 |    29   (0)| 00:00:01 |
|   1 |  JOINED CUBE SCAN |          |       |       |            |          |
|   2 |   CUBE ACCESS     | TIME     |       |       |            |          |
|   3 |   CUBE ACCESS     | SALES    |       |       |            |          |
|   4 |   CUBE ACCESS     | CUSTOMER |       |       |            |          |
|*  5 |   CUBE ACCESS     | PRODUCT  |     1 |   100 |    29   (0)| 00:00:01 |
------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   5 - filter(SYS_OP_ATG(VALUE(KOKBF$),2,3,2)='MONTH' AND
              SYS_OP_ATG(VALUE(KOKBF$),8,9,2)='TOTAL' AND
              SYS_OP_ATG(VALUE(KOKBF$),12,13,2)='CUSTOMER' AND
              SYS_OP_ATG(VALUE(KOKBF$),5,6,2) IS NOT NULL)

Statistics
----------------------------------------------------------
       2194  recursive calls
       1771  db block gets
       9287  consistent gets
       8183  physical reads
     103152  redo size
    3201497  bytes sent via SQL*Net to client
        492  bytes received via SQL*Net from client
         15  SQL*Net roundtrips to/from client
          9  sorts (memory)
          0  sorts (disk)
      67034  rows processed

SQL>

This query took 26 minutes and 47 seconds on 10gR2, but only 5.61 seconds on 11gR2. The math is staggering… roughly 286 times faster in the new version.

This is not just a newer, faster database (though there is plenty of that as well), but a drastic change in the way 11gR2 retrieves the data from the cube. According to A.A Hopeman from the OLAP product development team at Oracle, the term for this paradigm shift is “LOOP OPTIMIZED”:

For 11gR1 we initiated a large project called LOOP OPTIMIZED. This was a coordinated project between the OLAP Engine and the API to have the API produce meta data that lets the engine figure out how to loop as close to “sparsely” (think looping the composite) as possible. This is complex stuff. In 11gR2, the feature is more robust, handles more calc types and is just plain faster. 11gR2 also contains many other features such as improved compression to improve query performance.

David Greenfield, also from the OLAP product team at Oracle, explains what makes LOOP OPTIMIZED performance so much better:

“LOOP OPTIMIZED” improves the way that the AW loops the cube to get results. Suppose, for example, that you have a cube with three dimensions:

TIME: 10 members
PRODUCT: 20 members
CUSTOMER: 30 members

There are 6000 logical cells in this cube (10 * 20 * 30). But your fact will typically have many fewer rows, say 60 for instance. A ‘dense loop’ of the cube will literally loop all 6000 combinations looking for those 60 cells that have data. A ’sparse loop’ of the cube will loop exactly those 60 cells.

The difference in performance between a sparse and dense loop is dramatic.

Dramatic indeed.

“Oracle Essbase & Oracle OLAP: The Guide to Oracle’s Multidimensional Solution (Osborne Oracle Press Series)” is a joint effort written by Dan Vlamis and Chris Claterbos, who covered the Oracle OLAP side, and Mike Nader, Michael Schrader, Dave Collins, Floyd Conrad and Mitch Campbell covering the Essbase angle. What the book tries to do is to cover OLAP on the Oracle platform in general, initially going through some general OLAP concepts, then going through the two product offers and how they complement each other, and then diving deeper into the architecture, cube creation process and maintenance of each of the two OLAP platforms.

I’ve read a few Essbase books recently, most notably Edward Roske and Tracey McMullen’s “Look Smarter Than You Are With Essbase 11″, and there’s a fair bit of difference between how Essbase is handled in this book and the one by Edward and Tracey. The “Look Smarter” book is a bit more of a denser read, starts off with manual building of Essbase cubes and spends a lot of time on the internals, storage types, creation of rules files and so on. This book takes a bit more of a “product managers” approach to Essbase at the start, spending quite a bit of time positioning it vis a vis Oracle OLAP and going through some deployment scenarios and case studies. The “hands-on” part starts off by covering Essbase Studio first (the “Look Smarter” book only really covers this in passing at the end) which makes it probably a better book for beginners new to Essbase. Coverage of Essbase development is pretty comprehensive, and there’s chapters on usage with OBIEE (though more the front-end, not the importation and modeling of cubes in the semantic model) and some content on Planning, HFM and so on. This is a great intro to Essbase, and particularly useful if you’re considering the technology and not quite sure how it fits in with the rest of Oracle’s Fusion Middleware and BI/DW stack.

I was particularly keen to read the Oracle OLAP chapters though, as even thought I’ve worked with the product for many years, there are many Express-era concepts such as sparsity, dimension ordering, OLAP DML and so on that I could benefit from some of Dan and Chris’ thoughts on. These concepts are covered pretty well in chapter 3 of the book, with updates for 11g, though some of the newer concepts in 11g such as cube scripts, the new expression syntax, SQL access and partitioning could do with a bit more expansion – I guess this is the downside of sharing a book with Essbase, but it does mean there’s plenty of scope for an advanced, Oracle OLAP specific follow-on book if Dan and Chris have the time. One thing that is particularly useful though is coverage of the cube organized materialized view feature new to 11g – although there was a flurry of publicity around this around the launch of 11g, I haven’t seen a huge amount of uptake and the coverage and explanation of this should hopefully bring this to more people’s attention.

As usual for Oracle Press, the book is well edited and laid out and will be a great resource for customers new to OLAP who are wondering which of Oracle’s OLAP solutions to implement. From the Oracle OLAP side, you couldn’t get two more experienced developers and consultants to write a book on it, and I certainly picked up lots of useful tips and clarifications whilst working on the review. From the Essbase side, there’s some useful positioning stuff and a good introduction to cube building using Essbase Studio, and I’d say if you bought this book plus Edward and Tracey’s one as well, you’d have Essbase development pretty well covered off. Dan tells me that the book should be ready in time for Oracle Open World in two weeks time, so if you’re in SF and can get a chance to pop into the bookstore in Moscone West, you should be able to pick up a copy “hot off the presses”.

1. Oracle Business Intelligence Enterprise Edition. 2008 was the year that OBIEE went from early-adopter technology (at least for general Oracle customers) to a mainstream BI tool. A good proportion of these implementations were accompanied by the Oracle BI Applications, showing that the message of “buy vs. build” was resonating, typically with organizations who had tried building their own “first generation” DW themselves. Of course we’re still waiting for OBIEE 11g, with Answers+ being typically the 11g component (along with the web-based template builder for BI Publisher) typically being the bits that most customers are waiting for. That said, there were two major releases in the 10g codestream in 2008, with the 10.1.3.3.3 release providing Essbase support and the 10.1.3.3.4 release providing integration with EPM Workspace, as well as the new Sample Sales application that shows off development best practices. Certainly from my perspective I’m finding that customers are now implementing Oracle BI in the form of OBIEE, rather than Business Objects or Cognos, and for previous users of Discoverer there’s a lot of customer satisfaction with this new bit of technology. 2008 was a good year for Oracle BI and it’s good to have a product that finally holds it own in the marketplace.

2. Essbase. Speaking of Essbase, who would have thought that this OLAP server would be the hot new technology in the Oracle world in 2008? My article on Essbase and OBIEE was in the top 10 tech articles for OTN last year, and everyone I know is either learning Essbase or dusting down their skills. Most customers I speak to who are looking to implement OBIEE also want to talk about EPM (Planning, Financial Consoidation etc) at the same time, and Essbase is the common factor that brings all of these together. Conference-wise, the biggest new area at ODTUG Kaleidoscope last year was the Essbase track, and look out for further Essbase content, together with crossover Essbase / OBIEE content, at next year’s Kaleidoscope. Now’s the time indeed to know your ASO from your BSO, your MDX from your SQL.

3. Oracle OLAP. Whilst Essbase has been grabbing the limelight, Oracle OLAP has finally delivered, six years on from the initial 9i releases, on the promise of integrated OLAP in the database. Cube Organized Materialized views have provided SELECT-level integration with SQL and query rewrite, whilst the new analytic expressions language in 11g looks likely to make it’s way into regular Oracle SQL as a way of simplifying analytic expressions. Just like Essbase and MDX will be must-have skills for Oracle BI implementors, Oracle OLAP, like materialized views, partitioning and compression, will be must-have skills for Oracle DW implementors. It’ll also be interesting to see whether Oracle OLAP gets parity with Essbase as a multi-dimensional data source in OBIEE 11g and Oracle BI Administrator 11g, if that’s the case (in terms of importing data sources) then it’ll be simple choice between in-database OLAP and external server OLAP when it comes to adding analytic horsepower to your OBIEE implementation.

4. Oracle Data Integrator. Whilst OBIEE and Essbase have been getting all the news, 2008 has been curiously quiet for Oracle Data Integrator. A rather low-key Data Quality add-in came along with the 10.1.3.4 release, but whilst I hear about the tool talked about by customers occaisionally, I don’t see many implementations. 2009 should be different though as the 10.1.3.5 release is the first to be certified with the Oracle BI Applications; I suspect judgement will be reserved until customers see how well it works as an alternative to Informatica in this scenario, a lot of which won’t be down to ODI but more down to how will it’s integrated with the DAC and how well (first time) the migrated Informatica mappings work.

5. Oracle Warehouse Builder. Whilst ODI has been pointed to as being the future of Oracle ETL, Oracle Warehouse Builder has quietly gone about becoming the default ETL tool (including the option of hand-coding) for Oracle shops. Ironically, of course, everyone now says they like it once they hear it’s being phased out or integrated with Oracle Data Integrator, and a few frustrations remain particularly around the Byzantine rules around control centers, locations, configurations and local/remote databases, but the 10.2.0.4 release was a solid one and most of the functionality new to 10.2 (Paris) now works well. I remember going into Oracle customer sites and having to persuade them to use OWB, now it’s the default, and there’s a good product management team who take the time to blog, get out to conferences and engage with the community. Truly a product success story.

6. Exadata and HP Oracle Database Machine. Of course the big news in 2008, at least from the Oracle product marketing machine, was Exadata and the Oracle Database Machine. Realistically it’s not something you’re going to buy for yourself, plug into your laptop and give it a whirl, but it addresses a market segment that Oracle clearly felt they were losing, and the geek inside all of us loves a new bit of hardware to go with our software. I think, like RAC and so on, we’ll reserve judgement until non-Oracle database specialists get their hands on it, but it was good to hear about databases, and data warehousing, at Open World in September and I suspect 2009 will bring similar groundbreaking news from the DW front.

7. Alternative Analytic DBMS Technologies. One side-effect of the Exadata announcement, at least from my perspective, was that it opened up the question as to whether new hardware types are required when working with very large data warehouse databases. At the same time, vendors such as Vertica, Netezza and ParAccel raised their profile through hiring ex-Oracle product development people, reaching out to the blogger community and advocating non-traditional hardware architectures. At this point it’s difficult to say whether they are just a flash in the pan (I wouldn’t want to predicate my business on one hardware innovation) or whether we’re experiencing a Cambrian explosion of new technologies and vendors, but it’s certainly raised the provide of VLDB and data warehouses and their particular hardware needs.

8. The Economy. Outside of Oracle of course, the big news was the economy. Will customers pull in their horns and cancel all non-critical IT projects, or will the need to innovate despite cuts in budgets mean that BI&DW is seen as the only way to work out what’s important and what’s not? Certainly (fingers crossed) we’ve found that demand for BI&DW skills has if anything gone up, as an industry we’re able to deliver value fast and of course we focus on ROI and business benefits. I suspect that unless we experience a kind of IT spending “nuclear winter” then demand for BI that “delivers” will in reality be stronger than ever in 2009, like all recessions it’s a time for innovation and rapid business change, and organizations with a good view of their business going forward as well as going back will be the ones that emerge into 2010 stronger than ever.

9. Rittman Mead. 2008 was a transformative year for us, with our team going from Jon and I up to six full-timers and a number of associates. We’ve focused on the quality of our process, delivery and methodology, and as always we share whatever we learn and try and put back into the community as much as possible. As a small and agile organization we think we’re well placed to handle whatever the economy throws at us in 2009, and we’re looking forward to delivering on our book, developing further our methodology and delivering business-changing Oracle BI&DW systems for our customers.

10. Looking to 2009. Technology-wise, 2009 will be the most significant year for product releases for many years. OBIEE 11g should be out at some point (the first half of 2009 looks most likely), with Answers+, a new scorecarding application, a metadata API for the semantic layer and the Action Frameworks amongst the new features. OWB and ODI will become further fused together, we may get some new Essbase point releases and of course the 11gR2 release of the database should at least be announced in 2009. To my mind, 2010 is likely to be the year of convergence, but 2009 will have enough new releases to keep us busy and with the betas out soon, we’ll be writing the “how-tos” and seminar material in time for the official releases.

So that’s it for 2008; thanks to everyone who’s read the blog, good luck for 2009, and see you all back here in January.

This is of course pretty significant, as Oracle have been more or less the only major OLAP vendor to not support MDX as a query language, at least in their in-database OLAP product (Essbase of course supports MDX for Aggregate Storage Option cubes). The rumour about Oracle extending MDX support to Oracle OLAP has been circulating the various conferences for the past year or so, I’d heard that it might be through a third-party spreadsheet add-in rather than direct support in the database and obviously this is it.

Of course this falls short of direct support for MDX, I guess with Oracle’s clear backing of SQL as their preferred OLAP query language that would have been a step too far, but this does give customers who prefer Excel as their OLAP query tool (i.e. most of them) and want to create calculations and selections in MDX the ability to choose Oracle OLAP, with all it’s clear security, scalability and managability benefits, rather than having to use a separate OLAP server. I don’t know what pricing is like, and how well the bridge performs, but it’s certainly interesting and gives Oracle OLAP 11g customers a means to analyze their cubes in a proper multi-dimensional environment, at least until Answers+ arrives.

More details at the Vlamis blog and on the Simba Technologies website.

Well without revealing the whole paper just now; I am reserving that for the early morning (08:30) crowd (???) I get in Denver on April 17! I can say that Mark is right that I am enthusiastic about the technology.

Ignoring the fact that one of my first Oracle data warehouse systems used Oracle Sales Analyzer (and hence, Express) as a front-end, so I already knew quite a bit about Oracle OLAP; why should a person who has spent many years working in summary management and BI reporting be interested in OLAP cubes?  The answer to this lies in query rewrite.

One of the big problems in traditional data warehouse + reporting software configurations is getting good query performance; traditionally, you build a load of summary tables in the batch (once) that answer many user queries and hence remove the need to repeatedly make  the same expensive joins and aggregations at query time. Before query rewrite came on the scene each summary table needed to be mapped into the reporting tool , if you were lucky the query tool was aggregate aware and would choose the best summary table for the query, if not the users had to be instructed on the aggregate to use. Query rewrite changed that, we just needed to map in the base fact table and the denormalized dimensions and let the optimizer work out the best summary materialized view to use.

That still left the problem of being smart about the summaries to build – the wrong ones (or too few) and query performance is not as good as you could get, too many and it takes too long work out which summary to use! Step up COMV; build a simple OLAP cube, just the fact measures (no flashy OLAP calculations) and the dimensions, expose it as a query rewrite enabled  materialized view and away you go, one summary to cover virtually all ad-hoc queries.

Next time I will write a bit about COMV performance.

Anyway, take a look if you get a chance, take a look also at the OLAP Option pages on the Oracle Wiki, and if you’re going to Collaborate’08 don’t forget that Peter Scott is doing a conference session on the Thursday on this very subject.

I could run out the argument that I work in a BI consultancy and none of us here at Rittman Mead are 100% aligned to a specific expertise area; for example I do OBIEE and OWB as well as data warehouse design and performance. And besides, in my past I did look after one of the largest Oracle Sales Analyzer systems in Europe (ROLAP reporting on a couple terabytes of relational data), so there is quite a bit of Oracle Express Server (and OLAP DML) lurking in my past. But in reality my talk is going to look at the use of OLAP Cube Organized Materialized Views in the context of summary management and that, of course, is right at the heart of what I do!

Summary management has always been one of those ‘art form’ areas of DW/BI – which summary tables to build and how many of them are needed to balance performance, maintenance time and space usage. Query rewrite was great step forward in Oracle 9.2 (OK, it was there before, but I tended not to use it) at last, I only needed to map a single fact table (the base table) into the query tool, and for some third-party tools that was a significant thing. And now with Oracle 11g, the ability to transparently rewrite a query against the base fact table to a OLAP cube is a significant advance, it means that I needn’t think about the materialized views I need create since the cube effectively contains them all! This is a bit of a simplification of course and I will say a lot more in the 50 minutes or so of speaking slot – so if you are Collaborate 08 – and that is just a month away, you could get to hear the whole talk.

I was taking a look therefore at the latest Oracle OLAP newsletter when I saw a mention of what’s planned for BI Beans in the 10g release and what’s coming up in the 11g timeframe. To quote the newsletter:

“Oracle BI Beans 10.1.2.x is planned to be certified to work with Oracle OLAP 11g. BI Beans are being repackaged as ADF Data Visualization components (ADF DVT). These components are available in JDeveloper 11g and include Graph, Gauge, Geographic Map, Gantt Chart, and Pivot Table. DVT components work against pure relational as well as OBI EE data sources using standard ADF Data Control mechanism. In the first release of 11g direct access to OLAP cubes from the DVT components will be via SQL through the ADF BC layer or directly via Java code.”

My guess here is that BI Beans 10.1.2.x will be certified with Oracle OLAP 11g in 10g OLAP mode, as the main access method for the 11g OLAP mode is through SQL. Looking at the statement, this would follow as in their current form, DVT components use an SQL query to access their data, although the door does seem to be open for some further OLAP access optimization in a future release, whether this will be through the Java OLAP API (is this supported in 11g OLAP mode?) or some other method is obviously yet to be determined.

Taking a look at Data Visualization Components, they look pretty neat, and the graphing components seem to be a derivation of the code that was used to deliver graphs in the 10gR2 version of Discoverer, which of course had to support both OLAP API and SQL access.

Looking at the graphing options available, there’s quite a few more than are in the 10g version of Discoverer though, with funnel and gantt charts now available as well.

Something that’s new with DVT is a map component, that lets you take data from an SQL query and render it in a map. I wonder if it’s support maps other than for the USA?

There’s also what appears to be a thin-client, interactive pivot table (crosstab) component – one of the drawbacks with BI Beans was that the interactive, drag-and-drop crosstab was thick-client, Java applet only, whilst the thin-client version only allowed manipulation via drop-down menus and page reloads. From screenshots I’ve seen of Answers Plus (the next, OLAP analysis-capable version of Oracle BI Answers) the crosstabs it uses seem to use this style of pivot table, so I guess DVT is the technology they’re building this on – just a guess though.

Here’s the drag-and-drop ability, pivoting one dimension with another.

Looking through the Oracle-by-Example linked to earlier, what does seem to be missing is any sort of query builder you can put into your application; one of the major plus points of BI Beans was the OLAP query builder you could add to a project, giving you the same ability to build queries that you see in Oracle BI Discoverer for OLAP. I couldn’t see anything beyond an SQL expression builder in DVT, obviously in DVT we’d lose any OLAP elements of a query builder (select dimension members via a hierarchy, that sort of thing) but there doesn’t appear to be anything in DVT beyond a simple editor for an SQL query. It looks like DVT is something you use for adding data vizualizations to your application rather than building your own answer to Discoverer.

Anyway, I guess this answers the question about what’s happening with BI Beans. The existing 10g version will be certified to work against 11g OLAP, my guess is in 10g OLAP mode. For 11g OLAP access through SQL queries (or through the Cube Materialized View feature) and SQL queries in general, these new Data Visualization Components seem to be a spruced-up, thin-client version of BI Beans, albeit without an OLAP query builder or their own slightly abstracted Java API into Oracle OLAP, though as I said the statement on the Oracle OLAP newsletter does seem to suggest that more OLAP functionality might come in a subsequent release.