At Rittman Mead we tend to look a Business Intelligence through a series of levels of maturity, these typically go from being able to produce operational reports from a single system with no control and governance, to building an analytic system where data is integrated, dimensions are conformed, metrics are clearly defined, control and governance is applied to the data and power users can do sophisticated analysis of the data and make use of features such as drilling down, drilling across, aggregation and filtering. Reports built in this system could then be arranged on dashboards to provide user end user access to the information.

Users get value by being able to compare data from different business processes, being able to aggregate data and being able to look at data over time. There can be a number of steps between the basic operational reporting and this level of analytic reporting, but this level basically describes the core features of OBIEE’s Answers and Dashboards products.

We find most organisation are at a more basic level of maturity. In order to get traction from Business Intelligence projects, we have to engage at that organisation’s existing level of reporting maturity, there is no point proposing some wonderful high brow analytics system if the organisation is still doing most of its reporting using spreadsheets, I don’t believe you will get traction. You have to grow the maturity of the organisation with the development of the Business Intelligence landscape.

So this leads me to my original question of where the what if analysis has gone, how should organisations move to this? From a product point of view there are some obvious candidates: Oracle Data Mining (ODM), Essbase, Oracle OLAP and Crystal Ball. The questions are: what is the use case for implementing these? Where can an organisation get business value using this type of analysis? How could you construct a ROI case for these?

A meeting with a colleague identified the following scenario which I quite liked and thought made a clear and strong case. The starting point is at a level of maturity where an organisation had a good analytic system. The use case is the organisation wants to change their existing business model to either target another customer segment, or add another value proposition or product.

The most basic way of doing this would be to create a simple spreadsheet that read a bit like a Profit & Loss statement and had a figure for expected revenue and some figures for costs, these estimates would be based on the experience of the spreadsheet author and would at most contain worst case, expected and best case values. They could potentially turn out to be accurate, however they are not really going to carry any weight, especially if significant investment is going to be required to implement this change. What is required is a more qualified approach to modelling this change.

Using the products mentioned above one approach to solving this problem could be to start with ODM. The organisation could use some of the algorithms in ODM to start to predict measures like revenue, based on customers likelihood to churn and sensitivity to price. These calculated measures would be based on historic data the organisation had and would give a much more reliable view or model of the revenue for the new or revised business model.

However so far we have only got one version of the model and it may not feature all the measures we need, for example it may be possible to calculate some revenue figures, however cost may need to be manually input.The next step would be to use Essbase. If the model comprising of the associated dimensions and facts was then loaded to Essbase, along with the ODM calculated data the organisation could then publish this to several users and allow them to input data for measures that couldn’t be calculated and to create different scenarios for the model, based on different interpretations of how successful the model is, for example worst case, best case and most likely. In additional profit would be able to be determined for these scenarios as all the relevant measures would be present – either calculated by the analytics tool, ODM or user input.

Coming back to the question of investment, whether it was either internal or external, the organisation would now have a much more compelling case to persuade a potential investor. However I think it is possible to argue that there is still a missing element from this which is risk. The investor would be well within their rights to ask what risk was associated with their investment. This is where the final piece of the jigsaw comes in, Crystal Ball.

Crystal Ball allows users to run a series of (Monte Carlo) simulations based on high and low water mark parameters that are run against the model, the end result is that a level of confidence can be ascertained about the likelihood of an event. Crystal Ball works as an Excel plug-in, and as such can also work in conjunction with Essbase. The Essbase data can be loaded into Excel through its plug-in, then have the statistical analysis from Crystal Ball performed on it, then potentially read back into Essbase as another scenario.

The end result would be a business model both in real terms and loaded into Essbase that an organisation could go to an investor with understanding of revenue, profit and level of confidence (risk) and be in a much stronger position to secure the finance they need. As a further benefit the organisation may be able to negotiate a significantly lower interest rate on any investment as they can demonstrate low risk.

I think this story sets out a clear picture of how these technologies can be used together to support a genuine business scenario and it is certainly something I will be investigating internally with some business initiatives we have coming up over the remainder of this year.

1. There are quite a few talented good people here who have excellent knowledge about the end-to-end Oracle BI stack. In fact the 2 people whom we have hired now ,Ram Chaitanya and Jay Gandhi, are ex-Oracle like myself. Part of Oracle BI development happens locally here in India, and we thought it was ideal to hire talent with good knowledge whenever there is a good opportunity.

2. We have been getting some work recently in areas that are outside of Europe and US, like training/consulting in India, South Africa, Singapore etc. To cater to such requirements, we thought it was ideal to have an office in India which could potentially be a hub for us to involve consultants on projects that originate outside of Europe and US. Also, we have been recently involved in a big Exadata & BI EE implementation for a retail client. We had people involved from all three of our offices( like Pete, Paul, Stewart & myself), and we found that it always helps to have people spread around the world covering the entire 24 hour spectrum (with potential overlap) even in a full life-cycle implementation.

3. I have always wanted to involve myself in creating custom utilities/solutions for Oracle BI EE & EPM suites. Everyone in our company has an idea for what would be ideal for a custom utility that can help in faster deployment of Oracle BI, and we thought therefore that our India hub would be ideal to deliver such utilities.

Also whilst there is a bit of a cost advantage to having operations in India, good BI developers/architects always come at a premium irrespective of where they are based and we are therefore not really aiming at the offshore/outsourcing market with this venture. Our plan instead is to hire very good people in limited numbers and aim at the high-quality/expert end of the Oracle BI market with typically ex-Oracle staff within our team, working on our own projects as well as working with our operations in the UK and the USA.

Now that we have an office in India and if you are interested in joining us, do drop us a line at venkat@rittmanmead.com, and we will be glad to have technical discussions with you. If you are also an Oracle BI, DW or EPM customer looking for help on projects in the Asia-Pacific region, again we’ll be pleased to hear from you and to let you know how we can help.

Recently, Nigel and Mark Handford contacted me to let me know that both the BI Survey and the OLAP Report have now been brought together into a new publication called The BI Verdict, which takes their existing content and extends it with additional coverage. Having regularly read and consulted both the OLAP Report over the past ten years, and helped publicise the BI Survey, I’d thoroughly recommend the BI Verdict and urge you to take a look, particularly if you are looking to start a new project or make a tool selection.

Anyway, with all this happening, it reminded me that I’d been meaning for some time to approach Nigel for an interview for this blog. I’ve got a particular respect for Nigel as he always seems to champion the BI customer and isn’t afraid to call the various vendors on their strategy, and he’s been in the business long enough to have seen it all before and have some perspective on current developments. Anyway, here’s my questions and Nigel’s answers, and I’d be interested to hear any feedback or comments on what’s been said. Thanks again to Nigel and to Mark Handford for making this possible:

[Mark Rittman] : “There’s been a lot of innovation in the BI tools market over the past few years, plus a large amount of consolidation. What do you see as the major trends and opportunities in the BI tools market at the moment?”

[Nigel Pensde] : “Actually, I’m disappointed by the level of innovation in the BI tools market, even before the industry consolidation, which will certainly make it even worse.

Many of the claimed ‘innovations’ are actually rehashed versions of ideas that have been around for ages. For example, in-memory was the original BI architecture from more than 40 years ago (with APL), and column-oriented databases are almost as old. Dashboards, too, date back more than 25 years, to the brief EIS wave.

But there’s certainly been a lot of consolidation, which is probably bad news for customers, as they’ll be paying higher prices for products that don’t progress as much as they would otherwise have done (or which may be discontinued altogether). The new owners of the BI products are much more interested in integration between their many products, than innovation in non-core products.

The products are also likely to get more technical to install and implement, which may be good for consultants, but not for users.

I suppose the best opportunities will come for smaller, creative new vendors who take advantage of the likely mismanagement of the products acquired by large vendors. Many of the BI products now owned by the large non-BI vendors will fail to move forward or will not be promoted aggressively, thus leaving gaps in the market for nimbler vendors.”

[MR] : “There has been a lot of talk around in-memory analysis, desktop analysis and column-store analysis over the past two or three years, particularly with the release of products such as Microsoft PowerPivot and Qliktech’s Qlikview. What is your opinion of these products, how much impact will they have on the market, and are they solutions suitable for enterprise customers?”

[NP] : “Of course, QlikView is hardly a new product – it’s been around since the mid 1990s. The product sells well because it’s aggressively marketed, easy to use (at least for simple applications) and very fast. But it’s less successful for large, complex apps, and organisations that try to use it for such apps are less likely to be happy.

Of course, QlikTech will undoubtedly beef up the product’s functionality, but it has to be careful not to do this at the expense of ease of use. For example, products like Essbase and Microsoft OLAP Services (renamed to Analysis Services in 2000) started out as being aimed at business users. Now, both are far more capable and scalable, but also a lot more complex, and no business users could even think of developing new apps themselves (again, good news for consultants!).

PowerPivot is interesting. On the one hand, it’s just a way of making Excel 2010 bigger and faster, but not functionally richer. In this guise, I’m not sure how big the uptake will be, even if it’s free – after all, just how many people regularly need to download and analyse tens of millions of rows of data on their desktops? I think this may be more of a way for Microsoft to persuade its customers to upgrade to Office 2010 much earlier than they might otherwise have done, and for some of them to then use the latest versions of Microsoft server products like SharePoint and SQL Server.

But PowerPivot uses the very impressive new in-memory multidimensional VertiPaq engine, which will also feature in various Microsoft server products, such as SharePoint and Analysis Services (in fact, VertiPaq was developed by the Analysis Services team, and it will be an important new engine for Analysis Services). In this guise, it will certainly attract customers to these server products, by being easy to deploy, very scalable and extremely fast (but it currently can’t handle the more complex cubes that the older Analysis Services engines manage).”

[MR] : “Oracle have made big investments in business intelligence and enterprise performance management in the last five years. What do you think of Oracle’s strategy in the market, and how does it compare to IBM and SAP, two other large consolidators in the market? Do you see OBIEE having an impact in the market, and has the Oracle takeover of Essbase affected take-up and usage of the tool?”

[NP] : “Oracle has acquired many BI and PM products (partly as a by-product of other non-BI acquisitions, such as Siebel), rather than investing in building successful new products of its own. As part of this process, Oracle has inherited Siebel’s philosophy of providing pre-configured BI applications for its many OLTP applications.

This is probably a smart strategy, as customers who have invested many millions in shiny new transaction apps are probably not averse to spending a smaller amount to have better reporting and analysis of their OLTP data. OBIEE obviously plays a big part in this strategy. However, I don’t think OBIEE will have any impact in the larger (ie, non-Oracle) BI market.

SAP may attempt a similar strategy, but is in a weaker position to do so, as Business Objects did not bring with it a set of successful analytical applications that plugged straight into OLTP apps. And this isn’t even an option for IBM, which is not in the ERP business. To a lesser extent, Microsoft already does this with its Dynamics range.

Essbase sales into the Oracle base will certainly rise, but will probably largely disappear elsewhere as the product’s development tapers down. One irony is that Essbase had some very good dedicated OLAP front-ends when Hyperion was independent, but not for much longer. This means Essbase users will be expected to access it from OBIEE, which is not at all optimised for Essbase’s rich multidimensional structure. And this certainly won’t be an attractive option for non-Oracle sites.”

[MR] : “A recent article on The BI Verdict website mentioned a possible “OLAP Revival” following recent initiatives from the big vendors. Why do you think this is the case, and do not tools such as QlikView and PowerPivot negate the need for big investments in OLAP technologies?”

[NP] : “I didn’t write it, but it was a commentary on the increased investment being made in OLAP by all the major vendors, which remains true. For example, PowerPivot is the first deliverable from Analysis Services’ new VertiPaq engine. And one could argue that QlikView is a simple OLAP tool that fulfils exactly the same role today that Cognos PowerPlay did in the mid 1990s (ie, simple, business-oriented, in-memory analysis of multidimensional data).

IBM is also promoting TM1 much more actively than its previous owners did, and it’s to be the primary engine in Cognos Planning (which has always had its own proprietary engines until now).”

[MR] : “From your work with the BI Survey and BI Verdict, what do customers tell you are the critical success factors, and most common cause of failure, on business intelligence and OLAP projects?”

[NP] : “You probably expected me to cite the usual problems: lack of executive sponsorship and data access/quality. But, perhaps surprisingly, slow query performance is actually the most frequently-reported problem. Also, organisations that choose BI products taking into account their performance derive more business benefits than those who choose on the basis of vendor factors, like corporate standards.

This should be a salutary warning when people get over-excited about BI in the Cloud – that’s probably just about the worst way possible to deliver fast query performance.

Another key factor is involving business users in the actual implementation of BI and PM projects (ie, directly involved, not just sponsoring them). Projects that include business users in the team are notably more successful than those implemented just by IT people. And specialist BI consulting forms are more successful than either in-house IT or other types of non-specialist consultants.”

[MR] : “According to your Wikipedia entry, you’ve been working in the BI and OLAP industries since 1973. What do you think has been the most important innovation in the industry whilst you’ve been working in it, and what innovation do you think is over the horizon that will have the most impact on BI customers and developers?”

[NP] : “Not for the first time, Wikipedia is misleading on this. Back in 1973, I was a graduate trainee in an engineering company, and not using any BI tools. That came a couple of years later, and I didn’t actually join what we’d now call the BI industry until 1977.

Although I wasn’t using Express, the tools I was using back then had similar UIs (ie, a noisy teletype), and were aimed at OR specialists, rather than what we’d now think of as business users. Ironically, it means that modern tools are in some ways less sophisticated than their predecessors were 30+ years ago.

For example, FCS, the 1970s predecessor of Essbase, included Monte Carlo analysis, but Hyperion had to acquire a separate product (Crystal Ball) to get this capability into its 21st century product line. I’m also pretty sure that Express in the 1970s had statistical features not available in today’s Oracle OLAP Option and Essbase.

One thing that’s gone full circle is the architecture: in the 1970s, BI software was typically accessed using what we’d now call cloud-based, multi-tenant SaaS, using thin clients — in other words, dumb terminals connecting to a timesharing network’s remote mainframes (often in another country). You didn’t buy or rent the software, but just paid for usage. And it worked, so I don’t think anyone can claim that recent developments are ‘innovative’.

For example, I was responsible for a successful oil taxation model called Petrofisc some 30 years ago. This model was created using open source, collaborative techniques, and was widely used by oil companies planning their North Sea investments. It ran on a SaaS thin-client architecture and was even used simultaneously by competing oil companies negotiating against each other.

Of course, today’s BI software is far easier to use. It has a decent GUI (at least for the end users – techies usually still have to write code), even if it doesn’t do much more than its ancestors. Many things are now done automatically (and so they should be, given the huge increase in computer power with relatively modest performance improvements), and of course the volumes of data and numbers of users are much larger.

Getting hold of that data is now much easier, thanks to the move from very proprietary (often home-grown) transaction apps to modern ERP apps running on relational databases. Standard ETL tools and data warehouses mean that modern BI products don’t need to include such features themselves, as older products like Express once had to do.

However, many apparently promising innovations haven’t taken off, including:

• Advanced visualisation – for many years, it has been expected that innovative new ways of visualising business data would be the Next Big Thing, but most have failed to catch on. Instead, we have stupid gimmicks like animated speedometers and artistic 3D data presentations that just obscure important information. But we do finally seem to be getting some genuinely useful techniques, like spark lines, micro and bullet charts in mainstream products now. Needless to say, I made heavy use of these visualisation options in The BI Survey 8, with not a 3D chart or speedo in sight.

• Expert systems – these were all the rage in the 1980s and early 1990s, and it was hoped that they would help managers interpret the results from their BI applications, and suggest actions.

• Self-learning systems – again, this seemed a good idea 25 years ago, but it failed to take off. The idea was that BI applications would learn what metrics you spent most time analysing, what you then did next and how you liked your data presented. They would change the way they worked accordingly (much like modern automatic gearboxes adapt to your driving style). Thus, your ad hoc, interactive browsing habits would form the basis for automatic, adaptive dashboards and exception reports.

• Integrated data mining for end users – in the 1990s, BI vendors enthusiastically promoted simplified data mining for business users. But this was a big flop, as no-one but statistical specialists felt confident with such techniques.

• Integration of non-numeric information with BI – this has always seemed like an overdue idea, but it’s still not deployed widely.

• BI for mobile devices – OK, this hasn’t been a complete flop, but it turns out that BI users on the move would probably rather use a laptop than a smart phone or PDA for viewing BI reports and alerts.

• BI extranets – greedy BI vendors saw this is a brilliant way of selling even more licences to customers who already had all the licences they could possibly use for their own staff. But it seems that most companies don’t feel any great need to expose their internal BI information to outsiders. Nor is it clear who should pay for a BI extranet. So, yes, there are indeed a few BI extranets, but they are rare.

And what innovations are just over the horizon? By definition, I haven’t a clue, but the chances are that they will prove to be yet more recycled old ideas.”

Thanks Nigel, and you can find more information from Nigel (including some free analysis on trends in the BI Market) at the BI Verdict website

Looking forward to 2010, the big news later in the year – hopefully – should be the release of OBIEE 11g. I think we first heard news on this release back in 2007, and it’s certainly a well-anticipated release. If I was a betting man I’d expect to see it over the summer, hopefully around the same time that we get to see the 11g release of ODI (the contents of which even now are a bit of a mystery), and with the Windows releases of the 11gR2 database and OWB released earlier in the spring. But you never know, we’re certainly not privy to any release dates and we could be saying the same thing this time next year…

Expect to hear some interesting news on Rittman Mead as we go into 2010. Venkat has just put the finishing touches to Rittman Mead India, our new venture based out of Bangalore, with Venkat heading up a team offering support, training and expert services under his guidance. We’re about to relaunch our training services, starting with our OBIEE course and with a certification scheme that we’ll run internally first and then offer externally for expert-level developers. We’re also running another of our BI Forum events in Brighton next May, with the call for papers now open and a special guest presenter (who’s not me or Venkat, or indeed anyone from Rittman Mead) running a masterclass on the day before. Who could this be?

If OBIEE 11g gets delivered early enough in 2010, we should also have the Oracle Press book off to the printers by the end of the year at the latest, with our aim being to get the book in the shops no more than six months after the software goes into production. Back in 2007 when I started on the book, OBIEE had only just come out and it wasn’t all that clear what products would be in favour, and what would be sunsetted, once the dust settled. We also knew that OBIEE 11g would be out in 2008 at the latest (ho ho) and so we took the decision to hold fire until 11g was out, to maximize the “shelf life” of the book and give us a clear target to write for. Of course in the end 11g isn’t going to be out until 2010 which meant we could have got something out for the 10g release, but such is life, and at least we’ve built up some demand and anticipation for the book when it does come out. Following this, and if the demand is there, I’d also like to do something on the forthcoming “Fusion Apps” release of the BI Applications, which promises to as revolutionary in the packaged apps world as the 11g release of OBIEE has been for standalone analytics.

For me the highlight of 2009 had to be the BI Forum in May, as I’d always wanted to get together a small group of the most enthusiastic and experienced Oracle BI developers, and just chat, network and present on the topics we found interesting. I’d like to say thanks to all the friends and colleagues that presented, made the effort to come along and took part over the two days, and I’m looking forward to hosting the event again in Brighton next year. We’ll also try a Training Days event later in the year once OBIEE 11g is out, again watch this space for news later in 2010.

Other than that, have a great New Year, I hope 2010 will be prosperous and successful for you, and I’m looking forward to being back in action in January!

I’m based in the UK and so I had to wait until now to buy a Kindle, as up until recently it’s only been available in the USA. Amazon recently made it available for sale in quite a few English-speaking countries, with the Whispernet service that allows you to wirelessly download books now extended to 100 countries. It cost me about £180 or so and books are about the same cost as buying them in paper form (you have to buy through Amazon.com, and whilst prices are cheaper they have to add VAT on, as UK VAT is only waived for printed books).

From a usability perspective, it’s great. The size is about that of a paperback book, but it’s thinner and lighter so you can slip it into your jacket pocket or into your laptop back or rucksack. The design of the section generation Kindle is better than the first one, there’s a full keyboard and the device uses e-ink, which stays displayed with no power consumption (or very little). Battery life if you leave the 3G Whispernet connection off is a couple of weeks, with the connection it’s about five days, which is enough for most trips abroad for work.

Navigation around a book is through back-and-forward buttons, whilst some books have clickable tables of contents (though not all, which is a problem when you’ve got a reference book). I’ve found that the medium works best for linear books such as novels, as opposed to ones that you’d flick around like reference or technical books – some books don’t have clickable tables of contents as I said, and even those that do are difficult to “randomly access” as you can’t quickly scan through a chapter, page refresh is a bit slow and so on. As I said, novels work well, particularly as the last page you read is always remembered which makes picking up a novel after a break very easy.

I did try a newspaper subscription (The Independent), but the layout for these is very textual, very linear, nothing like a newspaper experience, so I cancelled during the free subscription period. For novels though (plus technical books that can be read start-to-finish) the Kindle has completely taken over from printed books for me, particularly as you can buy a new book and have it instantly downloaded over Whispernet in a couple of minutes.

From an Oracle perspective, a surprising number of technical books are available for the Kindle. I bought Jonathan Lewis’ “Cost-Based Oracle Fundamentals”, for example, which does have a table of contents and actually works well on the format as it’s readable in a linear way but handy to have as a reference. Quite a few of the Apress and Oracle Press books are available on the Kindle, but watch out for pricing – Edward Roske and Tracey McMullen’s “Look Smarter Than You Are with Essbase 11: An Administrator’s Guide” is listed on Amazon.com as $46.81 for the paperback version, but $71.24 for the Kindle version, which I think includes a surcharge due to the size of the book and the amount of Whispernet bandwidth it takes up (I’m guessing). Certainly given the electronic, non-physical aspect of the purchase I’d expect Kindle books to be about half the price of the printed version, in reality they are about the same price, perhaps slightly more, and there’s the VAT aspect that I mentioned that applies when you purchase from the UK.

So, in summary, I’m really pleased by the Kindle and it’s my exclusive way of reading novels (which gets a few strange looks from people on the train, as there aren’t many Kindles over here at the moment). For technical books, I still prefer a PDF on my laptop, but I’d ideally like to have these on the Kindle as well once they become more usable. I’m also keen to see what Apple come up with, with the rumoured Apple Tablet, as the Kindle is certainly very black-and-white, not a multi-media device, whereas the Tablet I’d guess will major on internet browsing and video. Still, for reading books it’s pretty good, you forget you’re reading from a device after a few hours, and the instant gratification bit of being able to order, and then read, from a list of thousands of books is pretty neat.

“The BI Survey 9: The Customer Verdict

We would very much welcome your participation in ‘The BI Survey 9: The Customer Verdict’, the world’s largest survey of business intelligence (BI) and performance management (PM) users (formerly known as The OLAP Survey).

As a participant, you will:

• Receive a summary of the results from the full survey
• Be entered into a draw to win one of ten $50 Amazon vouchers
• Ensure that your experiences are included in the final analyses

To take part in the survey on-line, visit: http://digiumenterprise.com/answer?link=270-5J9MMB9M

BARC’s annual survey obtains input from a large number of organizations in order to better understand their buying decisions, the implementation cycle and the business benefits achieved.

Both business and technical users, as well as vendors and consultants, are welcome to participate. If you are answering as a consultant, please answer the questions (including the demographic questions) from your client’s perspective; we will ask you separately about your own firm.

The BI Survey has always adopted a vendor-independent stance. While vendors assist by inviting users to participate in the Survey, Business Application Research Center (BARC) – the publisher – does not accept vendor sponsorship of the Survey, and the results are analyzed and published without any vendor involvement.

You will be able to answer questions on your usage of a BI product from any vendor. Your answers will only be used anonymously, and your personal details will never be passed on to vendors or other third parties.

The survey should take about 15-20 minutes to complete

* BARC (Business Application Research Center) is a leading independent software industry analyst specializing in Data Management and Business Intelligence. For more information on BARC please visit The BARC website, www.bi-survey.com and www.BI-Verdict.com“

QlikView, compared to Oracle’s relational and OLAP query tools, has a slightly different approach to storing and querying data. Whilst it can load data from relational and file-based data sources, it then stores the data in-memory in a compressed form with associations defined between data items rather than the traditional joins that we see in relational query tools. The idea behind these associations is that they are derived by Qlikview during the data load based on matching up columns across tables with the same name, and then when you query the resulting dataset you can analyze your data using these associations.

To take an example, the following QlikView demonstration shows data for a consulting company in a Windows thick client application. To analyze your data, you click on a data item, the numbers on the charts are then filtered by this value, and other data items that are associated with this selected value are then also made available for selection. The screenshot below shows this filtering in action, you can see the selected fields in green, the ones that are greyed out are no longer available for selection.

Now obviously this way of selecting data items, and this method of “association”, is fairly alien to traditional Oracle BI users and so I put together a simple example using the Sales History (SH) dataset available on recent Oracle databases. The first step in creating some QlikView reports is to define a new document, which in the end will hold the report layout, the data loading queries and the actual reporting data itself, which gets loaded in to memory when you open the QlikView document. Once you’ve defined the new document, you create a script to load data in from in this case an Oracle database.

This is in fact a similar process to putting load scripts together for Essbase and Oracle Express, with connections being made through ODBC and OLE DB for databases and direct connections being made to file and HTTP data sources. I get the impression that the most regular use case for data loading in QlikView is loading from files, and this would correspond to the target market for this tool which is desktop analysts who want to report on their data separately from the infrastructure around their enterprise BI tools and data warehouse.

Once you’ve brought your data in, you can view it either as a database (source) data model, or as the logical model that QlickView generates when it loads in the data. In fact for the SH schema, these are both the same as the joins in the Oracle schema are turned into the associations that QlikView uses, so what you see looks just like the regular sort of data model you’d see in any relational query tool.

The joins (or associations) are defined by QlikView during the import process by matching up columns of the same name – each match creates an association. This means that to create a join, you need to alias column names as they come in (to change ID to CUSTOMERID for example, if this is required to join to another CUSTOMERID), and you need to create aliases to break associations so that the STATUS column from the orders table isn’t linked to the STATUS column from the customer table when this isn’t really appropriate.

For an Oracle dataset with well defined primary and foreign keys, the logical table structure that QlikView creates in memory is usually pretty much the same as the Oracle schema, but for files or with data sets with lots of cross-related data, QlikView will create synthetic keys and synthetic tables to record the associations and assist with analysis. Internally QlikView tokenizes and compresses the data as well which meant that, for the Sales History data set that I worked with earlier, the total size of the QlikView document (including the data) was only 4MB and took about 1 minute to load and prepare, whereas the corresponding Essbase data set (as built in this OTN article) took up several gigabytes of space and took over an hour to load and prepare. Impressive stuff.

Loading the normalized data set that I used for this article on OBIEE data modeling produced the following logical QlikView table diagram, which as you can see is a direct reflection of the normalized Oracle data model.

This ability to report against normalized data obviously makes setting up the data model easier compared to OBIEE (notwithstanding the need to rename columns to make and break associations), but it then doesn’t make any distinction between columns that are measures and ones that are dimension attributes, which makes working with data in reports a bit more complicated.

So once you’ve got the data in, how do you create a report? And more importantly, how do these reports compare to Essbase or OBIEE reports? The actual report creation process is a bit like working with tools such as Hyperion Interactive Reporting or Web Analysis in that you construct an application made up of crosstabs, graphs, gauges and the like, which are then run either in the thick-client Windows application or online using a variety of thick and thin web clients. If I wanted to create a bar chart that allows users to select the product category, month and state of purchase and then graph product sales broken down by customer satisfaction rating, I’d add three field list boxes to my document and then create a bar graph chart, which would be automatically filtered by whatever I select in the lists.

In the screenshot above, I’ve first selected the “Computers” product category which filters the graph, and highlights the states and months that have sales for this category. If I then selected the CA and NY states, only the months that are associated with sales (quantity) for Computers and CA / NY sales would be highlighted, and so on. As such, the “Qlik” part of Qlikview seems to translate to progressive filters of your data based on the items you have selected, with the filtered-down dataset being made up of the data that will join (or “associate”) with the data items that you have selected.

If you’re looking to produce Essbase-like reports where data is shown in hierarchies, this is a bit trickier. There isn’t really a concept of hierarchies in Qlikview; you can create column groups which group together columns that are related, but I can’t see a simple way to select column members, for example, by their position in a hierarchy which is pretty standard in OLAP tools such as Essbase.

Displaying data in hierarchies is a bit simpler, especially as QlikView automatically calculates the relationship between columns and their values so that, for example, if you display product categories and products in a pivot table chart, the product are automatically shown under the categories that they relate to.

What I found trickier (as in I couldn’t do it) was to create a pivot table with regions, for example, across the top of the pivot, or to add page items or other controls. I’ve only used QlikView for a few days and so I may well have missed this feature, but none of the example documents that came with either of these features and therefore I suspect that it’s not something that the product does, for the moment.

Working through the examples that ship with QlikView 9, visualizations and fast access to data seem to be the main strengths of the tool. For example, the Sparkline graph type that features strongly in Stephen Few’s “Information Dashboard Design” book is present and correct in QlikView:

As are butterfly financial-style tables, used for displaying P&L reports and other finance data.

One of the examples uses Google Maps to illustrate data, in this case around house for sale in London.

Where QlikView is weaker in my view, compared to any of the Essbase-backed query tools, is around number-intensive and hierarchy-intensive applications. It’s not easy to select dimension members via a hierarchy, the scripting and calculation language that comes with the tool obviously doesn’t support the richness of languages like MDX and Essbase calculations scripts, and of course there are not the applications such as Planning and Financial Management that you’d expect to make use of in the Finance Department.

That said, for what it does (provide fast access to star and normalized data sets, including data sourced from files and from the web) it does the job very well, with excellent load and preparation times and sub-second response times, even with large data sets. You probably wouldn’t use Qlikview instead of Essbase or Oracle BI EE as they address different requirements, but as a sales analysis tool or a tool to analyze OLTP datasets, it takes some beating.

It wouldn’t surprise me to find that Oracle or another of the big vendors acquires QlikTech at some point as QlikView would make a nice complement to the more enterprise, heavy-duty query tools, especially when tools such as this and the forthcoming Gemini from Microsoft threaten to create desktop BI tool “bridgeheads” on users’ desktops as they are just so easy to use and to set up (can you imagine getting Business Objects and all its technology stack up and running over a couple of evenings, for example, let alone learning how to use it?). It wouldn’t also surprise me if vendors such as Oracle do more to bring their database and query tool processing in-memory, given the amounts of RAM available on users’ desktops these days and some of the advances being made in data compression technologies.

For anyone following the twists and turns of the BI industry, Panorama is of course one of the most interesting stories. Panorama were an Israeli start-up back in the early 90′s who went on to develop what was, at the time, a very innovative OLAP server (“Plato”) which Microsoft went on to buy and rebrand as Microsoft Analysis Services, as well as an OLAP client which they sold themselves and through an OEM deal with Cognos. For many years Microsoft then had a policy of not selling their own OLAP query tool, leaving the market to third-party partners such as Panorama, but then Microsoft went and bought ProClarity leaving Panorama in a bit of a sticky situation. About a year after the ProClarity announcement Panorama then went and announced the partnership with Google, moving into the “software as a service” market by plugging web-based versions of their OLAP client technologies into the Google Docs “cloud”.

Watching the two videos, it’s clear that Panorama’s Google Docs gadgets are aimed squarely at Microsoft Excel users, who either want a simplified environment, want to analyze their data on the web, perhaps don’t want to pay enterprise-level license fees for tools such as ProClarity (or even Panorama’s Novaview, their own desktop-based OLAP query tool), or just want to analyze their own personal data in a more Web 2.0-style environment. The guy from Panorama noted that most users of their gadgets started by uploading data from Excel, with the gadget environment designed to make analyzing your data as simply as possible, with for example an “I’m feeling lucky” button that gets the gadgets to make a best guess as to how you might want to analyze the data.

Under the covers, it appears that at least some of Panorama’s OLAP and OLAP client technology has made it into Google Docs and the Google cloud. After data is uploaded it gets automatically “cubed”, plus there are connectors through to Salesforce.com and other data sources. The tool is free (like Google Docs) for small data volumes and numbers of users, beyond a certain point it’s chargeable like Google Apps ($100/month was mentioned as the top price point).

So, the obvious question: how does this relate to Oracle business intelligence, is it something that should be considered a competitor? Let’s take the negatives first, then look at the positives. Firstly, it’s clearly aimed at users who would otherwise use Excel as their analysis environment, which I guess is a market you’d categorize as having more modest requirements, but is a very large market in size. There’s no semantic model or integration with environments like E-Business Suite, scalability isn’t clear and there’s no ETL component (though much of the ETL work may well be done automagically). As such it’s mostly a competitor for Microsoft Analysis Services, Excel 2007 and ProClarify and potentially also a competitor for Microsoft’s Gemini project, and so given the past history of Panorama, and Google’s relationship with Microsoft, I’d say that Microsoft are clearly the vendor in Google and Panorama’s sights when they launched this initiative. Also, you can’t ignore the privacy issue; customers have enough of a problem imagining uploading their most sensitive sales and performance data to a hosted Oracle BI environment that they’ve directly contracted for, let alone uploading it to Google to use in a potentially free service. I know for example of many customers who refuse to let their staff upload company data to Google Maps, as Google assert their right to own any data uploaded to their systems. I’m not saying that’s the case with this application but it’s a genuine concern for many organizations.

Looking at the positives though, certainly it potentially democratizes business intelligence and perhaps lowers the training barrier for people looking to chart and analyze their spreadsheet data. If you were a startup, a web business or perhaps a company worker looking to analyze their own data at home, it’d certainly be an interesting option particularly if you already use Google Docs. Indeed, at Rittman Mead we use Google Docs extensively to collaborate on documents, spreadsheets and so on, and it’d be relatively easy to take some of our internal data and publish it in the form of these gadgets internally on our intranet. One thing I found particularly interesting about the approach Panorama was taking with their open, Web 2.0-style approach to developing and releasing software, their use of twitter and blogs, and very frequent incremental releases of software (a new feature every day, was how they described it) which compares well with the long, drawn out release schedules for enterprise BI tools, especially now they’re becoming integrated with wider ERP and middleware suites.

So, an interesting interview, and it was good seeing Robert Scoble cover our area of interest. As I said, I think it’s aimed squarely at the Excel market, but there are some interesting lessons there about how BI software might be developed in the more collaborative, transparent world of Web 2.0. Take a look if you get a chance.

The past week I have been with a customer delivering a mixture of training and best practice advice. The training was based on my data warehouse design course, but heavily customised to fit their specific needs, and with complete new sections to look at data quality strategies and some specific approaches to managing a major BI development project. In reality, I guess that 40% of the material I presented came from existing Rittman Mead courses and the rest I created to fit the customer’s needs. I think this is a good way to work – I had always intended to write more extensively on data quality and the new slide set will be a useful add-in to our other training materials – and, of course, the customer gets the course they want.  I will write about some of the data quality issues we discussed in a future posting, a lot of this will be highly relevant to most BI / DW owners.

Next week I am working on some course customisations for another customer and then joining the others at our first ever Oracle BI Training Days – I have been working on this project for quite a while now and am really looking forward to seeing the fruits of our work. The whole team have put in a lot of effort; it should be a good event!