The application itself is one of the first Fusion applications and I initially saw a demo of it at Open World last year. I think it’s actually available for licensing now and is typically used on a hosted basis, with customers’ sales people uploading data to the application and the back-end system then analyzing it. Where Oracle Data Mining comes in is that it uses its algorithms to take each customer and predict which (if any) products each of them are likely to buy, generating predicted income figures and time to close based on its calculations. It then clusters the customers and their predicted product interest and value and places this on an Flash-based bubble-graph for users to interact with.

The other thing that the application does with the ODM clustering feature is to work out which other customer might provide a reference for the product that’s being predicted, based on similar attributes and predicted needs. All of this is put into a single Flash-based application that’s aimed at salespeople and account managers.

Obviously this is a fairly sales-focused application but its an interesting application of the Oracle Data Mining technologies. My understanding is that ODM is also going to be used at Oracle Open World to recommend sessions for attendees based on their profile and existing selection of sessions, as I’ll be over there it’ll be interesting to see how effective this is.

I first started out in the intelligence space (deliberately omitting the word business) working on high performance text stores. In reality this was all about storing streams of text in a highly searchable form, in effect the text was held as an index, here I came across then esoteric concepts as tokenisation, tagging, synonym dictionaries, and stop-words; however, I did not use stop-words as every word stored (even if misspelt) was potentially significant. Since those days analysis of free-text has become more mainstream. Linguistics academics use it to unravel language semantics (my wife’s dissertation was on computational linguistics) Others I know have worked on voice to text systems capable of identifying speakers and what they say. But although text is the most accessible (or understandable) form of unstructured data, increasingly people need to search binary data, whether it be biometrics, images, DNA sequences, spatial data or whatever.

Finding single records that contain text fragments, topics, DNA patterns can be an interesting indexing challenge, but when we start to move into the space where we looking for patterns that occur in multiple records we start to move well away from the normal data mining domain. Determining that a  particular DNA sequence correlates with increased risk of disease is a complex task; there is little reason (apart from the enormity of the computational task) that DNA samples could not be used to predict facial appearance. Computational problems exist in text analysis especially where rich vocabularies make semantic matching difficult – so how do you get cookie to match biscuit in some contexts and biscuit to match brown in others.

Why is there a ‘B’ in BI?

Well there needn’t be. A lot of what we do is for businesses, so the B makes some sense, but other organisations that are not businesses often use exactly the same techniques and tools to examine their data. The B could be some form of generic term to qualify the word intelligence, perhaps in a way to distinguish what we are doing from Military Intelligence, Criminal Intelligence, Market Intelligence or any other such qualifier. But this is a somewhat bogus division, BI (or any of the other intelligences) is not just a single discipline, and definitely not a discipline unique to a single division.

BI is about finding information

Storing information in a database is without point if you do not have a way to access it again, and access it in a form that suites your business purpose. But the way we need to get at that information varies with purpose:

• some people have an need to report on the historical, this often requires finding and reading a large amount of data, sorting it and aggregating it
• others look at historical data in varying amounts of detail, that is drill up, down and through the data and in so doing may exploit pre-built aggregations or OLAP cubes
• then there are those with an interest in the here and now, the operational reports, perhaps from live (transactional) feeds.
• finally, in the historical data camp, there are the miners looking for relationships and connections between events.
• and others are using the past to predict the future, looking at current events and a knowledge of past patterns to apply probabilities of outcomes.

But if BI is all of the above, how can we build a single physical model that encompasses it all. The answer is that we probably can’t. To an extent, the needs of co-located data to minimise bulk data reads contraindicates the needs of data mining; partitioning historic data flies in the face of live transactional feeds (it can be done though) and how do we do light-weight speedy predictions in a way that can be used in real-time by an agent using a CRM system?

Talking of Brighton, Jon Mead writes a few notes on data warehouse healthchecks. I completely agree with his approach – it is the same as mine :-) The article is worth a look. As Jon points out the exact steps taken in any healthcheck vary – there is such a variety of design goals and techniques in building warehouses that there can be no such thing as the universal data warehouse diagnostic script. It needs technical experience and a eye for business data analysis.

But at the end of the day we all are trying to answer these four questions

1. Does the data load quickly enough
2. Does the data load frequently enough
3. Can the data be read quickly enough
4. and, can you trust the quality of the data

Sounds simple but often needs an expert eye.

I mentioned two challenges yesterday, putting the data into the database and finding it again. Systems that can do one of these steps very well often falter with the other process. If you decide to use a very sophisticated tokenized column based database to give you blistering data read access you may find the computing required to work out where to insert the data is too expensive (cost or time) to make the process viable, conversely direct writing of data to disk may give us the best insert rate but something less than optimal for reading. If you index the raw data how much of an overhead is there in maintaining the index in the batch and does that process interfere with concurrent user activity; after all trickle-feed is a now viable dataload strategy? If you don’t index then how do you find what you are looking for? Not that indexes are always a help; take the example of looking for exception values, it is just not common to index fact measures.

By necessity our data load process is inserting one (or hopefully, many) data records into our data warehouse; that is, we insert complete records. Whether they come from flatfiles, XML files, database links, or whatever they are converted into database rows, maybe cleaned up first, but at the end of the day, inserted into the DW – we may be doing fancy stuff on the way with transportable tablespaces, partition exchange or change data capture, but the net effect is the same. All of the data passes through our DW and ends up written to disk.

But getting the stuff back is not the same, we are generally interested in a subset of the information stored in the DW, we may be filtering on a group of dimensional attributes and further only looking at one or two of the fact measures stored. We may also aggregate this data down to create a yet smaller result set. But the problem with conventional DW technology is that the whole result set is brought back to the CPU on the data warehouse to be manipulated and as we know this can be slow. The database appliance vendors push out pre-processing of the results to the disk arrays; some vendors using clustered PCs serving single disks, and Netezza uses a field programmable gate array between the disk and backplane, this is approach is particularly interesting in that some very sophisticated query logic can be executed next to the disk. The downside to processing “at the disk” is the case where the data interaction is between rows stored on different drives – as is the case in data mining and other pattern finding applications

It can be argued that coupled with sensible partitioning schemes we can minimise IO in a data warehouse (of any large size) by building pre-aggregated summary tables; the pain of the build query happens just once instead of each time the aggregate is needed and by getting the partition granularity right we can minimise aggregations operations to just the partitions that need rebuild.

But what when stray from the domain of of structured DSS type queries? That is away from those queries where (even for ad-hoc queries) we remain able to use our wisely chosen aggregates or exploit the partitioning scheme of the base data. Suppose we look at data mining where the statistical relationships between data items are being determined or we search unstructured data for patterns, relationships or just index its content. And now-a-days unstructured data is not just text (if it ever was) we have speech and speaker recognition, image recognition from the (now) trivial OCR, through fingerprints and facial recognition to the more complex ability search libraries of image components to identify photographic locations. These require inventive indexing techniques but they also require fast access to the underlying data, and for big datasets is that going to be possible?

Maybe ELDBs are not the way to go for unstructured data or data that needs extensive analysis. Perhaps the way to go here is through database federation, keeping the computation close to the disk and coordinating the outputs to produce the end result