I briefly discussed this last year on my blog and probably also over at Datageekgal’s blog (and congratulations Beth on the new job!) The indexing needs of the security services, medicine and a whole host of organisations that need to index patterns within a LOB type object will spawn some pretty clever indexing methods, and hopefully some of those will become accessible through database vendors products.

In a way there are some similarities with data mining, except that a LOB could (I think would) contain bit patterns for more than one index key value. We are probably talking about non-unique indexes, as for non-archival purposes researchers are usually concerned with finding similar records.

But one good thing about the need to index LOB contents is that they are usually non-volatile, a recorded conversation or a DNA sequence is historical fact and is not going to change so perhaps index updates are not going to be important. Most of the building blocks to do this type of indexing are already available (especially if we choose to create an index of the “index” by using some form of indirect table approach) the only bit to do is to write the domain specific code to identify the keys values in the LOB… hang on that’s the hard bit!

This is one of the fruits of the recent purchases by Oracle of Hyperion (Data Relationship Manager), Tangosol (Coherence *) and Sunopsis (Data Integrator) and with a bit of Application Server, BPEL and Enterprise Service Bus thrown in and the ability to use an embed data quality and profiling product from Trillium *

* Coherence and the data quality options are add-on to the base ODI Suite

This looks interesting, I might write more on this later

Step up VMWare Fusion – I have relatively rapidly built a Oracle Enterprise Linux 5 64-Bit VM guest OS to run on my Apple, cloned it as my base build, mastered how to change the name of the screen header for the VM machine so that I know which VM I am running and proceeded to install 11g database software and a small test database. This is great fun as before this week I had not touched Linux (let alone gnome) – true I could speak Solaris, Dynix and AIX, and it is so long since anyone allowed me a root password. So, now I have two linux servers, an empty one that I can base subsequent builds on and a working 11g database.

I suppose the next thing is to build a Windows VM for the stuff that needs Microsoft Windows (OBI SE-ONE, for example) – but why is Windows so expensive to license? If I do go with the clean ‘gold-build’ VM and clone it before adding the minimal software required for the purpose in hand I will need to watch out how I clone the machine – or else VMWare will change the MAC address of my virtual network adaptor and break the license key

I first started out in the intelligence space (deliberately omitting the word business) working on high performance text stores. In reality this was all about storing streams of text in a highly searchable form, in effect the text was held as an index, here I came across then esoteric concepts as tokenisation, tagging, synonym dictionaries, and stop-words; however, I did not use stop-words as every word stored (even if misspelt) was potentially significant. Since those days analysis of free-text has become more mainstream. Linguistics academics use it to unravel language semantics (my wife’s dissertation was on computational linguistics) Others I know have worked on voice to text systems capable of identifying speakers and what they say. But although text is the most accessible (or understandable) form of unstructured data, increasingly people need to search binary data, whether it be biometrics, images, DNA sequences, spatial data or whatever.

Finding single records that contain text fragments, topics, DNA patterns can be an interesting indexing challenge, but when we start to move into the space where we looking for patterns that occur in multiple records we start to move well away from the normal data mining domain. Determining that a  particular DNA sequence correlates with increased risk of disease is a complex task; there is little reason (apart from the enormity of the computational task) that DNA samples could not be used to predict facial appearance. Computational problems exist in text analysis especially where rich vocabularies make semantic matching difficult – so how do you get cookie to match biscuit in some contexts and biscuit to match brown in others.

In the beginning there was the product, and then came the customers, the systems that counted the money made from selling, the mergers and acquisitions that brought in new products (and thinking, processes and models) and finally there was the desire to tie all of this together in one holistic view of the enterprise. And there is our first problem, dotted throughout the company are fragments of a model to suit specific purposes. How an ERP system such as SAP treats product, may not be the same as how a product lifecycle system sees it; and it may well not be be sensible to coerce the two models to look the same (have you ever wanted to make a major change to an ERP system just for the sake of it?).

As my e-friend Beth mentioned unifying product is one of the trickier problems in creating a super model. Take for example an ice cream maker. Product development devise hundreds of recipes, and track them on their product development system, the best of these recipes go to taste trials and results recorded, the winners here go through to the marketeers who have a final say on whether they think that fig and chili ice cream will really be a commercial success. And then production recipes are created and costed, foods standards clearances obtained, launch budgets set and sales performance targets cast. Each stage we are dealing with the same product and the potential need to track back through this data when needed (think compliance auditors) Then add in the need to combine all of our systems with those of the competitor we just bought and need to make sure the commercial attributes of the product remain private for SoX compliance reasons, and the need to keep existing IT systems working without the need to be rebuilt to accept the new data model and the only practical solution is to build a metadata based model that links the existing systems.

Money laundering is a big issue in the UK, not necessarily because it is a major problem, but more because of its political focus. This manifests itself in two ways: the need to establish that you are who you say that you are when we you open a bank account (see here & here) and need to track large cash transactions both in the banking world and at stores. Somewhere there is conception that cash is not a legitimate form of currency; cash is the proceeds of crime (be it drug dealing or VAT (sales tax) avoidance) It is also the way which people without bank accounts and credit cards trade, and often the main part of the daily finances of all those cornershops that sell the odd carton of milk and a newspaper to passers by. So tracking potential laundering in retail transactions is not that clear cut; we need to detect the crime from the “noise”

Looking at fraud – we have two significant strands with credit cards; falsely obtain cards from providers by using stolen ids, which was the main focus in the San Diego Union Tribune piece Joel cited – that is fraudulent applications. The second is where a legitimate card is hijacked (stolen, cloned, or just the details used on the web) Here we need to develop techniques to verify the fraud at transaction time. Goodness know how much the banks are doing to achieve this.

But fraud is not restricted to credit card finance. Fraudulent return of goods is also costing retailers money. Again we need systems that can be used at point of sale to indicate whether the refund is valid. And this needs to happen quickly: the customer should not feel they are being questioned, and the extra transaction time should not slow the sales process too much.

It can be argued that coupled with sensible partitioning schemes we can minimise IO in a data warehouse (of any large size) by building pre-aggregated summary tables; the pain of the build query happens just once instead of each time the aggregate is needed and by getting the partition granularity right we can minimise aggregations operations to just the partitions that need rebuild.

But what when stray from the domain of of structured DSS type queries? That is away from those queries where (even for ad-hoc queries) we remain able to use our wisely chosen aggregates or exploit the partitioning scheme of the base data. Suppose we look at data mining where the statistical relationships between data items are being determined or we search unstructured data for patterns, relationships or just index its content. And now-a-days unstructured data is not just text (if it ever was) we have speech and speaker recognition, image recognition from the (now) trivial OCR, through fingerprints and facial recognition to the more complex ability search libraries of image components to identify photographic locations. These require inventive indexing techniques but they also require fast access to the underlying data, and for big datasets is that going to be possible?

Maybe ELDBs are not the way to go for unstructured data or data that needs extensive analysis. Perhaps the way to go here is through database federation, keeping the computation close to the disk and coordinating the outputs to produce the end result

Before I started out with data warehouses I did a lot of work on analyzing data, that is searching masses of information looking for links between data items – seeking out new and unknown connections, often connections separated by considerable time delays. To make this work I had to impose structure on the unstructured and this meant borrowing techniques from all over; artificial intelligence and fuzzy logic, network theory and even astro-physics (not all of my data sources were textual!). Borrowing continue apace, and now there seems little to delineate technologies. What I used to do then has become common and not just in the traditional DW monoliths of the past: Real-time credit card fraud detection is being pushed out to the point of sale by tapping in on the XML streams back to the centre, events recorded on different systems can be linked through messaging and not just rely of them being transported to a single data repository and being found by some batch process running way after the event has significance; as I said in an earlier post, the number of executive choices after an event diminishes as time progresses – there will be a stage where there is no longer a choice.

Maybe we are heading to future when the disciplines are no longer ERP and OLAP or transactional and decision support but become just Data Storage and Data Retrieval

Disks drives can only read from one location of the disk at a time, true, this data read rate may be high and the time to jump between locations may be low, but this is still a slow process compared to CPU and memory operations. If all of a system’s data sits on one disk then every disk read and write will go through a single point and consequently IO throughput will suffer. This would be particularly noticeable in a data warehouse where table scans are common events.

In a previous role (before moving back into the mainstream world of Oracle data warehouses) I worked on 'free-text' information systems and in particular those produced by the Memex company. The basic idea behind these databases is that each unique word in the database is a assigned an integer token and instead of storing the actual text only a stream of integers is stored. On data input each word is checked against a dictionary – if the word exists it is substituted with its token, if it is a new word the next available integer is assigned as the token and the word added to the dictionary. For example if the first phrase ever to be entered in to the database was 'The cat sat on the mat' then: the will be assigned 1, cat = 2, sat = 3, on = 4, the is already 1, and mat will be 5. Of course this is a gross simplification but you can see how this works. Data output requires each token to be decoded and built into a text stream.

Of course this word dictionary based approach has its own set of problems. We are storing each unique character string as a token – bad typing or spelling can cause multiple tokens for the 'same' thing, however, it is not always appropriate to 'correct' the input text to a standard form. Likewise the use of synonyms can be problematic – if the text says 'car' is this the same as 'automobile' or 'motor vehicle' or indeed is 'colour' the same as 'color' and then there are parts of speech – 'drive', 'driven' and 'drove' are distinct words, but people may need to search for any of the words. The approach adopted by Memex is to store the 'as entered' text (tokenized) in the database but to produce tools that change the search to use, in effect, an in-list of tokens for alternate words.
Searching for the use of a single word in the database is fast – if it is not in the dictionary then it is not the database otherwise the database is scanned for a bitwise match with the token. It is possible to enhance the search by looking for multiple words, phrase searching and proximity searching (that is word 1 appearing within n words of word 2). Further enhancements allow the documents to be tagged in much the same way as XML so that data between specific tags takes a particular semantic meaning (or if you like, structure) Tagged data is especially useful when coupled with specialised data mining and data relationship graphing tools.

Back to Mark's piece – is this a useful technique in BI? And can something similar be implemented in Oracle? Firstly, the Memex approach is non-relational and as such does not fit readily into the conventional relational model DW approaches (Kimball or Inmon) The target user of these free-text databases are typically looking for single (or very few) record based on a search, this is perhaps closer to the Electronic Content Management community (interMedia?) than BI where users are looking for quantitative results and trends based on multiple records. But can any part of this technique be used in a conventional Oracle data warehouse context? Of course it is possible to build a token dictionary, perhaps as IOT of word token pairs, this will need to be indexed on both tokens and word to permit efficient coding and decoding. But where could we use these tokens in a data warehouse? If we adopt a model where the dimensional attributes are only stored in stored in one or more 'reference' data tables and not stored in the 'fact' tables at all (well apart from the join key which is, in a way, a token!). If tokens are to save space in the database (because the token takes less storage than the original word – which may not be true anyway) we will need to use each token at least twice in the reference data table to recover the extra storage needed for the word-token dictionary. For a 3NF DW we will have a hierarchy of reference data tables with each table consisting of unique records, this reduces the potential for duplicating attribute tokens; it does not eradicate it though as items could easily share attributes, for example in a fashion DW a tee-shirt and a dress could share the attribute of 'blue'. In a fully denormalized DW there is far more scope for attribute sharing. But is it worthwhile? – the reference tables themselves are short – how many customers or products do you have?, probably less than 1000000 rows. And add the necessity to join all the queries to token dictionary table to create the selection criteria and again to decode the tokens to make the output human friendly then tokens seem to add a performace overhead that may not be worth living with