Our original plan was to cover Oracle BI Suite Standard Edition and Enterprise Edition, together with Oracle Warehouse Builder, and base the book on the 10g releases of the various products. The idea here was to take readers through the various tools in OBISE (Discoverer, Reports, Discoverer Portlets) and show how they could link in with the new tools (OBIEE, ODI and so forth). As we started writing the chapters though, it became more and more apparent that all of this would come together much more coherently in the 11g timeframe, with for example OWB being able to create OBIEE metadata, OBIEE Dashboards being able to incorporate Discoverer worksheets, Essbase coming in to the frame, OWB orchestrating the Oracle database bit and ODI having a more obvious role in bringing in non-Oracle data. There’s also a whole bunch of new functionality coming in around process integration (the “Action Framework”) that’ll be so central to OBIEE 11g and it’s integration with Fusion Middleware that we just didn’t want to leave this out.

In short, it made a lot more sense for us to wait for the 11g release of Oracle’s BI tools rather than write the book for what is obviously a “work in progress” in terms of integration and functionality. Coupled with the fact that, with publishing schedules as they are, the book would probably only be out for a few months before OBIEE11g and OWB11gR2 would start becoming available, holding fire and writing for these new versions made a lot of sense.

So the plan right now is to resume writing the book once the beta releases of OBIEE 11g and OWB11gR2 become available, and aim for handover of the manuscripts to the publishers towards the end of 2008, which gives us a publication date around the second calendar quarter of 2009. With a bit of luck this will coincide with the public release of the software, which means the book will be spot on in terms of topicality and should hopefully be something that’ll be of a lot of interest to readers of this blog. Whilst you can pre-order the book on Amazon (and I do appreciate the support here), in reality nothing will ship now until that time, and I’ll certainly keep you all posted on progress on the blog. In the meantime, thanks for the enquiries and interest and I promise it’ll be worth the wait.

Working on the OWB ETL chapter has certainly been an eye-opener. I’m positioning the book as being a sort of “developers’ companion” rather than a manual re-write and therefore took the decision early on to try and base the examples on what you see in the real world, rather than what you knock up in thirty minutes for a quick demo. My OWB setup, therefore, consists of three separate databases; an OTLP database that contains most of the source data, a Repository database that contains the design metadata and the staging schema, and a Data Warehouse Database that contains the ODS and Analyic schemas. I knew up-front that I was going to need Control Centers on both the Repository and Warehouse databases, as even though the 10gR2 repository is now “unified” (the repository tables now contain both design and control center tables) you need to create one on each database that you deploy to.

Where it gets tricky though is when you get to the point where you need to start deploying mappings and process flows to the other, “secondary” Control Center on the data warehouse database. By default, the Control Center Manager only controls the Control Center you’re currently putting design metadata in to, and to start working with the other one, you need to create a new Configuration (a new 10gR2 feature), associate it with the other Control Center, and then switch to this other configuration.

When you start working with this new configuration though, and start up the Control Center Manager, none of the deployment locations you need to work with – the ODS one, the Analyic one – are listed. I was eventually able to get them to appear in the (otherwise blank) list of available deployment locations just right-clicking on them within the Design Center and then selecting “Deploy”, but afterwards I found that I could double-click on the alternate Control Center in the Connections Explorer and then shuttle them across to the “available” area and tick the “Target” box to make them available.

It was all a bit complicated actually, and not something that you’d work out by just playing around with the interface. Creating a new Control Center was straightforward enough, as well as adding it to the Connections Center, but the bit around having to create a new configuration just to be able to deploy to a second database – surely the most common scenario – was very, very confusing.

Still, I think I’ve worked out the correct workflow when setting up OWB10gR2 when you’re planning to deploy the main warehouse tables to a database separate to the one containing your main Control Center:

1. Create a repository on the database you’re going to use as your “OWB Repository” database, and another one on the other database that you’re going to deploy to. Whilst in the Repository Assistant, create and register with the Control Center the schemas that you’re going to deploy to (staging on the OWB repository database, ODS and Analytic on the warehouse database, in my case)

2. Start up Warehouse Builder 10gR2 and connect to the repository on your OWB repository database. This is where you’ll create your design metadata. Using the Default Configuration and Default Control Center settings, create your source modules and the staging target module, together with their locations. This default configuration will then “own” those locations.

3. Register the other Control Center (repository) using the Connection Explorer (top right-hand part of the screen), then create a second configuration that uses this other Control Center.

4. Using this new configuration, create the ODS and Analytic modules together with their locations.

5. Now you’ve set all the configurations and locations up properly, switch back to the default configuration and go and create the tables, dimensions, mappings and so on required for the project.

6. When deploying to the staging module, use the default configuration and control center as normal. When deploying to the ODS and staging locations though, switch first to the other configuration and then deploy using the Control Center Manager. The ODS and analytic locations should then be visible in the Control Center Manager; if they’re not, go back to the Design Center and enable them by double-clicking on the relevant control center in the Connections Explorer, and select the locations using the second tab of the dialog.

Anyway, that’s how I think it’s meant to work. Tonight I’ll be going back to the project and re-creating the repositories, locations and modules to get it laid out just right for the book examples, and then hopefully I can move on from working out the examples to writing the chapter proper. Tomorrow, I’m working during the day and then going up to Heathrow for my flight out to Stockholm in the evening; when I come back on Wednesday I’ve got a couple of days back in the office to do some prep for a “data warehousing for business users” course I’m writing for one of our university clients. That’s it for travel then for a while; I’ve been to some interesting places recently (Chester, in the photo at the top of the page and in this flickr group, being one of them), but it’ll be nice to be back home for a while after being on the road for the best part of four weeks.

I suppose what I should really write about is the process of putting my book together. It’s actually quite interesting really; I signed the contract with Oracle Press just before Christmas, got my ISBN number (there’s actually a placeholder on Amazon.com now, which makes it all sort of official), and now I’m working on the initial set of chapters before the publishers appoint the technical editor and so on. This is sort of the “probationary period”, where they see if you can actually get yourself in gear and produce chapters in a reasonable time; at this point you’re looked after by an acquisitions editor who guides you through the first phases, works with you on style and formatting and just generally makes sure things are “on track” before hiring technical editors and copy editors.

So far I’ve produced the introductory chapter, and the first one on Oracle Warehouse Builder, the one on configuration and data modelling. A couple of aspects have been a challenge; first of all, keeping to the alloted number of pages for each chapter, for which it’s just as important you don’t overshoot as undershoot as the book’s projected sale price is budgeted on a certain number of pages – if you’re going to go well over, it’ll cost more to produce, and more often than not this is more down to you being too verbose rather than your initial estimate being wrong.

The other thing is settling on a style; I’m aiming more for the Tom Kyte “tell and show” approach rather than just rewriting the manual, but it’s tricky at least initially to settle on a style that’s appropriate for a technical book, but that isn’t realy boring. The approach I’ve ended up taking is spending the first third of the chapter setting out, at a high level, what each component does, with the remaining two-thirds being a walk-through of how the product is used in a typical scenario. I’ve created a set of example data that I’ll use consistently through the book, and whilst I won’t go through every step to build the warehouse and report against it, I’ll cover the key points and hopefully readers will be able to follow along. I don’t know if this is acceptable as an approach (or whether there is “an acceptable approach”) but I’ll know soon, as it’s with their editors at the moment to review before moving on to the next chapter.

Working out how much time to devote is also a challenge. My timetable says that I have a month to do each chapter, with ten more due over the next ten months. I’m currently spending at least two hours each day (in the evenings) on it, but I’m trying to discipline myself to spend at least half of the month working through examples, testing things out and so on rather then spending the whole time banging out words and only really covering what I knew beforehand. A major motivation for me was the chance to properly work through some things I’d only briefly covered before, and so for example in the ETL chapter I’m just about to work on, I’ll be using the multi-configuration feature, version control, scheduling, data auditors, experts and Discoverer integration, features I’ve only looked at briefly before but that I want to cover properly in the book. In a way, I’ve got a tough audience to cover in what I’m doing; for most people, many of the products I’ll be covering will be completely new to them (XML Publisher, BI EE, even OWB 10gR2) but there’ll also be people, probably attracted by the blog, who’ll be after “expert tips and tricks” as well. And all with a budget of about 30-60 pages per product in the suite.

Anyway, that’s it for author musings. Moving on a bit, how cool is this?

If you look. that’s the Windows version of Oracle Warehouse Builder running in Apple OS X – or at least it looks like it; what it actually is is OWB running in a Windows virtual machine using Parallels and the new “Coherence” feature (screencast here) in the latest beta release. What this does is let you start up a virtual machine as normal, then switch to this new coherence mode, with all the windows applications then appearing in their own application windows on your OS X desktop. This means you can have OWB, AWM, Microsoft Word, even Internet Explorer and Outlook all running on your apple desktop as if they’re OS X applications, you can drag and drop to them, and work with them just like any other applications.

The way it seems to work is to “lose” the virtual machine window container, plus the Windows desktop, and instead just show the windows applications directly on your apple desktop. At some point the effect breaks down – when you drag a Windows application across the screen, Parallels shows the Windows desktop under where the application originally sat until you stop dragging, and you have to move the taskbar to the top of the screen and put it on auto-hide to get it out of the way, otherwise it’ll display just like any other windows application. But it’s pretty cool though – when working with applications in place, they integrate completely in with your OS X applications and you can leave Outlook, or OWB or whatever just running in the corner and otherwise work entirely with your Mac applications. Not bad.

That’s it for me for now though. As I said, I’m doing a free seminar on the 17th up at Oracle City Office (places still available, I understand) and then it’s off for the BI Masterclass, organized by Oracle University, in Austria on the 22nd/23rd January, Slovenia on the 25th/26th and Sweden on the 30th/31st, and I think one’s also being organized for Spain at the end of February. Until then, it’ll be heads down and hopefully I won’t get too many edits back from the Oracle Press team.

It’s a bit too early to get in to too much detail about the content and book structure, but the aim of the book will be to take developers through the process of building a business intelligence system based on the Oracle database, Oracle Business Intelligence Suite Standard and Enterprise Edition, with database design and loading being done through Oracle Warehouse Builder and Sunopsis Data Conductor. In the best tradition, it’ll be full of examples and development techniques, it’ll touch on areas such as SOA and business process integration, and will be all-new material developed specially for the book.

Over the next twelve months I’ll be posting examples as I work them through for the book, plus a bit of insight into the process of writing a book and then getting it published. Coupled with this, and some other changes that I can’t go into now, it looks like 2007 is going to be a fairly busy year for me but hopefully by the end of the year it’ll all be worthwhile. For now though, it’s back to putting the examples together, then it’s on to the first of the Oracle Warehouse Builder chapters.

I took the day off today as leave so that I could try and get the basics of the data model together; I’ve divided the task into two sections, firstly coming up with the data model, secondly putting some data in it (again, not a trivial task). In the past, and for the seminars I’m currently running, I’ve put examples together using the sample schemas that come with the database, that come with Analytic Workspace Manager or that come with Siebel Analytics, but for the book I thought I’d put my own dataset together, which has the added benefit of allowing me to shape the examples such that I can show off specific features of the tools – integrating data across platforms, using change data capture, reporting across subject areas and so on.

Anyway, I’ve based the sample data on a fictional IT software software vendor and services provider (stick to what I know…) called “Porthall Software”. Porthall Software uses Oracle Database 10g as it’s primary database platform, but has certain sets of sales and customer data on Microsoft SQL Server 2000. Their product catalog is held in an XML document, and certain sets of reference data are on spreadsheets and flat files. Their budgeting application is based on Microsoft Analysis Services, and they use Oracle Warehouse Builder 10g for loading their Oracle databases, and Sunopsis Data Conductor to load other platforms. Finally, they have some legacy HR data in a Microsoft Access database which they want to access via Generic Connectivity, and legacy Orders data that they want to bring over from another Oracle database via Transportable Tablespaces. To make it all work, I’ll need a couple of 10g databases and of course SQL Server and Access running alongside – hopefully my Parallels virtual machine with 1.25GB of RAM allocated will be enough.

The source data set starts off with a couple of Oracle schemas:

• PORTHALL_HR, a Human Resources dataset that we’ll load via a database link, and
• PORTHALL_ORDERS, an OLTP-style order management scheme that we’ll load via a database link, and via asychronous hotlog change data capture.

The PORTHALL_HR schema is pretty straightforward, with employees, jobs, departments and genders tables – this is the easy set of source data to help kick things off.

The PORTHALL_ORDERS schema is a bit more complicated, as I want to match it up to other data sets later on – some customer intelligence data that we’ll load from spreadsheets using an OWB Expert, and some customer data we’ll load in from a CRM system.

The order_stages table is something I’m going to use in the chapter on analytic business processes (BI and SOA); the table holds the dates that an order is at, and I’ll put together a BPEL process that calls out to the BI Suite EE BI Server at the “customer credit check” stage.

Incidentally, if you’re interested in how I did the table diagrams, it’s with a freeware tool called Schemester – it’s useful if you’ve already planned out the table and join definitions, and you want to quickly set the tables and up and generate some diagrams. It’s not too clever when it comes to generating the DDL – it seems to keep using the same constraint names which causes errors when you rn the scripts, but I’ll just manually edit these myself later and create a single install script for the sample data.

Next I’ve got a second set of HR data which I’m going to store in Microsoft Access – this is to represent a set of employees who’ve recently joined via an aquisition, with the point being to show off how data can be integrated (the columns don’t match up with the other HR data) and how Generic Connectivity can be used to connect to non-Oracle sources.

To show off how file data can be integrated, there’s a set of inventory information in a CSV file.

and a set of budget data in an XLS file – OWB can’t usually work with XLS files without a whole bunch of work around generic connectivity, whilst the BI Suite EE BI Server will connect directly, and allow you to join to the data without first loading it into the data warehouse.

There’s going to be another spreadsheet setting out the financial periods, which we’ll bring in and then supplement using some derived columns to put together an Oracle OLAP-compatible time dimension. Next, I’ll bring in some CRM data from a the PORTHALL_CRM schema in an Oracle database over a database link, with some of the columns deliberately vague (country and city fields left as free-form, customer ID optional, no validation on customer name or contact name) so that I can demonstrate the match-merge and name and address matching in OWB.

The budget forecast data will be in a Microsoft SQL Server 2000 Analysis Services cube; I haven’t diagrammed it here as I’ve not got SQL Server installed yet.

Next, there’s some additional orders data from the PORTHALL_ORDERS_LEGACY schema in a legacy Oracle database, that we’ll bring over using the transportable modules feature in OWB.

Later on, when I come to build the data warehouse to support BI Suite Standard and Enterprise Edition, I’ll create staging, operational (atomic) and analytic layers, and we’ll copy some of the data across to the SQL Server database containing the Analysis Services budgets cube to test out the features of Sunopsis Data Conductor, and to show BI Suite Enterprise Edition running against Oracle and SQL Server data at the same time.

The final bit of source data will be a product catalog in XML format. I’ll be using this later on to supplement the data in the Oracle data warehouse, and I’ll bring it into the BI Server to see how it handles XML data.

So that’s it for the source data design. The next steps are to design the target schemas, and then to go back to the source tables and start populating them, with the reference data being keyed in and the sales, inventory and budget data being created programmatically (hopefully…). We’re not doing performance tuning, so large volumes of data aren’t essential, but I think the most difficult thing will be to make the data “make sense”, and to add some quirks and interesting bits into it so that I can point to “insights” that the various tools provide. Anyway, I thought you might be interested in seeing how I put the sample data together, if you’ve got any suggestions on other things to add in, let me know. Hopefully later in the week, I’ll look at the target schemas and plan out some of the features I’ll be looking to show off.