All the software it uses is available for download on OTN, and now Essbase is on version 9.3.1 it’s an easy install without any of the license management stuff you used to have to do on earlier releases. Take a look if you’re interested, add comments here if you’ve got any questions after working through the article.

This posting just concentrates on the Data Warehouse. Think of the project if we didn’t need a Data Warehouse, initially in terms of cost: no database license required, no ETL/Data Warehouse development required, no ongoing maintenance and maybe in the eyes of the business no black hole of money and time where the hairy developers go away, grumble about data quality and take longer than everyone thought. Think of all this new way from a business point of view: there is a new reporting requirement, they sit down with a business analyst, add the data source into the physical and then business model, graphically create the joins, and 10 minutes later the data is on the CEO’s dashboard, marvelous, and not a hairy developer type to be found anywhere in the process, so no cost and no hold-ups. But wait a minute: why have we been spending money developing Data Warehouses over the past couple of decades?

What was the Data Warehouse actually doing?

For starters, they prevented queries being run against the live transactional systems. Query and analysis tools can be very powerful, and hence can generate complex queries. If each of these queries was being run against the live systems then performance would be impacted, plus the reporting system would be dependent on all the systems being live all the time. Thus loading all the data into a ‘reporting database’ protected the transactional systems.

But what about our real-time Data Warehouse, managers need up to the minute information to make informed decisions? Real-time Data Warehouses can be a double-edged sword, it is great to have up the minute information, but the downside is that reports keep on changing, it becomes more difficult to reconcile information or to get a consistent view of the organisation. Sometimes it is actually useful to have a static view of the data for 24 hours. Real-time Data Warehousing can also be implemented using replication features like Change Data Capture in the database if required.

Data Warehouses also offer a number of functional advantages:

• they can store historic data that may have been archived from the transactional system.
• they can store the history of dimensions, so facts can be correctly categorised when they happened.
• they can store data from different systems.
• they can store the data in a way that makes it easy and efficient for users to query, this means that a large volume of data can be accessed and used effectively.
• they can use features like partitioning, bitmap indexes, materialized views to further speed up those queries.

What were those hairy developers actually doing?

They are writing ETL programs (or using a tool to) transforming the data from a number of disparate systems, and possibly a number of disparate organisations so that it can be presented in a unified model. This process can be easy for a small young company, however for larger company, with a history of acquisition this is far more complex. The reason this process can take such a long time is that it can be very hard to combine data from different systems. In the world of OBI EE we would create a join between the different tables – this is great in theory, but only really works if the keys are aligned between the systems. Generally they are not. Add to this the fact that the data quality maybe of poor quality and suddenly you find that the join is ‘losing’ a large number of records, or they are being incorrectly categorized.

The ETL process, and the hairy developers, will address this process and actually get in a state it can be compared across systems. I have worked on projects where the ETL team have lead/driven the business in the whole data quality and cleansing process.

So to turn the argument on its head, if we can do all this with the ETL process and in our Data Warehouse, and this does add value, why are we hacking things together in OBI EE, surely then everything should be done at the back end? Again there is some truth in this, if we are going to define complex time series calculations, and we can do them during the ETL phase, then it supports the ‘one version of the truth’, it means everyone will be using the same calculation and it can be rigorously tested.

If we had no constraints on time or money we may do everything at the back-end, or in the ETL, however the reality of it is that there just isn’t time (or money), or we often don’t know what the requirements are until later in the project. If we have written the ETL code and it is tested and working correctly and the business decides it needs another calculation then it may be a lot more practical to write it in situ in the report, or in the business model than to rework (and hence retest) the ETL code.

This is where OBI EE and an iterative development methodology come in. In a typical BI/DW project requirements are initially stated at the outset of the project, then they are then added to, changed and refined throughout the process. Users do not know what they want to report against until they see the data in a report in front of them – it is only then can they see the possibilities. To take the example of the calculation and how this could be implemented using this methodology and OBI EE: initially it could be prototyped directly in Answers. It could then be reviewed, refined and tested. Once agreed the calculation could be added to the business model, this way it would be available to all the users and they could add it to reports, this may lead to changes, or finding cases where it doesn’t work correctly. If the calculation becomes a core part of the system, is found to be stable and performance could be improved by materializing it, then it may be added to the Data Warehouse and hence the ETL code. We find that as the requirement becomes more stable it filters from front-end tools back into the database Data Warehouse.

This gives us the foundation of our methodology for approaching projects. It is initially important to gather and then implement/prototype the core or high priority requirements in the first few iterations of development, our understanding of these requirements only needs to be ‘good enough’ at this point and it is important to get them in front of the users. OBI EE can be used to do this. As they are then reviewed and refined they can be incorporated further into the system.

Once we requirements reach a certain stability they can be passed back toward the back-end of the system on future development iterations. This means that the more complex coding tasks, the ones that take time can be carried out on better understood requirements. This is where the Data Warehouse fits in and where it still can and does add value.

Importantly OBI EE gives us the ability to handle these changes seamlessly, we can change the physical objects supporting the business model. Mark wrote a series of postings about this process entitled Migrating OBIEE Logical Models to use a Data Warehouse, these describe how to re-wire the business model to a Data Warehouse.

So this means that old school ways, such as Data Warehousing, are not dead, interesting the latest Kimball Design Tip (I think you have to subscribe to get access – it’s free) discusses how names of the concepts of what we now call BI&DW have changed. What will make a project successful is finding a balance between how to optimally use each of the technologies, depending on user requirements, complexity and quality of source data and a myriad of other factors, and most importantly having a methodology that can stitch these approaches together.

One thing we’ve added recently is a section where we take the various heterogenous sources that we use in the exercises and migrate them to a small data mart using Oracle Warehouse Builder 11g, then “re-wire” the logical model to point to this single data source instead. In my experience this is one of those areas you’re never going to get on a standard public OBIEE course, but it reflects how developers might want to take their system on once they’ve gone beyond the prototyping phase.

Anyway, if you’re interested, drop us a line and we’ll tell you more about it. We’ve also recently put together an Oracle BI Publisher for Developers course that goes through report building but also integration with environments such as ApEx and Oracle Forms, I’ll post more on that when we pull the datasheet together.

“For example, suppose you are interested in the effect an advertising spend is having citywide. Your company invested in renting several prominent and expensive billboards in a city to advertise your latest products. Your current BI system can probably return the percentage sales growth by city area over the last month as a list of raw numbers from which you can create a graph and work out whether your billboards are earning their keep. Alternatively, you could mash up the same values as a heat map laid over the city and see instantly whether the high growth correlates with the billboards’ locations.

Now suppose that your BI system tells you that, in the past, sales of hot dogs double whenever the local baseball team plays at home after winning three games in a row. It’s a fascinating factoid, but it doesn’t make you any extra profit unless you can get the extra dogs to the relevant stadium in time for the right game. Your company is very unlikely to store baseball results, but there are plenty of sites out there that do. And there are plenty of shipping companies that provide data about trucking availability and pricing. On its own, your BI system is simply predicting the future. But mash up that prediction with these external data sources and suddenly your BI system is able to predict the future in a way that you can act on quickly to increase profits.”

Of course all this “mashup” talk reminds me of when I used to be a DJ, but it’s no doubt a very interesting idea and one that, as Dan said in his email, links in well with all the talk around BI and Service-Orientated Architecture. I’ll be keeping an eye on this area in future, and it’s got me thinking about putting something like this together for one of my UKOUG conference submissions.

If you’re a regular reader of this blog you’ll probably know that my interest is more around business intelligence than Web 2.0, enterprise applications and so on, but I thought I’d come along as I’ve got a tangental interest in these areas and of course I thought it’d be interesting to meet Charles Phillips in person. Uncharacteristically for me I ended up listening for most of the session rather than asking questions, I did get in a question around how Web 2.0 might influence business intelligence but for the most part, most of the discussion was around collaborative technologies, how Oracle plan to embed these in their applications to the point where they’re planning to put together “Enterprise 2.0″ teams within each Oracle’s seven regions.

It was interesting to hear Charles’ take on why Oracle feels the innovations in such areas as instant messaging, social networks, collaborative working and so on can apply to the heavy-duty enterprise applications that Oracle sells. The obvious counter to suggestions that Web 2.0 can apply to ERP applications is that you don’t want your financials application to look like Facebook; the point Phillips made though was that the ideas behind collaborative apps - social networks that form themselves, validation of expertise through peer endorsement, sharing of ideas, ability to engage with micro-markets - are all ones that can add value to the enterprise. If you divide what Web 2.0 ideas can bring to enterprises into two areas - internal and external - you can certainly imagine technologies such as Twitter, IM, blogs, wikis and so on being used to maintain communication with customers, and similar technologies being used within the organization to foster communication and idea sharing. Phillips talked about how these ideas are being used within Oracle; apparently developers are being encouraged to “tag” themselves to say what skills they have, what projects they’ve worked on, with their peers then voting on each others skills to validate claims of expertise. There was also talk on what Oracle plan to do with these technologies later down the line, perhaps eventually using technologies such as Web Center, Stellent and so on to establish a beachhead in the collaborative space, then extend this back to more legacy technologies such as email to take on Microsoft and IBM in the Exchange/Lotus Notes space.

Anyway, even though most of the discussion was outside my particular area of expertise it was a useful hour or so. Charles Phillips came across well; he’s obviously a very smart guy and he gave pretty straightforward answers to the questions - I’ve been in similar sessions in the past where it’s been like the Oracle person was answering from a press release. Kudos to Oracle and Charles Phillips for doing this; the questions were by no means easy the audience was a pretty smart bunch as well. Throughout the session there was a photographer taking photos, I worked out after a while that they were all of Charles Phillips and one thing that made me smile when I accessed the photos online was how well they came out when you use a professional photographer - the one below is a typical one and compares well the usual ones we try and take of each other when doing talks at conferences, where I usually end up looking very shifty or that I’ve got something wrong with me (though nothing will ever beat this one).

Incidentally, if you followed the interview with Phil Bates that I ran in two parts on this blog in April, I’m just waiting for Phil to come back from his holiday before I get him to answer your questions. I’m also going to run a similar interview with Jean-Pierre Dijcks, the Product Manager for Oracle Warehouse Builder, so if you’ve got any questions on OWB product direction add them to the comments or drop me a line and I’ll put them to him.

I would be tempted to split the event date and time into two separate dimensions a ‘date’ dimension with a granularity of day and a ‘time’ dimension that has sufficient resolution to capture the events of interest; there is no date component in the time dimension, for example events that occurred at 09:18:00 on May 6 2008 will share the same time dimension key as events that occurred at 09:18:00 on February 5 2008. This approach keeps the time dimension compact, we only need to key the possible time values that occur in a single day, and more importantly it opens up the possibility of simple time range queries aggregating over many days.

Anyway, what I thought would make it interesting is to open up questions to readers of this blog; if you’ve got a question on Oracle and Web 2.0 (social networks, building applications collaboratively, that sort of thing) just add it to this posting and I’ll try and ask them at the event. Also, I’m hoping to stretch the subject to “BI 2.0″ as well, so again if you’ve got any questions in this area, add them as comments as well.

When working with OBIEE, my understanding is that you can have one BI Server per physical/virtual server, which can connect to one or more repositories, though only one of them is the “default”. You also normally then set up one BI Presentation Server instance which connects through to the default repository for the corresponding BI Server. In reality this means that each OBIEE environment consists of one BI Server and one Presentation server on the same physical server (together usually with an installation of BI Publisher, configured to use BI Server security), or in some cases the BI Server and BI Presentation Server might be split onto different physical servers, and possibly further clustered on to more servers if the expected load is high.

You can, it should be noted, set up additional Presentation Servers on the same physical server, each of them connecting through to the same BI Server but with different “default” repositories, which gives you a way to create your DEV, PROD and other environments on the same physical server (if it’s a beefy Unix box, for example). However this is a fairly complex task (see this blog posting by Borkur) and it’s usually easier to have a simple pairing of one BI Server, one Presentation Server (plus the BI Publisher, Delivers etc servers) per environment with all of these contained on their own physical server. The rest of the migration steps in this posting therefore assume that you’ve got each environment set up in this straightforward way, with OBIEE already installed on each server and a single installation of BI Administrator that can connect to both of them.

When you think about migrating an OBIEE environment, there are four main things you’ll want to move across:

1. The BI Server repository, which contains the enterprise semantic layer, variables, security settings (including users and groups if you use RPD-based security) and connection pool settings through to the physical data sources. The contents of the repository are normally contained with a single RPD file contained in the %ORACLEBI/%server/Repository directory, with %ORACLEBI% being determined by the SAROOTDIR environment variable and the current RPD in use (only one can be online at any one time per installation of OBIEE) being determined by the %ORACLEBI%/server/config/NQSConfig.INI file.
2. The BI Presentation Server web catalog, which contains the reports, dashboards, prompts, ibots and so on, together with a separate set of users and roles (usually propagated from the BI Server repository) and their permissions on the web catalog objects. Unlike the BI Server repository, the web catalog is stored in a set of XML files, one per web catalog object organized into folders, plus an ATR file per folder that contains permissions on the objects within them. The files that make up the web catalog are held in the %ORACLEBIDATA%/web/catalog directory, with %ORACLEBIDATA% being determined by the SADATADIR environment variable.
3. Connections through to physical datasources, held on the server containing the BI Server, which when working in an Oracle environment are usually contained in a single TNSNAMES.ORA file. The connection pool settings in the BI Server repository references the connections in this file.
4. The BI Publisher Report Catalog, which contains the XDO files that define the BI Publisher reports. It is assumed that BI Publisher is configured to use BI Server security or LDAP security, therefore the users and groups in the BI Publisher OC4J container will not need to be migrated. This catalog is normally found in the %ORACLEBI%xmlp/XMLP/Reports directory.

Note that this list isn’t exhaustive - if you know of anything else that needs migrating (perhaps to do with Disconnected Analytics, perhaps the configuration files for the Presentation Server) let me know by adding a comment to this posting.

In a typical customer situation, the environment migration lifecycle looks something like this:

Step 1 : Initial Development (”First Cut”)

All development of the repository and web catalog initially happens in DEV. At some point, the initial work is complete and the DEV environment is ready to be copied into PROD (or TEST, or whatever)

Step 2 : First Deployment to Production

The initial creation of the PROD BI Server repository is done through copying the DEV RPD file in to the PROD environment. Copying across the entire RPD file will bring across all subject areas, physical models, presentation models, security settings, users and groups (if RPD security is used), variables and so on. If your OBIEE server does not have a TNSNAMES.ORA file on it to connect to the source database, you will need to copy this across from the DEV server as well. You can either copy this repository “interactively” by using the filesystem explorer application and cut and paste, or you can script the copy using a batch or shell script. If you need to make changes to the repository “en-route” to the PROD environment, you can instead of copying the repository “whole”, export it into a UDML text file using the using the nQUDMLGen.exe utility within the %ORACLEBI/server/bin directory, alter any settings that you need to change using a perl script for example, and then import the amended text file into a new, blank PROD repository using the nQUDMLExec.exe utility. See this posting on this blog on UDML and repository migration and merging, and this posting by Venkat on automating changes to connection pool settings using UDML during a migration.

The corresponding Web Catalog is migrated by copying the contents of the %ORACLEBIDATA%/web/catalog directory to the corresponding directory on the production server. This directory contains top-level folders for Delivers and Answers/Dashboard contents, like this:

With each top-level folder then containing sub-folders and then individual file pairs for each web catalog item, like this:

The BI Publisher report catalog is migrated by copying the contents of the %ORACLEBI%/xmlp/XMLP/Reports directory and the %ORACLEBI%/xmlp/XMLP/users directories over to the production BI Publisher environment.

In all cases, the RPD file, the contents of the Web Catalog files, any UDML files and BI Publisher report files can be stored in a version control system to create versioned releases of your project metadata.

3. Subsequent Full Refreshes of Production

If production ever needs to be fully refreshed with what is in development, then the above process can be repeated. Note however that ordinarily, any reports or customizations that happened in the production environment (including users’ custom dashboards) will be overwritten using this process; these can however be preserved if the sub-folders below the %ORACLEBIDATA%/web/catalog/web_catalog_name/root/users`are saved before being overwritten by the DEV web catalog files, and then copied back after the migration takes place.

4. Incremental Refreshes of Individual Subject Areas, Tables, Columns & Corresponding Presentation Areas

Incremental migrations of elements of the BI Server repository can either be done interactively or through scripting.

Interactive migrations are carried out using the Copy / Paste feature in BI Administrator, and can be used to copy across any element from subject area down to individual column from one repository to another. To do this, open two copies of BI Administrator with one having the DEV repository loaded offline and one having the production repository loaded offline. Use the Copy and Paste feature of the tool to copy elements from one repository to another, like this:

Check the repository consistency afterwards, and if all is OK then open the production repository online to start using the migrated objects.

Automated (scripted) migrations are carried out using the nQUDMLGen.exe and nQUDMLExec.exe utlitiies, which can export and import repository elements from one repository to another (again this posting on this blog for examples of these utilities in use.) Automated migration in this manner is usually preferred in real, production situations as it removes the chance of operator error when migrating from one environment to another. If the physical tables to which logical tables map are the same in both the DEV and PROD environments (which they ideally should be) then logical tables should preserve their mappings when migrated from one environment to another. Scripted migrations are also the only way of migrating non-table and dimension settings from one environment to another, such as variable settings, user and filter settings etc). You can also take the text files created using these utilities and store them in a version control system to create versioned “patches” to your OBIEE metadata.

5. Incremental Refreshes of Individual Reports, Dashboards, Alerts, Prompts etc from the Web Catalog

Like repository migrations, Web Catalog migrations can be done either interactively or through manually copying filesystem objects, potentially in an automated fashion using batch files.

Interactive migrations are carried out in a similar way to repository migrations, but this time using two copies of the Catalog Manager application open, one connected to the DEV web catalog and one connected to the PROD web catalog. Like the BI Server repository, the web catalog can be opened offline by navigating to the %ORACLEBIDATA%/web/catalog/catalog_name directory (be careful that you open the correct directory, otherwise the Catalog Manager application will helpfully create a new web catalog for you rather than point out that it cannot find one at this location), or online by connecting to the BI Presentation Services URL. Once connected, you can cut and paste objects between the two catalogs as shown in this blog posting here.

Automated (scripted) web catalog item migrations can be accomplished by copying the web catalog element (a file of the same name as the report, dashboard etc with no file extension) and it’s corresponding ATR file (the same name with ATR at the end, as shown in the screenshot above) from the DEV web catalog to the PROD one using filesystem copies.

These copies can be batched up and automated using .BAT or unix shell scripts in order to copy across multiple reports and other elements, note the comment on this by Adrian Ward at the end of this blog post. Again it’s worth noting that automated migration in this way is usually preferred by customers as it removes the chance of someone inadvertently or deliberately changing a report during migration, or needing “live” access to the production environment. Note also that the two Web Catalogs need to have the same user and group settings for permissions to migrate correctly across both environments.

Miscellaneous Object Migrations

On any mature OBIEE system, there are no doubt other elements of each environment you would also wish to migrate. Some of these might include dashboard themes (XSL and other files), TNSNAMES.ORA files and/or ODBC DSNs, any clustering files and settings and BI Server users and groups if you are using RPD security (this would need, I believe, to be done using UDML scripting, see this other posting by Venkat). If anyone reading this knows of any others then add these as comments to the postings.

Migration Example : Complete Environment Refreshes whilst Preserving User Reports and Dashboards

So that’s the theory of how you go about migrating from one OBIEE environment to another. For this particular customer though, we can take a slightly simplified approach and release updates to the production environment by just copying across what’s in development, as long as we have a method to preserve any reports or dashboards that users create in the production environment and restore them after the migration takes place. This is a bit of a simpler process than using nqUDMLGen.exe and nqUDMLExec.exe as we won’t be migrating across individual parts of the repository or individual reports or dashboards. In addition, this customer uses the same database instance for their DEV and PROD OBIEE installations, so we can use the same TNSNAMES.ORA contents for all environments.

To put in place a routine for this migration, we will need to carry out the following steps, organized in to two parts; the first part (steps 1-5) will be concerned with gathering up all the repository and report metadata from the DEV environment and copy it to a staging area, with the second part (steps 6-9) copying this metadata across to the PROD area, after first taking a copy of any user-generated reports and dashboards.

1. Somewhere in a staging area, we create a directory for the metadata to be migrated. We call this OBI_RELEASE_x, with x being the release number.

2. Within this directory, we create four subdirectories
   1. RPD
   2. WEBCAT
   3. WEBCAT_USERS
   4. TNSNAMES

3. We then create two Windows or Unix batch/script files, with three input parameters
   1. REPOSITORY_NAME (to hold the name of the repository to be migrated)
   2. ORACLEBI (to hold the root directory of the Oracle BI Server installation)
   3. ORACLEBIDATA (to hold the root directory of the Oracle BI Presentation Server installation)

4. For this customer’s installation, they are not using BI Publisher or Delivers, so we will leave this out of the migration at this stage.

5. We then create a first script that copies the DEV environment metadata into the OBIEE_RELEASE_x directory ready for migration. This script performs the following steps:
   1. copy the %REPOSITORY_NAME%.RPD file from %ORACLEBI%/server/repository/ into /RPD directory.
   2. Copy the %ORACLEBIDATA%/web/catalog/%REPOSITORY_NAME% folder to /WEBCAT
   3. Copy the TNSNAMES.ORA file used on the server to /TNSNAMES

6. We then create a second script, which is used to copy the metadata from the OBIEE_RELEASE_x directory to the PROD server. It carries out the following steps:
   1. Shut down BI Server so that you can overwrite the repository file.
   2. Copy the /RPD directory to %ORACLEBI%/server/repository directory
   3. Copy the %ORACLEBIDATA%/web/catalog/%REPOSITORY_NAME%/root/users to /WEBCAT_USERS
   4. Copy the /WEBCAT directory to %ORACLEBIDATA/web/catalog/
   5. Copy the /WEBCAT_USERS directory to %ORACLEBIDATA%/web/catalog/%REPOSITORY_NAME%/root/users
   6. Copy the /TNSNAMES directory to the database client /network/admin directory
   7. Restart the BI Server

7. Once these scripts have run, to test that migration has happened correctly we perform the following steps:
   1. Go in to BI Administrator, check you can connect, check you can do row counts, check global consistency. Check that all users you expect are there
   2. Log in to dashboard, check that all reports and all users’ reports and dashboards are present.

I’ve tested this approach out on my DEV and TEST environment and it migrates all data, reports and dashboards correctly.

As I mentioned previously, what I haven’t tested it with is BI Publisher and Delivers, but the principal should still be the same. As usual, any comments or suggestions are most welcome.

Although it’s not so big outside of the North America, baseball is of course one of America’s most popular sports, with big TV contracts and top players on contracts worth tens of millions of dollars each year. Baseball is unusual though in that it lends itself to statistics gathering more than most sports; unlike football (soccer) baseball is a structured game where events happen in a defined order and the individual contribution of players can be easily measured. What certain people had begun to realise though was that the statistics that players were usually measured on - stolen bases, runs batted in and so on - were not really the key factors in whether a team won or not, and in fact less glamourous measures - walks (the equivalent, I guess of 0-0 draws or packing your defense in football), on base percentage and avoiding an out were better indicators of whether a team would win. The problem was, the traditional measures of player skills tended to emphasise “glamour” players, players who once in a while turned a game or pulled off some amazing feat, whilst the less glamourous measures tended to favour the more pedestrian players, the ones that turned in a solid performance each month but were never considered stars.

What Billy Beane and the Oakland A’s did was take this analysis, which was called “Sabermetrics” after “Society for American Baseball Research”, and used it to pick players who weren’t glamourous, weren’t the established stars, were perhaps fading stars who could still do one thing - say, ensure they always got on base - and buy them for a pittance. Then, they took these players and proceeded to grind out effective but boring victories, to the point where within their regional league they had the lowest payroll figure and the highest amount of points scored and the end of the season. Going back to the analogy, it’s like a premier league side selling all their star players, searching the lower leagues for players who never lost the ball or took risks and then going on to get themselves into Europe.

Where the business intelligence interest is, is around their use of analytics to spot what are effectively inefficiencies in the market, the market for baseball players. The market is pricing players on one set of statistics that actually aren’t good indicators of team success, what Billy Beane and the Oakland A’s did was determine what the true significant statistics were, buy players based on that and then, in effect, win games at a lower payroll cost. In effect it’s arbitrage, as the players that the Oakland A’s bought then often went on to be transferred to higher spending teams who didn’t use Sabermetrics, whereapon the A’s would then take the money, but more mis-priced players and repeat the process again. The trick though was to spot what the true signifying statistics were, and this is where techniques such as regression came in as hundreds of amateur statisticians ran the numbers and tried to establish just what player traits and actions were most likely to lead to either runs being scored, or hitters being caught out. Going back to the “Competiting on Analytics” book, it’s a classic case study of an organization gaining competitive advantage through an analytical approach to their business.

Of course the real inspiration wasn’t the raw numbers, they’d been available more or less for free for many years, it was the ability of Billy Beane to firstly spot the opportunity, then have the courage to act on it, and then to determine, on an ongoing basis, the correlations and the causations. Once the Oakland A’s had their initial success some of the other, better funded teams copied the ideas, hired some of their staff, meaning that the A’s then had to repeat the process, finding more inefficiencies and so on, to keep things going. One interesting twist to the story was that this approach gave them great results in the league, but when they got to the play-offs and the World Series, they found they couldn’t reproduce their success; they put this down in the end to the fact that their analytic approach meant that on average, they’d win more games than lose, but for the one-off, sudden death games, one star player could turn the match. Also, they were clear in saying that the intended benefit of the approach wasn’t to make them the best in their league, it was to deliver results at a much lower cost, which still meant that for the big, end of season games, they would almost expect to lose as they hadn’t invested the required huge amounts of money on the star players that were needed.

Again going back to the Premiership and football, Bolton in Sam Allardyce’s days are the obvious comparison, they hired in either unknowns or faded stars, used the ProZone statistics, got amazing value out of them but were never really going to win a cup or get into the Champions League, they didn’t have the equivalent of a Ronaldo or Gerrard who could win a match for them single-handed, but for a commensurately high salary.

Anyway, I thought it was a good book, and a good companion to Competing on Analytics. It gets a bit too baseball at times (I certainly had to turn to Wikipedia once or twice to work out what a “pop fly”, for example, was) but it was good nonetheless, and probably a book you’ve heard being discussed at some point or other if you’re from the States as I understand it caused a bit of a controversy.

Another requirement within the same organization was to send reports that are generated directly to the printer, without the intermediate step of displaying them, say, in the Adobe Acrobat browser plug-in. In this case, BI Publisher was due to be invoked from an ApEx application, and whilst they could put a button or a link on the page that when clicked, invoked a BI Publisher report via the user’s default Web browser, the user would then have to click on the Adobe Acrobat plug-in’s “Print” button, select the print destination and initiate the printing themselves. In their ideal world, the button on the ApEx page would just call out to BI Publisher and have the report sent straight to the printer. Again on the face of it this wasn’t going to be easy, as BI Publisher, unlike Oracle Reports, only lets you specify the report format and the report template, not the report destination, when you invoke a report. Also, BI Publisher, unlike Reports, doesn’t have a tag or a function that you can include in a report definition to switch printer trays mid-way through the report generation, so again we were at a bit of a loss how to meet these requirements.

Thinking about it for a while though, I did remember that when you schedule a BI Publisher report, you can specify the report destination, the tray if it’s a printer, and you can specify a time of “immediate” for the scheduled execution.

Now this looks promising. If we broke the report up into several separate reports, one for each printer tray, and submitted them all at the same, we could select the printer tray to print to and they’d all arrive at more or less the same time, such that the end user wouldn’t really know that they had been delivered separately. Moreover, as the report invocation would come from an application, the user would just run the report as normal, in the background several report components to several printer trays would be scheduled, and the user would just think they’d requested a single report that printed to several trays. The only thing missing from this, compared to the Microsoft Word mail-merge approach, is that the report couldn’t be sent to the user’s “default printer” - BI Publisher has no concept of a default printer, all it can do is either send the report as a PDF to the user’s Web Browser, whereapon the user could use the Adobe Acrobat browser plug-in to print to their default printer, or in the solution we were going to use, we’d need to replicate this functionality by maintaining a list of available printers within ApEx and BI Publisher, and allow the user to specify which printer was their “default”, ensuring we printed to that one in future.

Following this then, my first thought was to try and invoke the scheduling of a report via a URL, as you can with Oracle Reports. Looking through the documentation though, there’s no programmatic way (at least based on my short investigation) to invoke the scheduling of a report, at least through a URL or through the command line. One thing I did remember reading about recently though was a Web Services interface for BI Publisher, which if it was anything like the Web Services interface for OBIEE, would potentially give me a way to schedule a report from a ApEx via a Web Services method invocation. Before I could try this out though, I needed a way to set up a set of “virtual” printers, FTP servers and email servers on my laptop so that I could check that scheduling itself worked correctly. To do this I downloaded and installed Bullzip PDF Printer to create a virtual printer for my laptop, configured Microsoft IIS to provide the access to this virtual printer via a URL, installed FileZilla Server as an FTP server and ArgoSoft Mail Server as a local email server. Once all of these were installed and configured and I proved that I could schedule a report and sent it to these three destinations, I was ready to start working with the BI Publisher Web Services interface.

Now in reality I hit on a bit of luck with the setting up of BI Publisher Web Service access for ApEx as I came across this excellent article by Tyler Muth that documents the whole process. In the article, Tyler uses a tool called soapUI to do the initial Web Service test, which you can use initially to check out the various parameters you send to BI Publisher via this method. Sure enough, the section on invoking a report includes a parameter for the printer tray.

Now unfortunately the one thing I can’t do at this stage is actually test it with a printer with multiple print trays, as our printers back in the office only have a single tray and the Bullzip PDF Printer software only simulates a single tray. But if we work on the assumption that the tray parameter in the BI Publisher Scheduler request method actually works, if we can get the Web Service call working this should prove a solution, at least one we can take back to the client in a week or so’s time to try and implement.

I used soapUI to put together a proper Web Service request to BI Publisher as a test, requesting that the report be delivered to the email server, FTP server and printer, leaving out the tray parameter at this stage. This is what the request XML looked like:

Executing the Web Service call using soapUI caused the report to “print” via the virtual printer, get delivered as a PDF to the FTP site and emailed to my local email server as an attachment, so it looks like it worked. So now to ApEx to see if I could invoke the report from an application page, again following the instructions on Tyler’s blog page.

The first step within ApEx was to set up the Web Service request as a manually-defined Web Service reference reference, which allowed me to add all of the optional parameters around delivery destinations and so forth. The service description URL I determined by taking a look through the WSDL file for the BI Publisher Web Service.

One this was done, the next step was to set up a button on an application page that invoked the web service. In reality, the button on the real customer application will actually need to call several web services, one for each of the report elements, and as this would be a fairly common process it would make sense to create some sort of shared procedure that would take a report name and any report parameters as inputs, and then actually call out to BI Publisher for all of the individual report elements, with that procedure then being tied to a button on a page. For now though I just wanted to check that it worked, so I set up a simple button and process on my ApEx page and told them to invoke the Web Service reference that I just set up.

And it worked. Clicking on the button I created led shortly afterwards to the report arriving via email, via the printer and via the FTP site, and it shouldn’t be too tricky to extend the page so that multiple processes, one for each report, get fired off when the button is pressed, or we use a procedure or something like that.

So what we’ve done here (or potentially, as I haven’t yet got my hands on a multi-tray printer to test it out) is a way of taking a report that requires printing to multiple trays, and instead splitting it into one report section per tray and then scheduling it, for immediate delivery, via the BI Publisher Web Services interface. I’m due back there in a week or so’s time, so I’ll get the chance then to integrate it into their ApEx application and check that the tray support in the Web Services API actually works as advertised. The client is also interested in integrating BI Publisher with Oracle Forms as well, and this other document on OTN sets out how this process works, it’s not quite as straightforward as with ApEx but the principal is still the same.