OBIEE 11.1.1.6 : MDS XML Repositories, and Integration with Source Control

February 22nd, 2012 by

So if you’re a repository developer working on Oracle Business Intelligence 11g, probably the most interesting new feature announced with the release of OBIEE 11.1.1.6 yesterday was support for version control within the BI Administrator tool, and a new repository format called MDS XML. Let’s take a look now into what MDS XML is, and how version control integration with the BI Administrator tool works.

Up until now, Oracle BI Repositories were stored in binary files called “RPD” files, after the .rpd filename extension. Using a single monolithic file was a design decision nQuire took back when the product was first developed, and allowed their consulting team to rapidly deploy metadata sets onto servers without having to worry about database connections, deployment procedures and so on. However, having all of the repository metadata in a single file causes us lots of issues now, in particular:

  • Its pretty-much impossible to extract individual objects out of the repository, and put those objects under version/source control, and
  • Having all metadata in a single file means we end up jumping through hoops when trying to run multi-developer projects, giving us the pleasures of the Multi-User Development Environment

So, primarily I think driven by the requirements of the Fusion Apps development team, the 11.1.1.6 release of Oracle Business Intelligence introduces a new format for saving repositories; the MDS XML format. Similar, but not quite the same as the XUDML format that’s used when patching repositories, MDS XML breaks the repository into individual XML files, with each one corresponding to a first-class object such as a logical table, logical dimension, physical table, physical join and so on. MDS XML is only available when working in offline mode, so to save a repository in MDS XML format, first open it offline, and then select File > Save As > MDS XML Documents…

Once you’ve saved the repository in this format, the title bar for the Oracle BI Administrator tool will display the folder name, and if you look inside the filesystem folder, you’ll see your repository stored as a set of subfolders containing individual XML documents.

If you then subsequently want to open the repository saved in this format, you open the Oracle BI Administration tool as normal, but this time select File > Open > MDS XML …, which then prompts you to select a folder rather than an RPD file, and then supply the repository password. So what’s the point of this? Well, the answer is in the next question that the Oracle BI Administration tool asks you – do you want to open the MDS XML repository standalone, or use source control?

Choosing the Standalone option just opens the repository as normal. If you subsequently want to upload this repository to the Oracle BI Domain using Enterprise Manager, you’ll need to re-save it in the classic RPD format, and then do the upload using that file. But how does the source control option work?

Let’s look at a situation where you’re using Subversion as your source control system, and TortoiseSVN as your SVN client. You’ve set up your SVN repository using the standard folder structure, so that it looks like this (I’m using the free-as-in-beer Visual SVN Server as my SVN server and Apache server combined):

Now to configure the Oracle BI Administration tool to work with Subversion, I need to go into the Tools > Options… menu and select the Source Control tab. This presents me with a dialog with two buttons; one to Edit a configuration file, the other to create a New one. The Oracle BI Administration tool is actually designed in this release to work with any (in theory) source control tool, by mapping a set of internal commands to add files, remove files and mark files for update ito the actual commands used by the source control system you’ve chosen; let’s press the New button to see how this works.

When I press New, I get a dialog where I can enter the actual command-line commands used by my source control tool to match the generic source control operations that the Oracle BI Administration tool will perform.

Now obviously this might sound a bit daunting, so if you’re using Subversion, there’s a pre-defined configuration file you can use instead. Pressing Cancel, I then press the Edit button instead, select the subversion template file, and view the options that have been added for me.

This looks OK, so I press Cancel and go back to the main menu. Let’s now go through the check-out, development, save and check-in process.

1. I’m starting now with an empty /trunk/Repository folder in the Subversion repository. I start therefore by using TortoiseSVN to check-out this directory to somewhere on my filesystem, like this:

2. Now, I want to save my repository in MDS XML format into this checked out folder (in SVN terms, the working copy folder). I do so by opening the regular RPD file in offline mode, and then selecting the File > Save As > MDS XML Documents… option. I select the folder that’s just been created by TortoiseSVN as my working copy folder, and then save the repository in the MDS XML format to this folder.

3. Then, if I want to upload this first source-controlled version of the repository to SVN, I can come out of the Oracle BI Administration tool and right-click on the folder, and select SVN Commit… from the TortoiseSVN shell menu.

As at this point, all of the folders and XML files will be unversioned, I need to add them to the working set, and then press OK.

TortoiseSVN will now upload the folders and files to the SVN repository, like this:

4. If I then check in the SVN repository, I’ll see my uploaded BI repository in this MDS XML format.

5. So, what the process that you go through if you subsequently want to check this repository out again, make some changes, and then upload the changes back to the source control system? This is where the source control integration in the Oracle BI Administration tool comes in. Let’s start by using TortoiseSVN to update our working copy of the SVN repository files, in case anyone else has committed and changes in the meantime.

Now, to open this repository under version control, I select File > Open > MDS XML… from the Oracle BI Adminstrator tool, enter the repository password, but then select Use Source Control rather than Standalone, and use the Browse… button to pick up the SVN configuration file I opened a while ago.

6. When I now go on to work with the repository, the title bar for the Oracle BI Administration tool has the message (under source control) next to the folder name where the MDS XML-format repository is saved, and it now keeps track of all the object add, delete and edit operations I perform whilst working with the repository.

7. Let’s now make some changes to the repository, deleting that Staff (Parent Child) logical dimension, and adding a new subject area. Now, when I save the repository, I’m asked first to check global consistency, and then a new dialog comes up confirming the source control actions that this will lead to. In this case, my changes have lead to five new MDS XML files being created, six being modifed, and two being deleted; information that the Oracle BI Adminstration tool will then pass to the Subversion client, via those command-line mappings we saw in the configuration file.

Once I press the OK button, the Source Control log dialog comes up, and shows me the progress of my files being synchronized to the working set in the folder I checked-out of the SVN repository.

Then, I press Close to close the dialog, and my repository is saved.

8. All that’s left to do now is to return to TortoiseSVN, and make the final commit back to the SVN repository (remember, all that the Oracle BI Administration tool has done use SVN commands to synchronize the list of changes you’ve made with SVN’s local record, it’s not committed the files back to the SVN repository). So, I right-click on the folder and select SVN Commit, and I can see from the Commit dialog that the SVN client knows about the files I’ve added, deleted and modified in the Oracle BI Administrator tool.

I then press OK and TortoiseSVN, the SVN client, updates the SVN repository with the changes.

9. If I then want to view the history of changes in SVN, I can use TortoiseSVN again to view the change log, where I can see the repository revisions and who’s made the changes.

So, there’s the basic process, and that’s what MDS XML is for. I can certainly think of lots of ways this could be improved, but it’s better than where were were before, and it’ll be interesting to see how this work with branching and merging repositories, or trying to take versions of objects from previous check-ins, or even other repositories, and trying to merge them together using these XML files. But the thing to bear in mind is that you’ll need access to a version control system such as Subversion, and a client such as TortoiseSVN, to make this all work, with the Oracle BI Administrator’s role being mainly to keep track of what you’ve changed, and relay that information to the version control tool before you try and check-in your updated repository.

Comments

  1. Christian Says:

    Hi Mark,

    Any idea where the Marketing definitions and especially the variables with their init blocks end up? As far as I can see they’re not in the MDS XML?

    Cheers,
    Christian

  2. Gaurav Misra Says:

    Good Content Thanks…

  3. Fiston Says:

    Great posting as always, thanks for sharing!

  4. Scott Powell Says:

    Hi Mark, just curious if you have any insight on where this is headed. For example, I’m guessing in the future that RPD files will be obsolete and replaced by tables holding XML within the obiee_biplatform database schema?And that the the admin tool will just be adding / changing / deleting XML entries that way?

    Thanks!
    Scott

  5. Sachin B Says:

    Hi Mark,

    What happens in case of more than one user checking stuff out simultaneously ?
    Does this approach handles changes the way MUDE does ?

  6. Mark Rittman Says:

    It doesn’t control the process around multi-user development – all this does is created versioned copies of the repository objects, it doesn’t handle merge conflicts, or stop one developer overwriting an object with their revision. Despite what the manuals say, I don’t think it’s an alternative to MUD.

  7. Mark Rittman Says:

    Christian

    Marketing definitions get created in their own sub-folders (SegCatalog, QualifyingKey, TargetLevel etc). So do init blocks (InitBlock folder).

    regards, Mark

  8. Stuart Wallace Says:

    Hi Mark,

    An interesting development for sure. I agree with you that it’s probably the most interesting addition to the product but then I remember a hellish Friday afternoon that turned into Friday evening that turned into Friday night at work back in 2004 on the phone to Siebel product development in the States trying to get MUD to work. Any advance on that has to be a winner.

    I note though what you say about it not being a replacement for MUD but, as chance would have it, the day this was released I’d just finished the chapter on Version Control in the superb book Agile Analytics by Ken Collier. Some of the tips in there about how this would work in an Agile team could potentially allow you to use this new source control functionality to replace the need for MUD. A recommended read in any event.

    Any word on whether Oracle will release an OEM’ed version of a preferred source control platform at some point? I know there are other BI vendors out there who ship with Subversion and that’s handily included in their install .exes.

    Cheers for now,

    Stuart

  9. Stewart Bryson Says:

    This is very good Mark. I’m already using it on an RPD I’m working on, and it all works great.

    The only thing I can’t figure out is why this wouldn’t completely replace MUD. Now that the file is broken up, and it is textual instead of binary, we can use the full compliment of diffing, merging, etc. When there are conflicts, we can work through them. However, perhaps that is the issue… the “conflicts” on the XML files will likely be incomprehensible and difficult to decipher. In all likelihood, we would need the visual OBIEE tools to show us what the real conflict is, and ask us what we want to do with it.

  10. Brian Says:

    Great explanation, thank you! In case it helps somebody using TortoiseSVN, the assumption in this example is that you included the command line client tools when you installed TortoiseSVN. Without the command line client tools, svn.exe is not included in the install!

  11. rejina Says:

    I have completed till the point 6 succefully.
    Afetr that i made some changes in the RPD and saved and clicked yes to proceed for the source control.But in the Source control log file i am getting the following error.please guide me.I have used default svn template for the configuaration file.what are the mandotory changes required in the template file.


    Admintool> Creating file oracle\bi\server\base\PresentationCatalog\29d9eb45-0c34-1000-806e-0ad4109a0000.xml

    > svn.exe add oracle\bi\server\base\PresentationCatalog\29d9eb45-0c34-1000-806e-0ad4109a0000.xml
    [nQSError: 46105] Error executing process: “svn.exe” add oracle\bi\server\base\PresentationCatalog\29d9eb45-0c34-1000-806e-0ad4109a0000.xml
    The system cannot find the file specified.”

    Thanks.

  12. prat Says:

    This does not work for us with the OBIA 7.9.6.3 repository. We converted the RPD to MDS XML and when we try and save the same, the tool throws errors during Global Consistency Check step. Raised this with Oracle Support and they have recommended that we wait till 11.1.1.7 release of OBIA.

  13. Juan Says:

    Rejina, you have to add the C:\Program Files (x86)\VisualSVN Server\bin at the beginning of your PATH environment variable. Remember to restart your BI Admin Tool to assure it can read the new PATH variable.

  14. Paul Says:

    Hi!
    I can make some changes like modifying or adding new elements in the RPD and save them. The source control log says:
    “Files have been synchronized to source control. Remember to commit
    changes after testing.”
    But when I delete an element in the RPD, the source control log says:
    “> svn.exe delete –force oracle\bi\server\base\PhysicalTable\aed72c00-0c64-1000-806e-c0a804d60000.xml
    [nQSError: 46105] Error executing process: “svn.exe” delete –force oracle\bi\server\base\PhysicalTable\aed72c00-0c64-1000-806e-c0a804d60000.xml
    The system cannot find the file specified.”
    There is only a user making changes.
    What’s wrong?

  15. Biconsultant Says:

    Try to paste the bin folder of the visual svn server which you will find where visual svn server is installed in the program files to the repository which is in the server.using this method when any client checkout on their system the full repository ie. rpd along with the bin folder is saved on their local system,so now adding the files is very easy it will not show the error “failed to deliver changes to source control/system cannot find the file specified”.

Website Design & Build: tymedia.co.uk