Installing obi-metrics-agent, Graphite, and collectl

March 27th, 2014 by

In the previous post I introduced obi-metrics-agent, an open-source tool from Rittman Mead that can collect DMS metrics from OBIEE and write them to several formats including graphite. Graphite is a tool for storing and graphing time-based metrics.

Now in this post we will look at how to install the tools and get them up and running on your server. The install process has been tested on both OL5 and OL6 (and so by extension, Cent OS and RHEL). There’s no reason why it oughtn’t work on other *nixes, but it’s not been tested.

I’m using Oracle’s SampleApp v309R2 server which is built on Oracle Linux (OL) 5 so that anyone can try this out at home running it on VirtualBox (or VMWare Fusion). I’ve also tested this on the latest SampleApp v406 – there is one additional step to run, see the paragraph below for details

There are three parts to the obi-metrics-agent installation.

  1. obi-metrics-agent – python script to extract DMS metrics from OBIEE
  2. [optional] Graphite – store and graph data, including that extracted from obi-metrics-agent and collectl
  3. [optional] collectl – OS metrics viewer. Can send data for graphing in Graphite

Only one of these is mandatory – the installation of obi-metrics-agent itself. That in itself is just a case of cloning the repository in order to get the obi-metrics-agent python script on the server. However, the remaining articles in this blog series will assume that graphite and collectl have also been installed, so these are covered too.

Assumptions & pre-requisites

  1. Server has internet access (try ping google.com to check)
  2. Installation is running as the oracle user
  3. Installation path is /home/oracle/graphite
  4. User has sudo rights

Before you run this on SampleApp v309R2 (not v406), you must first run the following to fix a bug in yum’s configuration. If you’re not running SampleApp, you don’t need to do this:

You need the EPEL yum repository setup (for packages such as git, etc):

  • OL 5/Cent OS 5/RHEL 5 (including SampleApp v309R2):
  • OL 6/Cent OS 6/RHEL 6:

Additional step for SampleApp v406

If you are installing this on Oracle’s SampleApp v406, run the following first:

If you don’t do this, there is a package conflict when dependencies for py-cairo are installed that causes the whole setup to fail. The root error you’ll find is:

Installation

The bulk of the installation is done by a couple of shell scripts. To get these, you should clone the git repository. The following steps will do this.

Now run the following to install git and clone the repository:

Then launch the installation:

  • OL5: (including SampleApp v309 and earlier)
  • OL6: (including SampleApp v406)

This will install git (if not present), and create a folder called obi-metrics-agent in the installation folder of OBIEE (FMW Home). You can put obi-metrics-agent wherever you want, this locating is just a suggestion.

The installation for Graphite builds a standalone python environment using virtualenv in which graphite will run under Python 2.6. The reason for this is that OL5 ships with Python 2.4 (Noah was seen coding in this shortly before boarding his ark) and the OS’ python version cannot be easily upgraded without causing all sorts of complications. Virtualenv is generally a good way to deploy python applications in isolation from others, so is also used for the OL6 version of the install.

The script will take a few minutes to run. Once complete, you can test the installation following the steps below. Do note that Graphite is somewhat notorious for installation problems, so whilst these instructions have been tested, you may hit quirks on your own server that may need a bit of Google-Fu to resolve. If you want to follow step-by-step instructions detailing each step, they are provided:

If you’re on a different *nix platform then feel free to adapt the install scripts and submit a pull request. Graphite works flawlessly on Debian-based distributions, and there is no reason why you should run it local to the OBIEE server on which obi-metrics-agent is running.

Testing the installation

Testing obi-metrics-agent

First, start up the OBIEE stack and make sure that you can login to OBIEE. One it is running, you can run obi-metrics-agent by entering:

Expected output:

For more advanced options, including sending data to graphite, see below.

Testing Graphite

Go to http://<server>/ and you should see the Graphite launch page:

2014-03-25_09-32-17

By default Graphite will collect data about its own performance, of the carbon agent. This means you can easily test that Graphite’s graphs are working by selecting one of the carbon metrics to graph. Expand the Metrics tree and click cpuUsage which will add it into the graph in the main pane:

2014-03-25_09-33-37

Graphite is installed with a default user oracle with password Password01.

Testing collectl

From the command line enter collectl:

(press Ctrl-C to exit)

To check that collectl is successfully sending data to Graphite go to Graphite and under the Metrics folder you should see your machine’s hostname (for example obieesample for SampleApp). Under that should be the metrics that collectl is sending to Graphite:

2014-03-25_09-45-10

Graphing OBIEE DMS metrics in Graphite

Now that we’ve gone through the installation, let us see how to use the three components together, graphing out some OBIEE DMS data in Graphite.

Run obi-metrics-agent.py as you did in the test above, but this time specify the output carbon and carbon-server parameters

Now go to Graphite at http://<server>/ and under Metrics you should see all of the OBIEE DMS metrics in folders. Note that it may take a couple of minutes for the metrics to first appear in Graphite. If you don’t see them, or the complete list, wait a minute and then hi

2014-03-25_12-05-19

(the DMS folder is metrics relating to DMS itself – the OBIEE DMS metrics are under OBI)

Find the metric for Presentation Services logged in sessions, Oracle_BI_PS_Sessions.Sessions_Logged_In and click on it to add it to the graph. By default the graph shows a large time period so click on the Select Recent Data icon

2014-03-25_12-08-13

Set the time period to 5 minutes, and click the Auto-Refresh button at the bottom of the graph.

Now log in to OBIEE, and go back to the Graphite page – you should see that the metric has increased accordingly.

2014-03-25_12-13-03

I’ll look at using Graphite in detail in a later blog post; this is just to demonstrate that the collection and rendering is working.

Different ways of using obi-metrics-agent

obi-metrics-agent has three modes of operation:

  1. Collect and parse metrics to output
    • Near-real-time rendering of OBI metrics in Graphite
    • Near-real-time load of data into Oracle (via CSV & external table)
  2. Collect metrics to disk only
    • For analysis at a later date
    • If graphite/carbon is not available to send data to
    • If outbound network bandwidth is constrained (or could be, by OBIEE)
    • For lowest host system overhead
  3. Parse existing metrics from disk
    • Parse previously collected data and output to Graphite or Oracle (via CSV & external table)

Syntax

For further details and syntax examples, see the USAGE.MD file in the github repository.

Tags: , , , , , , , , ,

Comments

  1. Andy Rocha Says:

    Robin,

    This is brilliant. Very well done.

  2. Mark Seger Says:

    Good stuff, but I’m a little confused about the collectl to graphite linkage as I don’t see the collectl option ‘–export graphite…’ and was wondering if this is magically configured by your setup.

    One of the reasons I ask is there’s a feature I included with the graphite connector that most people don’t take advantage of and that’s the ability for collectl to collect/record data at one frequency, say every 5 second or so, and send a subset of it to graphite at a different frenquency, primarily as a mechanism to avoid overwhelming graphite which you can do if you have a lot of machines trying to talk to it. Perhaps this is not an issue in this case.

    you might also want to talk a look at colmux, which allows you to display collectl output across a cluster machines in real-time, sorted by any column, sort of a ‘cluster top-for-anything’.

    -mark

  3. Guest Says:

    Hello, do we actually need the ‘oracle’ user to do this or the user created for obiee? I ask because we have ‘obiee’ as user and no access to the oracle user. Thank you.

  4. Norm Says:

    Hello,
    Thanks for creating and sharing this tool.
    Could you advise on the changes that would be required if the user was not ‘oracle’.
    Thanks,
    Norm

  5. Robin Moffatt Says:

    Hi Mark,
    Thanks for the comment. You’re right, the collectl->graphite config is done automagically.
    I’ll have a look at the other two options you mention.
    Thanks for writing such a great tool, I wouldn’t be without it!
    cheers, Robin.

Website Design & Build: tymedia.co.uk