Inside My Home Office Development Lab (VMWare, OS X Server)

August 3rd, 2013 by

A little bit off-topic for once, but I thought one or two readers might be interested in the setup I’ve got back at my home office, for testing out and developing stuff around Oracle BI. Like most Oracle tech enthusiasts, I’ve been a long-time user of tools such as VMWare and Virtualbox for desktop virtualisation, but over the past couple of years the scope and scale of Oracle Fusion Middleware has meant that a bit more of an “enterprise” approach had to be taken in my test lab. In addition, we’re an Apple-centric family back home, with iMacs in most rooms, iPads and iPhones everywhere, so I thought it’d be a good opportunity to try out OS X Server as well, to see if I could add a bit of management to the whole network, and introduce things such as user and hardware policies, DNS and centralised software update, that sort of thing – as most people do on their home networks … (!)

Let’s go to virtualisation first. Over the past few years I’d been buying a number of Mac Mini servers, each with 16GB RAM and managed centrally through Apple Remote Desktop, as shown in the screenshot below. These worked well, went in a cupboard under the stairs and ran nicely “headless”, with desktop access again being through Remote Desktop, or VNC. Remote Desktop can also manage VMs that run VNC as well, which comes as standard with Red Hat / Oracle Linux, or can be installed separately into Windows-based environments.

NewImage

This worked well – and had the benefit of isolating one or two VMs on each Mac Mini, so performance issues didn’t hit other VMs – but its an inefficient way to run VMs. So, inspired by Steve Karam’s excellent “Make your own VM training/lab environment for $900″ blog post, I first built a VM Server with 32GB RAM and a bunch of SSDs and HDDs, then built a second server with 64GB of RAM and even more disks. Then, initially with the free version of VMWare’s ESXi hypervisor, and then with the full VSphere setup courtesy of VMWare’s “Guru Program”, I put together a two-node cluster of VMWare ESXi hosts managed through VMWare VCenter, that can host upwards of 20 or so VMs as part of a single managed server pool. The screenshot below shows this setup in action, and you can see the various VMs under the two hosts (servers), with various options to migrate them, manage them and so on.

NewImage

Part of the driver for getting VMWare, in particular, working at home was that many of our customers use it for development, DR and so forth, and I wanted to test out VMWare’s VMotion feature as an alternative to OBIEE’s WebLogic-based clustering. VMotion (other virtualisation vendors have similar technologies) allows VMs running on one particular server, but with their files stored on shared storage, to automatically restart on another server if the first one goes down. It’s not quite the same as clustering – each machine is independent, and in this example only one runs at a time – but if a customer is just looking for a HA solution, and they already make heavy use of VMWare, its an interesting alternative that I want to try and model in my own lab if possible. I did also look at Oracle VM, but initially on the UEFI-based hardware I first used, I couldn’t get it to boot, and to be honest the VMWare suite is very easy to use, very “industry standard”, with the only real drawback being the requirement for a Windows server to run the VCenter management software.

Apart from the two servers in an ESXi VSphere cluster, I’ve also got a software (FreeNAS) NAS actually running on a VM in that cluster, but i’m phasing that out in favour of a dedicated hardware NAS, a Synology DS413, as the functionality of a NAS VM isn’t quite up there with that in a dedicated unit; it’s running on the same machine as the VMs, taking up their disk space, and using the Synology I can also set up RAID on the four disks to protect against disk failures. I’ve been very pleased with the Synology so far, and I’ve now got around 10TB of NAS storage on the network, accessible from all the Macs in the house.

We’re actually a Mac-centric household here, with iMacs in the kids rooms, a 32GB iMac in my office, a few laptops lying around, and a bunch of Apple TVs and Airport Expresses so we can beam music and video around the house. About six months ago I thought I’d have a play around with Mac OS X Server to see if I could set up a central server for the house, to enable me to manage and maintain all the iMacs as well as test out features such as profiles and centralised backup that I’d been considering for the Rittman Mead office.

OS X Server is now set up with Open Directory (Apple’s LDAP directory, their equivalent to Microsoft AD) and with each iMac having a network logon, with the kids files stored on the server and the ability for anyone to log into any machine in the house, using their central ID. DNS is set up so that all the iMacs, VMs, servers and so forth have proper fully-qualified domain names, and we’ve also set up a central Software Update cache, file sharing and the like. The screenshot below is from the OS X Server “Server” app, and you can see the various network services provided by OS X Server.

NewImage

There’s also a few Mac Mini servers scattered around the house – one hosts OS X Server, another runs VMs prior to importing them into ESXi, etc, so that the home office network extending out to the rest of the house looks like this:

NewImage

VMWare ESXi, VSphere (the overall platform) and VCenter (the management server, used when there’s >1 ESXi host you want to run services on) is the most impressive part of the setup, though. As ESXi is a bare-metal hypervisor (like Oracle’s own Oracle VM), there’s little of the OS overhead that you get when running VMs on a regular server or desktop, plus you can take advantage of features like VMotion (for high-availability of VMs when their files are stored on shared storage), host-to-host migration and so on. In the two screenshots of VMWare VCenter Console below, you can see on the left, the set of VMs currently assigned to the larger of the two servers, with the graphs down the right-hand side showing the profile of CPU and RAM usage; whilst on the right, details of one of the VMs, in this case running Oracle Linux and E-Business Suite 12.1.3.

NewImage

What this means in-efffect is that I’ve got pretty-much every bit of BI-related Fusion Middleware software either running, or suspended but ready-to-run, making it a lot easier to put examples together, try out some integration technique, or refer back to an older version of the product if needed. In addition, as mentioned above there’s an EBS 12.1.3 instance running, a VM running Hadoop, Hive and there rest of Oracle’s big data stack, as well as various Oracle demos and beta releases.

And that’s a lot of Oracle technology to manage – so, as we recommend to our customers, I’ve got another VM this time running Oracle Enterprise Manager 12cR2, linked-in to most of the VMs and managing and monitoring them in various groups.

NewImage

I’m about to upgrade EM to 12cR2 to make use of its new ability to manage an Exalytics server as a whole unit, but for now I use it to check the status of the various VMs, teach myself EM Cloud Control, and apply patches to the databases and other non-Bi components.

NewImage

Finally, to access the various VMs, I either VNC into the Linux-based ones using Apple Remote Desktop, or use Microsoft Remote Desktop Connection to access the Windows ones. For the various OBIEE environments, Safari bookmarks synced via iCloud are the main way in, with a VPN server in the home router to provide access from the outside (I did try using the VPN Server in OS X Server, but found it too flakey).

NewImage

So there we go – a bit geeky I know, and probably a bit over-the-top for most home development labs – but if you’re interested in more details on how I set it up, or you’ve done something similar with other technology, Oracle VM or Open Stack for example – drop me a line or add a comment to the post.

Comments

  1. Geordee Says:

    Really impressive. 26 Computers, virtual and real.

  2. Peter Says:

    Hi,

    Thats a pretty nice setup.
    Any chance of some HW-details regarding the esx hosts? I’m guessing both are ‘whiteboxes’?

    As for vCenter, there is an appliance that runs Suse/postgres that can be used instead of the windows/sql-server setup. I’m using that and it runs perfectly.

    regards
    /P

  3. Saichand Says:

    Excellent setup!

  4. Rus Says:

    Interesting. Have you got any other types of monitoring/graphing in your set up, or just Oracle Enterprise Manager? We’re running ESXI here and mostly use Munin/Nagios but can’t find any hooks in to ESXI to let us graph anything, and VSphere isn’t exactly historical.

    Cheers

  5. Mark Rittman Says:

    @Geordee – there’s a lot more than that, that’s just the list of VMs etc that are registered with Remote Desktop. It also generally didn’t list the Windows-based ones, as you need to specifically install VNC Server / Client on them to remote into them via this method.

    @Peter – the 32GB machine is as per the article by Steve Karam. The 64GB machine was a bit different, as there aren’t many motherboards that support 64GB of RAM and a “consumer” CPU (as opposed to Xeons, etc), so the main components for that machine were:

    - Asus P9X79 Deluxe Motherboard (Socket 2011, Intel X79, DDR3, ATX, Bluetooth, Dual Intelligent Processors 3, DIGI+)
    - Corsair Carbide Series 300R CC-9011014-WW Mid-Tower Gaming Case
    - Intel Core i7 3820 Quad Core CPU (3.60GHz, 10MB Cache, Socket 2011, 130W, Sandy Bridge)
    - Corsair Builder Series CX 500 Watt ATX/EPS 80 PLUS Bronze Power Supply Unit
    - Corsair CMZ64GX3M8A1866C9 64GB 1866MHz CL9 DDR3 Vengeance Memory Kit – Black (8 x 8GB DIMM)
    - Crucial CT064M4SSD2 64GB M4 SATA III 6Gb/s MLC 2.5 Inch Internal SSD
    - 3 of Western Digital Caviar 3TB SATA 6 GB/s 64MB Cache 3.5 inch OEM Internal Hard Drive Green
    - Corsair CWCH60 Hydro Series H60 High Performance 120mm Radiator All-In-One Liquid CPU Cooler

    @Rus -only OEM this point, but we use Nagios in the RM office dev setup, Robin Moffatt has written several articles for the blog on this topic.

Website Design & Build: tymedia.co.uk