vmware | Steve Dolphin

Quick: vmware tools on freebsd

I’ve never bothered installing vmware tools on my freebsd systems, but as it’s Christmas and we’re doing various pieces of maintenance I thought, “why not?”

In order to do this I first needed to install the compat6x port:

cd /usr/ports/misc/compat6x/ && make install clean

I then mounted the freebsd.iso from the vmimages folder, mounted the cdrom into existing folder (/cdrom) and copied the folder to the temporary directory before extracting and installing:

mount -t cd9660 /dev/acd0 /cdrom cp /cdrom/vmware-freebsd-tools.tar.gz /tmp tar -xf /tmp/vmware-freebsd-tools.tar.gz ./vmware-tools-distrib/vmware-install.pl

Upon completion the installer then runs the first time required config to set up the installation.

Enlarge 0 Comment

Installation of Dell EqualLogic MEM 1.0.9 (1.1 EPA) and vSphere 5

So the 1.0.9 version of Dell’s MEM module (ug, that’s as bad as PIN number), scratch that, of Dell’s Multipathing Extension Module for EqualLogic was released last week. I promptly ignored it for a week before reading the line that said “fully supported in production environments” (whoops) and installed it today. It wasn’t easy.

I’ve already upgraded vSphere Centre to version 5, and so I made an install package for MEM 1.0.9 (1.1EPA) – Update Manager then told me that it wasn’t required anywhere, “that can’t be right”, I thought to myself, and then realised that I needed ESXi 5.0 to install MEM; but you can’t install ESXi 5.0 while MEM1.0 (or anything pre 1.0.9) is installed. A little bit of a catch 22.

I then embarked on a whirlwind tour of command line chaos, using PowerShell to create my own installation .iso which combined the ESXi 5.0 install with the Dell MEM bundle, I uploaded that to the Update Manager and used it to upgrade two hosts, and I’m shocked to say that it worked almost perfectly, so, for the simple commands you need to use, see below!

To do all this you’ll need to download and install the vSphere PowerCLI.
Continue Reading →

Quick: vCMA (vCenter Mobile Access Server)

If you’ve not seen it yet, VMware have have flung out a product called vCMA – it allows you to manage your VMware estate from a mobile device, and although only in first release and officially a “power toy”, it’s pretty neat.

What’s even more neater* is that you also use the free iPad application to connect and then administer remotely with a shiny interface (no, you can’t do it on the iPhone I’m afraid).

Anyway, the point of this was to note the default options, after downloading the OVF file and installing it,

the default URL is https://ip-address/vim
the management URL is https://ip-address:5480
the management username/password by default is root/vmware

*yeah, why not?

Using the Dell EQL MEM Module to simplify my backups (also, thanks again, veeam!)

Many posts cover the installation and performance benefits that come from using the Dell Multipathing Extension Module (MEM) on EqualLogic arrays (check the spoonapedia.com one), but the big difference for me was a bit of a pleasant side-effect in terms of handling backups! I’ve covered off this strategy from a high level in my previous blog post, Backup Strategies with Virtual Machines in VMware using Veeam, but I wanted to explain in a little bit of detail how I actually got there: it was down to the MEM!

Before – accessing data from within the OS

Because the file servers I’ve been working on access a lot of data (8TB worth), the original setup involved using the EqualLogic Host Integration Tools (HIT kit) from within the file server OS to access LUNs on the EQL array – this provided valuable multipath access and proved to be a very successful way of handling access to the data. The problem was that is complicated backups quite significantly, I could use veeam to backup the OSs (and I did), but I had no way of backing up the actual file data.

With various bad experiences from using market-leading backup software such as BackupExec in the past I wasn’t in a rush to go out and spend money on a software solution to handle all this file data, so I resorted to a very low-tech solution: I bought a nas box and did a nightly robocopy.

This was simple, but it was awful: the backups didn’t finish in time (they’re being taken over a 100mbit LES), they never caught up with themselves. It was a waste of time and it basically meant no backups were worth having.

After – install the MEM and let ESXi deal with it

But then the MEM came out, and essentially claimed to offer the same (if not better) performance via ESXi – no messing around with the HIT kit any more, and more importantly, a chance to re-evaluate my first decision about not using .vmdks… I changed my mind.

The first time round I did thick LUN straight into Windows, formatted as NTFS, simple. This time I re-evaluated and did a thick LUN on the EQL and then allocated thin disks in ESXi and mounted them to the file servers… This gives me greater flexibility if a disk gets close to its limits but it also now means that the extra .vmdks are picked up by veeam allowing me to replicate my previously successful backup strategy.

In summary…

Veeam now handles the file data as incremental .vmdks which means it only transfers the changes in the .vmdk files – the entire series of backups finishes over the 100mb LES in about 12 hours (which, bearing in mind I run it once a week at the weekend is ideal); the previous robocopy never finished in that amount of time: the size of the data transferred is obviously the same, but because robocopy iterated through every single file and folder for a comparison it took much longer whereas now veeam just… does it, and it was a product that I already had so didn’t require any extra spend (not to mention the money that could now be saved on not upgrading the LES to 1GB purely for the purposes of backup).

Next?

Now being quite satisfied with this setup I’m going to investigate the series of advice from ErikZandboer on optimising his ix2-200 backup speeds, specifically the post that looks at jumbo frames to target storage.

Backup Strategies with Virtual Machines in VMware using Veeam

A recent tweet from @win2ksrv and then retweeted by @veeam reminded me that I was going to write about the most recent backup strategy that I’d put in place using Dell EqualLogic SANs, VMware and of course, veeam, it went like this:

What is everyone’s Veeam backup strategy? What do you backup to & how do you get it offsite? Where do you place Veeam itself?

The basic setup looks like this:

The Production site has a Dell EqualLogic array and a local NAS box (that’s the black thing), the backup site which is connected via a 25mbps internet-based VPN simply has a larger NAS box (it deals with multiple sites). VMware has been used to create all the file server disks (they are .vmdks) and veeam is installed in another virtual machine using appliance mode to access the SAN.

There are essentially two main risks that we want to mitigate against here:

Accidental deletion / corruption of files
Complete site wipeout (i.e. full blown disaster)

Continue Reading →

Quick: Installing vCentre Update Manager 4.1 on a 64-bit OS

I had an “interesting” day today rooting round the VMWare Knowledge Base articles. It started off simply: I wanted to move my local SQL Express Edition vCentre database onto the much larger, more powerful, centralised SQL server. The KB that describes this, Moving the VirtualCenter SQL database, makes the process simple and flawless, until…

I go to uninstall the local instance of SQL Server and notice there are two, a 32-bit installation and a separate 64-bit version, quickly writing this off as a stupid glitch I uninstall both only to realise why there were two: for some reason Update Manager 4.1 (which can be installed on a 64-bit OS) needs a 32-bit DSN, a 32-bit database connection, to work. It achieves this by default by installing a local copy of 32-bit SQL Express Edition.

Some of you will already be familiar with the joy of using the 32-bit ODBC driver in Windows 7, and this is no different, but the KB, Creating a 32bit DSN on a 64bit Windows machine explains it all for those of you that haven’t.

Finally I was in a position to open up the UI again – but wait, another error message, what now?! A tiny bit more digging pointed me towards the first fix in Enabling Update Manager fails with the error: database unavailable or has network problems, which advises changing the account that the update service starts with to an account with administrator rights.

Once that was done it worked perfectly, so there you have it, a cleverly disguised list of links to VMware KB articles that I had to use when installing update manager 4.1, or more specifically, Moving the SQL databases for vCentre 4.1 when update manager 4.1 was already installed.

Enlarge 0 Comment

New office success!

A fairly unique opportunity arose for me towards the end of 2010: I was able to design, purchase components for, and then oversee the implementation of a new office infrastructure. This was due to a new physical build, and based on the success of the previous SAN and Infrastrucure project, I was essentially given full control over all aspects of the IT infrastructure.

It was an extensive piece of work and covered about six months of my life before finally going live at the end of November 2010, it involved visits to the site before building works were completed; setup of an entire dedicated test lab; and various correspondances thrasing out the business requirements for the new build.

The actual move-and-go-live happened over a weekend, the staff downed tools on Friday, had their PCs moved over the weekend and were able to log on and start work again Monday morning, a huge achievement bearing in mind that the entire system had changed: everything from the cables to the printers to the back end storage was now completely different.
Continue Reading →

Quick: Installing APC Network Shutdown (PCNS) on VMWare ESXi 4.1

This is actually documented in APC Answer ID 11144 but I couldn’t find the link without actually e-mailling APC and asking for it, so hopefully by posting it here with a more sensible title others might stumble across it.

Since ESXi 4.1 was released the automatic-shutdown of hosts using PowerChute Network Shutdown (PCNS) no longer works and needs patching, to do this you must firstly completely uninstall the existing installation of network shutdown by running the uninstall script, mine is located here:

sudo /opt/APC/PowerChute/uninstall

Download the new version from the APC website (the username and password is embedded on that page, but at time of writing it’s ftp://apcftp.apc.com and then, despite what the instructions say, I didn’t rename install_en.sh to install.sh I just ran it:

Download the above file and extract it, giving you five (5) files including the README.txt
Copy these files into /tmp along with the original installation files
Run sudo /tmp/install_en.sh
Follow the on-screen instructions
As per the APC Answer file, copy the shutdown and uninstall commands over, mine looked like this: cp /tmp/shutdown /opt/APC/PowerChute/group1/bin/shutdown and cp /tmp/uninstall /opt/APC/PowerChute/uninstall

(Screenshot showing list of files before installation)

This felt quite a bit better to me, it actually requested an IP for the ESXi host it was supposed to be controlling as well as registering with the management interface on the UPS correctly – this is more than I got when I used the version for ESXi 4.0 – I have tested this by issuing PowerChute shutdown command from the UPS and both of my ESXi boxes shut down correctly.

Remember that you’ll need to have configured the Low Battery Duration to at least 5 minutes under the UPS web interface for this to be effective and worth having.

A template for new offices

Now that the first and largest implemtation is almost complete, I have to start thinking about producing an easy-to-understand template that allows us to build site offices in a practically identical manner. One of the main goals of this project was to be able to have a repeatable template of systems and processes that allowed us almost dump a pre-fabricated solution into a physical building, this sort of template would be tried and tested and be the first step to producing a business service of “deploy new office” – a service which until now has been very ad-hoc and piecemeal.

From a very high level there a few things you need to deploy IT systems into a new building:

Connectivity (Telephony and Data)
Hardware (Servers, Cabling, Computers)
Software (Client OSs, Server OSs, Applications)

Our template includes a base for things already which has come as a direct result of building up the first office: We know that a new site wants Dell EqualLogic storage so that it can use the inbuilt replication between sites (for backup etc); two servers for redundancy; an uninterruptable power supply and environmental monitoring. We know that the site needs at least 10mb of internet connectivity, but keeping in line with recent work this will actually start at 25mb, and we know that there is a need for phone lines (for staff to make calls, as well as for alarms etc). In terms of software, the Client OSs are all still going to be Windows XP, (but with a view to upgrade to 7 later on throughout) and the server OSs are going to be Windows Server 2008 R2 – the datacentre licensing here becomes a bit of a no-brainer, which I’ll explain in a follow up post.

So that’s it, that’s the start of our template for a new site office! I’ll explore the three high-level requirements in more detail in future posts.

Backup diagram using veeam and iomega nas

Enlarge 0 Comment

Using veeam to backup the new virtual infrastructure to Iomega NAS boxes

I think it’s safe to say that veeam have been fairly “market disruptive” with their backup software, they’ve joined in on the virtualisation game at the right time and they’re offering a simple product that does exactly what people want: backing up vmware virtual machines.

To major things have lead to this success in my opinion, firstly it’s not very expensive, it works out to be, give or take, £500 per physical socket. So if you’ve got two 2xCPU servers like R710s and you buy some cheap storage you can get your virtual environment backed up from anywhere between £3,000 and £5,000 – which isn’t really that unreasonable.

A virtualisation company based in Bristol, Computerworld, recommended the Iomega StorCentre Pro range as being suitable dumps for this sort of backup, and I settled on a simple 1U device offering just over 5TB of data when using the built in raid (8TB raw), it cost just under £2,000. Update: don’t buy these if you have large backup files! It was so cheap that I decided to buy two, and then use a staged backup system which would handle our file servers as well as the veeam server backups, it looks a little like this:

Firstly the veeam server queries the vCentre server (vSphere in the diagram) to get a list of virtual machines
veeam then drills down into the SAN to access the data directly for a speedy backup
this data is then written to a NAS box which is plugged into the same switch, for speed
Periodically data is then taken to the off-site backup location. In my diagram the two are linked by 100mb LES, but this could be over a WAN link or even a different room in the same building.
The data is written to a NAS box in the off-site location too
The Windows machine in the off-site location then performs backups to tapes using the local copy of data, rather than the live copy

Staged backups aren’t new by any means, but they’re still a handy way of ensuring that if your backup to tape DOES take more than a few hours overnight that you don’t affect your production systems.

I haven’t actually implemented the tape drive portion of this diagram yet (6) but it’s going to be ordered pretty soon.

All in all with this system now in place for just over a week my virtual machine backups only take 2 hours to backup 22 machines, I can definitely live with that. I’ve also added a spare ESXi install to the veeam console so that I can test restoring individual machines to it, it seems to work pretty well and the user interface is intuitive enough that anyone could do it in an emergency.

In this particular instance I haven’t yet implemented a full business continuity plan, but when I do I know that veeam will come into play again due to its ability to replicate virtual machines from one ESX(i) host to another and perform failovers when asked. All in all I’m very happy with the purchase!