Creating an Azure VHD from Ubuntu Cloud Images on Mac OS X

We have many different ways of delivering the Lateral API to clients who would like to install it in their own environment. One of those is as an Azure VHD for deployment to Azure VMs. In this post I will cover how to create a VHD that is fully compatible with Azure from an Ubuntu Cloud Image base.

I found the Azure documentation to be a bit outdated. In fact in various places it didn’t work at all. So I thought I’d try to create a version that at least works for Mac OS X users. Well, that works until it itself becomes outdated and stops working. At which point I or hopefully someone else will take it upon themselves to write an updated version.

There were also quite a few gotchas that I will explain in due course. One that deserves an honorable mention is the fact that the size of the Azure VHD must be aligned to the nearest whole MB otherwise the image will be rejected after upload. I mean, seriously Microsoft??

Dependencies

There are several dependencies required before we can start:

  • VirtualBox with VBoxManage and VBoxHeadless on your path
  • Go for the azure-vhd-utils-for-go
  • Azure CLI for interacting with Azure from the command line
  • qemu for qemu-img used in create-vhd script to resize and convert images
  • coreutils for gtruncate used by the create-vm script
  • cdrtools for mkisofs used by the create-vm script
  • gawk used by the create-vhd script

You can use Homebrew to install the last 4 with the following command:

brew install qemu coreutils cdrtools gawk

Next you need two scripts; create-vm and create-vhd. The create-vm script was modified to work with OS X from the example written by Scott Moser. Thanks Scott! Have a read through the comments of the two scripts to get an idea of what they do. Then download them to a location of your choosing and ensure they are executable:

Creating the VM

Use the create-vm script to create a VM in VirtualBox from the Ubuntu 14.04 image:

./create-vm my-trusty-vm

If you want to use a version of Ubuntu other than 14.04, just modify lines 8-9 of create-vm to point to the version you want.

Once that has completed, you can start the VM in headless mode:

VBoxHeadless --startvm my-trusty-vm

Now you can SSH into the instance from port 2222:

ssh -p 2222 -A ubuntu@localhost

The password is set at line 35 of create-vm to ubuntu. We strongly recommend changing this to something secure or even better setting up key based authentication.

When you’re in the instance there’s a bit of setup required for it to work with Azure. See here for some more details about what’s being done here:

sudo apt-get update
sudo apt-get install -y linux-image-virtual-lts-xenial linux-lts-vivid-tools-common hv-kvp-daemon-init
sudo reboot

After the system reboots, edit /etc/default/grub in a text editor, find the variable called GRUB_CMDLINE_LINUX_DEFAULT and change to:

GRUB_CMDLINE_LINUX_DEFAULT="console=tty1 console=ttyS0,115200n8 earlyprintk=ttyS0,115200 rootdelay=300"

Then update grub:

sudo update-grub

And install the azure agent:

sudo apt-get update
sudo apt-get install walinuxagent

Now that’s all installed you can do whatever you need to get the VM up and running how you like it. Install packages, clone your code to the VM and set it up to start at boot, etc.

Preparing the VM for deployment

This step is important to get the system running successfully on Azure. When the instance is in a good state and you don’t want to make any more changes, run the following:

sudo waagent -force -deprovision
history -cw
logout

And safely shutdown the VM:

VBoxManage controlvm my-trusty-vm poweroff soft

Now you can use the create-vhd script to create an Azure VHD from the VDI. Note this will require about 60 GB free disk space:

./create-vhd my-trusty-vm

This will take a little while to complete. The script takes care of the resizing of the VHD to the nearest MB as required by Azure so you don’t have to worry about that!

Uploading the VHD

Once completed we are ready to upload the VHD to Azure. The documentation from Microsoft says to use the azure-cli to upload the image. However, trying to use it caused this error for me. This response from March 2016 basically says it’s being looked in to and in the meantime we should use the azure-vhd-utils. Which does work! So to install that run:

go get github.com/Microsoft/azure-vhd-utils-for-go

And then you can run the tool to upload the image:

azure-vhd-utils-for-go upload --localvhdpath my-trusty-vm.vhd --stgaccountname NAME --stgaccountkey KEY --blobname my-trusty-vm

Replacing NAME with the storage account name and KEY with the storage account key. You can get these from the Azure Portal. azure-vhd-utils-for-go is good because it doesn’t upload empty space. So if the VHD is 30 GB on your disk it will only upload the parts of the image that have data in. This speeds up the upload process greatly.

Now you have to create an image from the Azure VHD stored in blob storage:

azure vm image create --blob-url VHD_URL --os Linux my-trusty-vm ''

Replacing VHD_URL with the URL to the Azure VHD in the storage account. You can get the full URL to the VHD by navigating to it using the Microsoft Azure Storage Explorer, right clicking the file and choosing “Copy URL to Clipboard”. The '' is required because the CLI is expecting a file to be uploaded there. If you leave it as an empty string it works and takes the VHD from the storage account.

Lastly, you can create a VM from the image:

azure vm create --userName ubuntu --password 'Pas$word1' --vm-size "Standard_A4_v2" --ssh --location "West Europe" "MyTrustyVm" my-trusty-vm

Once that is completed, you’re done! You can SSH into the instance that was created from the Azure VHD now. Note that the password specified in the above command is required but doesn’t actually do anything. The password to the VM was set earlier by the create-vm script to ubuntu. Important: Enable key based authentication now otherwise you have a very insecure instance running!

Conclusion

I hope that this post helps people in the future who like me found the Azure documentation to be a bit broken and confusing.

References

The Author
Making stuff with computers at Lateral.
Comments