Why We Do This

The default image provided as the RHEL7 image from Azure Marketplace is woefully out of date. The kernel is 3.x and has some critical CIFS bugs which will cause issues with Azure Files. A number of critical sub-systems are out of date and need to be upgraded.

What Could Go BANG!

The kernel is NOT RedHat and so you may get push back from RedHat on support for this kernel if you start hitting edge cases. However, we use the cloud. If you’re tickling kernel bugs and no one else on the Internet is – you’re doing something horribly wrong. Rebuild.

3rd party software support may also push back on non-standard kernel. Push back on the push back!

Safety First

If in doubt, please backup your VM before you do this. If this fails it’s VERY hard to get the machine back into a production state.

How we do this

Update the base platform from the RedHat Repo

  • Login and sudo su to root.
  • Change to the \ directory
  • Update yum with yum update && yum upgrade
  • Accept the defaults and allow the update to happen – this may take some time, do something else in the meantime.
  • Install yum install yum-utils and then run package-cleanup --oldkernels

Upgrade the kernel from elrepo

  • Import the ELRepo public key with rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
  • Install the RHEL7 repo with rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
  • Install the new kernel with yum --enablerepo=elrepo-kernel install kernel-ml
  • Check which menuentry is the new kernel with awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg and make a note of that number
  • Set that number as the default kernel with grub2-set-default <<NUMBER>>
  • Regenerate grub with grub2-mkconfig -o /boot/grub2/grub.cfg

Update any encryption patches and dracut

  • Create a file called yumupdatefix.sh in /root with the following contents
#!/usr/bin/env bash

# get and test path to source of most recent install
unset -v ADE_SOURCE
ADE_SEARCH=/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux*
for i in $ADE_SEARCH; do
  if [[ -d "$i" ]]; then
    [[ "$i" -nt $ADE_SOURCE ]] && ADE_SOURCE=$i
  fi
done

if [[ ! -d "${ADE_SOURCE}" ]]; then
  echo "patch failed - no source directory found matching ${ADE_SEARCH}"
  exit 1
fi

# get and test path to patch file
ADE_PATCH=${ADE_SOURCE}/main/oscrypto/rhel_72/encryptpatches/rhel_72_dracut.patch
if [[ ! -f "${ADE_PATCH}" ]]; then
  echo "patch failed - no patch file found matching ${ADE_PATCH}"
fi

# replace string {Encrypted_root_partition} by partition number (=2 in rhel azure gallery images)
sed -i.bak s/ENCRYPTED_DISK_PARTITION/2/ "${ADE_PATCH}"

# patch and run
bash -c "set -e; patch -b -d /usr/lib/dracut/modules.d/90crypt -p1 < ${ADE_PATCH}"
/usr/sbin/dracut -I ntfs-3g -f -v --kver `grubby --default-kernel | sed 's|/boot/vmlinuz-||g'`

The above script will generate a new initramfs image corresponding to the new kernel version with the patched up version of Dracut modules – getting warnings is FINE. Really.

  • Enable execution of this script with chmod a+x yumupdatefix.sh
  • Execute this script with ./yumupdatefix.sh
  • Ensure that the image it builds is the same version of the kernel-ml that was installed.
  • Reboot with reboot
  • Pray or even Prey
  • When the machine comes back up, it should take a few minutes, check the new kernel with uname -msr
  • Profit.

References

http://elrepo.org/tiki/tiki-index.php
http://elrepo.org/tiki/kernel-ml
https://blogs.msdn.microsoft.com/azuresecurity/2017/07/13/applying-updates-to-a-encrypted-azure-iaas-red-hat-vm-using-yum-update/
https://gist.github.com/mayank88mahajan/38faf934c86b89ad766c4c16dcd5f4aa

 


Attempting to look like I know what I'm talking about and giving a subtle shout out to Joyent.
Attempting to look like I know what I’m talking about and giving a subtle shout out to Joyent.

Presenting at VelocityConf 2013 London was an experience. My thanks to them for allowing me the opportunity to reach so many really bright people. I also go to meet some of the people that make cfengine happen!

cfengine ninjas with Nakarin Phooripoom and Bishwa Shrestha
cfengine ninjas with Nakarin Phooripoom and Bishwa Shrestha

I didn’t go through the practical material that I wanted to do – 90 minutes wasn’t really long enough for something as deep as CFengine. I will be doing the practicals separately and uploading to YouTube in the next few weeks when I have the time.

For now, please enjoy my slide deck. As always comments via e-mail or Twitter please.


The Cloud is a way of thinkingfeeling and implementing  platforms and software that enable us to move past the concerns of unitary machines and into those of autonomic services.

There are many blogs, books, talks and presentations which define ways to build a ‘cloud’ architecture for your services. Sadly most concentrate on how to work within one framework or another built by corporations trying to tie you to their way of thinking.

The cloud has always been much more than this. I speak of course of RackSpace, Amazon and any number of other ‘cloud’ providers. Put simply:

Neither virtualization nor tooling defines the Cloud

The plethora of tools and tooling frameworks which surround these group of companies is immense. All of them designed to fix the woeful inadequacies of the underlying platform when dealing with the Cloud idea.  These companies and their products have their place in the world and many a business owes them their existence and prosperity. However:

The general acceptance of a method of execution does not define the success of understanding of an idea.

These companies do not offer you access to a real cloud. They offer large-scale virtualization technologies which mimic what a cloud should do – it’s a bit like putting an after market exhaust and a dump valve on an on old Nissan. Sure it sounds good but you really won’t pull away from anything very fast at all.

The speed, ease, reliability and flexibility of deployment, scaling, monitoring and operation within these platforms is inadequate.

The idea isn’t to scale in 30 minutes. It’s to allow the machines to scale themselves within context and constraint within minutes of understanding the need to do so.

The idea isn’t for a humans or machines to watch simple statistics to aid decision-making. It’s to allow your services to gain a contextual understanding of each constituent part’s operation and allow the convergent intelligence inherent to make real-time decisions.

The idea isn’t to re-invent the wheel when it comes to deployment but to leverage decades of experience to make sure we move past this triviality and tackle the hard problems.

That’s impossible in most current ‘cloud’ platforms unless dealing with the most trivial of services.  The last decade of my professional life has been in one ‘cloud’ environment or another. Uniformly, all have failed to show any glimmer of real understanding of the Cloud idea. Till last year, but more on that later.

This is the small, gentle introduction to what will be a series showing you how to build a real cloud services architecture.

You’ll need a GitHub account, a Joyent public cloud account, a local CFEngine installation and some understanding of Python and Bash.

It will be fun, I hope you’ll join me .

Follow me (@khushil) or bookmark this blog to get each installment as it arrives. Click on the ‘About Me‘ page to find out how to get in touch with me.