the meat grinder


Having just been on a CAB call with over 60 people running through a list of over 400 items to work through I’m reminded why I try really hard not to work in places where these things happen.

When you work in a truly agile workflow we don’t need these. We don’t need these because a good agile workflow can fully replace a traditional CAB if the agile workflow is cross-disciplined throughout the business.

At the end of several hours, everyone who was speaking sounded dispirited and thoroughly pissed off – including the leader on the call. That sucks. If you’re the leader and the meeting is depressing you – imagine what everyone else feels like.

Seriously – stop having CABs but if you really have to have one these points may help you:

  • Circulate the CAB items early and anything with a LOW to NONE impact rating shouldn’t be discussed – they should be automatically approved unless someone wants to call out one of them during the CAB.
  • Don’t have CABs that last more than an hour at the very most.
  • Group your changes by impacted areas so you can release people quickly.
  • Don’t speak over someone when they’re speaking – especially if you’re leading the CAB.
  • Don’t get pissed off at people on the call – that’s unprofessional and upsets everyone on the call.
  • Build a cadence to your voice and maintain it. Humans take their cues from a leader of a group – be a good positive leader, not one that sounds like they don’t want to be there.
  • Use a good online communication tool that works for everyone – bad quality voice or video adds an extra cognitive load where enough already exists.
  • Stop having CABs. Seriously.

Upgrading a RHEL7 Azure Image


Why We Do This

The default image provided as the RHEL7 image from Azure Marketplace is woefully out of date. The kernel is 3.x and has some critical CIFS bugs which will cause issues with Azure Files. A number of critical sub-systems are out of date and need to be upgraded.

What Could Go BANG!

The kernel is NOT RedHat and so you may get push back from RedHat on support for this kernel if you start hitting edge cases. However, we use the cloud. If you’re tickling kernel bugs and no one else on the Internet is – you’re doing something horribly wrong. Rebuild.

3rd party software support may also push back on non-standard kernel. Push back on the push back!

Safety First

If in doubt, please backup your VM before you do this. If this fails it’s VERY hard to get the machine back into a production state.

How we do this

Update the base platform from the RedHat Repo

  • Login and sudo su to root.
  • Change to the \ directory
  • Update yum with yum update && yum upgrade
  • Accept the defaults and allow the update to happen – this may take some time, do something else in the meantime.
  • Install yum install yum-utils and then run package-cleanup --oldkernels

Upgrade the kernel from elrepo

  • Import the ELRepo public key with rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
  • Install the RHEL7 repo with rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
  • Install the new kernel with yum --enablerepo=elrepo-kernel install kernel-ml
  • Check which menuentry is the new kernel with awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg and make a note of that number
  • Set that number as the default kernel with grub2-set-default <<NUMBER>>
  • Regenerate grub with grub2-mkconfig -o /boot/grub2/grub.cfg

Update any encryption patches and dracut

  • Create a file called yumupdatefix.sh in /root with the following contents
#!/usr/bin/env bash

# get and test path to source of most recent install
unset -v ADE_SOURCE
ADE_SEARCH=/var/lib/waagent/Microsoft.Azure.Security.AzureDiskEncryptionForLinux*
for i in $ADE_SEARCH; do
  if [[ -d "$i" ]]; then
    [[ "$i" -nt $ADE_SOURCE ]] && ADE_SOURCE=$i
  fi
done

if [[ ! -d "${ADE_SOURCE}" ]]; then
  echo "patch failed - no source directory found matching ${ADE_SEARCH}"
  exit 1
fi

# get and test path to patch file
ADE_PATCH=${ADE_SOURCE}/main/oscrypto/rhel_72/encryptpatches/rhel_72_dracut.patch
if [[ ! -f "${ADE_PATCH}" ]]; then
  echo "patch failed - no patch file found matching ${ADE_PATCH}"
fi

# replace string {Encrypted_root_partition} by partition number (=2 in rhel azure gallery images)
sed -i.bak s/ENCRYPTED_DISK_PARTITION/2/ "${ADE_PATCH}"

# patch and run
bash -c "set -e; patch -b -d /usr/lib/dracut/modules.d/90crypt -p1 < ${ADE_PATCH}"
/usr/sbin/dracut -I ntfs-3g -f -v --kver `grubby --default-kernel | sed 's|/boot/vmlinuz-||g'`

The above script will generate a new initramfs image corresponding to the new kernel version with the patched up version of Dracut modules – getting warnings is FINE. Really.

  • Enable execution of this script with chmod a+x yumupdatefix.sh
  • Execute this script with ./yumupdatefix.sh
  • Ensure that the image it builds is the same version of the kernel-ml that was installed.
  • Reboot with reboot
  • Pray or even Prey
  • When the machine comes back up, it should take a few minutes, check the new kernel with uname -msr
  • Profit.

References

http://elrepo.org/tiki/tiki-index.php
http://elrepo.org/tiki/kernel-ml
https://blogs.msdn.microsoft.com/azuresecurity/2017/07/13/applying-updates-to-a-encrypted-azure-iaas-red-hat-vm-using-yum-update/
https://gist.github.com/mayank88mahajan/38faf934c86b89ad766c4c16dcd5f4aa

 

Don’t make Apple do what you won’t!


In regards to https://www.nytimes.com/2018/01/08/technology/apple-tech-children-jana-calstrs.html?partner=rss&emc=rss and https://www.bloomberg.com/news/articles/2018-01-08/jana-calpers-push-apple-to-study-iphone-addiction-in-children I have something to say.

I’m a father of an eight-year-old.

I am an avid Apple fan.

My daughter got an iPad for her fifth birthday.

She loves it to bits.

Here are some ground rules we have for her to follow:

No iPad before 10am or after 6pm.
2 Hours a day iPad use.

The rest of the time she get’s human interaction from myself or her mother when she’s at home or she’s with friends or at school.

Software can’t do what YOU as a parent won’t.

If you can’t handle your kids then maybe you should admit that you’re a bad parent and get help with that instead of trying to ask software companies to build algorithms to make up for your lack of ability.

As to what effects it has? Take a look around you – it’s a negative one. If you allow your children to get all their interaction through a screen you’re damaging their cognitive and social skills. It’s not rocket science – it’s common sense and evidence.

Whenever people ask for studies in this or that it means they’re trying to get a different answer than the simple one.

For heavy investors in Apple and other technology giants, the answer to use their products less and maybe refrain from buying more of their products isn’t palatable.

So, instead of the simple answer which works, we’ll get complex waffle that doesn’t and then some piece of technology that’s shiny but doesn’t actually do very much.

If you have a child who has access to technology, restrict that access and replace it with human interaction. That’s the answer. Simple.

Catching The Cat


On the subject of encryption that’s getting so much press, one of my friends asked me what the PM was thinking. While we don’t actually call each other up and thus I have no idea what’s she’s thinking at any given moment, I may have an idea of how she’s thinking:

Simply put I suspect it’s a call for options. When you come up against an intractable problem you begin with an impossible answer. It’s an old methodology.  One that most of us were taught in school because we grew up before we could Google for everything. To get people out of their comfort zone you have to push them in unreasonable directions. I expected various technology groups to come up with options but so far all we have is people screaming and making a lot of noise.

One of the best ways to stop insurgents operating in this or any country is to disrupt communications – that’s hard to do because of encryption.

Encryption is the mainstay of much of geopolitics, commerce and humanity from the dawn of the common era. It’s is a boon and a curse. Over the last fifty years we have become extremely dependent on it and its usefulness. However, all these technologies of today were invented in isolation from reality in a past sure of the goodness of all men.

The problem is that those charged with developing these protocols have become used to the constraints of the technology and we need to think beyond them. At one time these technologies were the privilege of the developed world. However as ubiquitous technology opens the doors to more and more people our enemies use these techniques against us. The answer so far from the technology community is “it can’t be done”, where “it” refers to back doors in encryption. That’s not an acceptable answer because it’s not addressing the question.

When the government drafts an outrageous bills it’s looking for constructive responses. It’s asking for more effort from the subject matter experts to evaluate the real objectives.

The very idea of encryption is because we don’t trust anyone. Thus it’s impossible to accept that we should allow those who work against us to use our own technology against us.

While the ultimate decryption key is a sharp knife to a nerve cluster, that kind of behavior applied wholesale leads to a dark and dismal future and isn’t always a viable option. We’re still waiting for our technology experts to come up with an answer but many seem so enamored with their toys they can’t see past them.

Encryption is a tool.

A tool is used to execute an answer to a question.

The question is Security.

Isn’t it?

There’s an interesting blog post by Mythic Beasts on why encryption is vital. They seem to be missing the point. Everyone knows encryption is vital to the continued economic deliverable of the Internet as well as basic technology security.  While this blog post is an obvious political statement, we were rather hoping for options. Turning around and telling the wider society that the cat is out of the bag and that’s just tough is a stupid and arrogant thing to do. We’ve unleashed this double edged sword and we can’t put it back in the sheath but we must have more of an answer if we’re not to look like complete idiots to the rest of society. Like a child that spills their milk but just pouts and won’t clean it up.

When we look for an unreasonable answer this kind of response isn’t wan’t we’re expecting from people who should know better how to handle intractable problems.

So far there’s been little option provided which seems to suggest everyone is happy with the knife and nerves option.

Which is dumb.

So here, in clear and plain terms, is the question:

Given that encryption is easy to acquire and utilize, given our enemies have the access to same technologies as us, what are the options available to our society to ensure we are able to disrupt encrypted command and communication channels our enemies use whilst maintaining our freedoms to use it?

We all know we can’t put the cat back in the bag. I refuse to believe that a trillion dollar discipline such as ours can’t come up with some feasible answers that don’t involve the road to perdition.

If this is too tough a question for us, perhaps we’re not really worth the fuss.

Personal Responsibility, Privatization and the NHS


We can’t keep expecting any public service to absorb any and all demands made of it. We can’t keep expecting the public purse to keep pouring money into services at whim. That’s just not how a country works or grows. We can’t keep raising taxes to pay for run-away demand. We’ve been treating public service as if they’re in the same world as they were created in. They aren’t.

To my mind there are three things at play here:

Firstly, many of the NHS staff are outstanding. I have yet to run into clinical staff that are bad.  Even the junior doctor who managed to prod my abscess like it was a lift button wasn’t bad. You just can’t do that job effectively year after year if you don’t care about what you’re doing. Nurses in particular do an amazing job on the whole.  There are bad apples in the medical profession just like there are in any profession but being a public service it’s harder to get rid of bad staff when you’re already short staffed. Some effort being better than no effort.

Secondly, the NHS infrastructure is, for the most part, outdated. Sure, many hospitals build extensions, patch problem areas, build shiny facades but actually underneath its old and failing. Rectifying this will cost a great deal of money.  I’ve had to spend some time with a relative recently in a hospital which has a nice shiny new centre but where the main building needs attention. Most hospitals and care centres need to be relocated and rebuilt. They’re now in the wrong place and can’t grow. There isn’t the money for that in the public purse.

Thirdly, there is not accounting in any of the published reports for personal responsibility. It seems that we are suggesting that no matter how we behave, what risks we choose to take – the NHS should “just be there” like mommy and daddy to make it all better. It’s time to ask why we think that’s ok.

To the question of why I support privatisation:

  • We need to be able to manage, rate and reward staff better. Bottom line, shit performing staff should be canned. Excellence should be rewarded. Mediocrity nurtured to grow. This is hard to do in the NHS because of various issues. By the way, this goes for all jobs, just turning up doesn’t count. Business knows how to find, train, invest in and exit staff.
  • We need to rebuild our health service infrastructure. Hospitals built in the 1970’s are in the wrong place now and/or too old. There’s no parking or the charges are likely to induce cardiac issues. We need new centres of care to replace these old places which can be sold on to bring billions in to the pot.
  • We need to modernise the tools and treatments we use. In some areas we’re leading the world. In others we’re a notch above using leeches and calling the shaman. Ok, I exaggerate – I’ve never actually seen a shaman being called.
  • We need to fund research in a more transparent way – it’s bloody Ponzi scheme in disguise at the moment. With all the millions raised by donations to various charities why do all these new drugs cost us so much? If you’re interested go learn how all this money ends up in the same group of companies we end up paying for the drugs.
  • The NHS isn’t a fluffy thing. It’s a vast organisation and we need better management of its resources by people who do management well. The NHS don’t do management well. There’s a conflict of interest. Management is a skill just like anything else. Get in organisations who know how to do this. This isn’t a place you want to “promote out of harm’s way”.
  • We need to be able to hold that management to account better and reward/penalise without affecting services. Penalising a trust effects the pot available to care. Penalising a company makes for an interesting shareholder meeting.
  • We need the slogan “Your health, your choices” mean more than where you get treated. It should mean we accept the responsibility for our health and should pay toward treatment when it’s the result of poor personal choices. Getting pissed and ending up in A&E shouldn’t cost the rest of the country. Smoking and getting cancer treatment shouldn’t cost the rest of the country. Causing an accident and needing treatment shouldn’t cost the rest of the country. Palliative care should be means tested for contributions against your estate.
  • If you’re not a UK citizen you should be paying for your care. When we leave the EU you definitely should be paying for your care or your insurance should be. We are not a freebie. If you’re a UK citizen abroad you should be paying for your care – that’s what travel insurance is for.
  • Being retired shouldn’t mean a delegation of the responsibility for your healthcare costs to the rest of the country. Yes you paid in and yes you also got lots for it during your working life. If you have the means you should pay something towards the cost of your treatment and care. If you don’t we’ll take care of you. You shouldn’t be able to hide your assets by transferring them to your children so it looks like you haven’t the means either. Your direct family should be taken into account at a different rate.
  • Medical insurance is a cost. We value what costs us money. You drive carefully partly because you know the impact on your bottom line if you drive like a dick and cause an accident. The same should apply to your health. If you’re unemployed or within the non-taxable bracket your healthcare should be free.

 

When everyone takes more responsibility for their choices there’s more money in the pot to help those who don’t have the means. When everyone sees a tangible cost each month for their lifestyle choices they get to make those choices better informed of the consequences. We don’t have to pay for everyone’s choices. Just ours and those who didn’t have any other choice.  By the way, I’m still at a loss as to how anyone in the history of the world ever thought inhaling smoke wasn’t going to cause problems.

When we use private investment to upgrade and manage infrastructure and services we have better accountability and more leverage. If you want better health care then you’re going to have to take some responsibility for it and not just assume government will take care of you. It’s not actually your parents.

As always none of this exists in isolation to everything else that we have to take care of and consider.

We have to reach for it all, for everyone.

#VoreTory or #VoteKhushForGlobalOverlord but whatever you do #GoVote

Engineering Small


Stay focused on the engineering reality defined by your capabilities and requirements. Line up the engineering tasks with the business value. A shift in the engineering tasks should be a conscious decision as a result of conversations between the technical and business leaders in order to better deliver on the business value. Technical debt can be discarded when the engineering objectives change. Remember all technology is transient and that pride cometh before a fall.