VMware upgrade checklist and good practice

Recently I’m doing a lot of upgrades, mostly because vSphere 6.0 end of general support is on 12.03.2020. In this post I would like to describe some good practices. I will write about vSphere upgrade, but most of this things are universal also for other products. First lets describe what is the difference between update and upgrade.

Update vs Upgrade vs Patch vs Migration

Difference between update and upgrade depends if you are changing the version or you stay at the same major version. Upgrades involves major changes to the software, so when you are running vSphere 6.0 and want to go to vSphere 6.5, you are doing upgrade. During upgrade ESXi host configuration can be changed.
Updates are releases that makes a smaller change, for example going from vSphere 6.0 U1 to vSphere 6.0 U2 (U stands for Update).
In VMware there are also patches, which are smaller changes then updates (example security patches). Because updates and patches do not involve major changes to the software, ESXi host configuration is not affected.
When during upgrade you changing vCenter Server for Windows to vCenter Server Appliance, it is migration.

Check VMware documentation about upgrades, updates and patches

First step

First you need to decide to which version you would like to upgrade to. Usually going with the just released version isn’t good idea. In the past there were several cases, then couple days after release there was some issue with upgrade and VMware removed it until fix was available. Download installation files only from trusted sites. Always check MD5 checksum after download.

Licenses

Verify if you have correct license. Usually vCenter and ESXi license is for major version of product, so for example when you have license for vSphere 6, you can install 6.0, 6.5 and 6.7, but for 7.0 (once released) you will need new license key.

Test upgrade in Lab

Something that isn’t obviously for all 🙂 First you need to have a lab, the best if it will be exactly copy of your production environment (the same backup, monitoring, HW, etc.). I would recommend to create change ticket also for upgrading test environment, so you will already have template to use in production environment. I’m usually running upgrade in test environment 1 month before planned prod upgrade.

Create change ticket

I’m starting my preparation with creating change ticket for upgrading test environment. This will be my template for production environment upgrade change. I’m creating change task (CTASK) for each action done during upgrade, which will be my check list. Usually I’m creating separate change for vCenter upgrade and separate for ESXi upgrade – vCenter upgrade requires some break, when some functionalities are affected. In case of ESXi upgrade, if you have spare hosts in clusters, it can be done later without affecting production.

Example change:

  1. Preparation tasks (1 day before upgrade window):
    • Copy verified installation ISO files on datastore / shared place.
    • Export host profiles
    • Check access to OS, vSphere, SSO, database
    • Check if there is some update in Release notes (maybe some new bug?)
    • Generate support bundle
  2. Before installation tasks (1 hour before upgrade window):
    • Generate report in RVtools
    • Inform customer / helpdesk / other teams, that change is going to start
  3. Start change – preinstallation tasks
    • Check if everything is working OK (if there aren’t any new alerts, if all hosts are connected to vCenter, etc.)
    • Suppress monitoring
    • Create database backup
    • Connect to ESXi host, where vCenter and vCenter DB is running. Shut down those VMs and create snapshots (or clone, but from my experience offline snapshot is enough)
    • Start VMs
  4. vCenter Upgrade
  5. Testing after vCenter upgrade
    • Steps here also depends of your environment. Some examples:
      • Check if all services are up and running
      • Test if you are able to login via vSphere Client / Web Client (sometimes you need to install new web plugins)
      • Check if all hosts are connected
      • Check alerts
      • Test if 3rd party tools can connect to vCenter (vCloud Director, NSX, vROPS, vRealize LogInsight, vRO, vRA, some other tools like backup, monitoring, cloud portal etc.)
      • Test vMotion, VM deployment, etc.
  6. If anything isn’t working, do rollback (restore VM from snapshot / clone)
  7. Upgrade VMware Update Manager
  8. Inform customer / helpdesk / other teams that vCenter is upgraded.
  9. ESXi upgrade (usually I have seperate change for it and I’m running ESXi upgrade in different time, then vCenter upgrade – even in office hours)
  10. Usually after couple days I’m deleting snapshots

Company policy

Check your company policy when you need to create change, get all approvals, inform customers. For example you may have change draft 6 weeks before planned upgrade, you need to discuss it on a CAB meeting (Change Approval Board) and 1 month before maintenance window, you have to inform all parties (help desk, customers, other teams) about planned break.

Tips & tricks

  • Don’t forget to delete snapshots 😉
  • Create a template for VMware support in case something will go wrong
    • Example: I was doing upgrade of vCenter from version <version> to version <version>. On step <this and this> I had an issue:
  • If you have vCenter on Windows, clean up C:\temp and %USERPROFILE%\AppData\Local\Temp – couple time I got issue because of it

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.