In this blog post, we are going to discuss how to migrate encrypted vSAN cluster from one vCenter server to another vCenter server. There is a reason to write this post as with encrypted vSAN cluster steps are different and if we miss any step, it may cause vSAN disks to be locked and downtime for vSAN cluster. Hence, it is imperative to follow the step by step procedure.
Below environment details are must to be captured from existing vCenter Server before going for migration process.
[root@localhost:~] vmware -vl VMware ESXi 6.7.0 build-13644319 VMware ESXi 6.7.0 Update 2
[root@localhost:~] localcli vsan cluster get Cluster Information: Enabled: true Current Local Time: 2019-05-30T00:43:59Z Local Node UUID: 5cedfe44-ccc1-b8e2-88cc-00505601e17a Local Node Type: NORMAL Local Node State: BACKUP Local Node Health State: HEALTHY Sub-Cluster Master UUID: 5cedfe78-e686-c30d-4769-00505601e17b Sub-Cluster Backup UUID: 5cedfe44-ccc1-b8e2-88cc-00505601e17a Sub-Cluster UUID: 52bab0ab-74b3-bb9d-4cb6-6937787c5aa1 Sub-Cluster Membership Entry Revision: 2 Sub-Cluster Member Count: 3 Sub-Cluster Member UUIDs: 5cedfe78-e686-c30d-4769-00505601e17b, 5cedfe44-ccc1-b8e2-88cc-00505601e17a, 5cee2ed1-0d66-73a4-d660-0050569a7c66 Sub-Cluster Member HostNames: localhost.vhabit.com, localhost.vhabit.com, SmallESXi.vhabit.com Sub-Cluster Membership UUID: 4433ee5c-2b7d-dd80-5040-00505601e17b Unicast Mode Enabled: true Maintenance Mode State: OFF Config Generation: 31ad6051-5228-4676-a75c-a153fd36cf6b 5 2019-05-29T07:31:47.121
vSAN Storage Information:
[root@localhost:~] localcli vsan storage list | grep -i cmmds In CMMDS: true In CMMDS: true
[root@localhost:~] localcli vsan storage list | grep -i diskkey DiskKeyLoaded: true DiskKeyLoaded: true
[root@localhost:~] cat /etc/vmware/esx.conf | grep -i kek /vsan/kekId = "ab091586-81e3-11e9-bd13-0050569a293a" [root@localhost:~] cat /etc/vmware/esx.conf | grep -i hostkey /vsan/hostKeyId = "ab1ade4b-81e3-11e9-bd13-0050569a293a"
[root@localhost:~] cat /etc/vmware/esx.conf | grep -i kmip /vsan/kmipServer/child/kmskey = "KMSSecurity/KMS2Encrypt" /vsan/kmipServer/child/kmipClusterId = "KMSSecurity" /vsan/kmipServer/child/name = "KMS2Encrypt" /vsan/kmipServer/child/address = "192.168.2.123" /vsan/kmipServer/child/port = "5696" /vsan/kmipServer/child/old = "false" /vsan/kmipClusterId = "KMSSecurity"
VMKernel Port Information:
[root@localhost:~] esxcli network ip interface tag get -i vmk0 Tags: VSAN, Management, VMotion [root@localhost:~] esxcli network ip interface tag get -i vmk0 Tags: VMotion, Management, VSAN
In my lab I have encrypted stretched cluster with tiny linux vm running on it to check connectivity issues while migrating to new vCenter Server. Let’s consider some prerequisites which are mentioned in my previous blog post here https://bit.ly/2VeeLRs
Important step to take in existing setup:
- To maintain cluster membership set these values on all the hosts in the cluster “esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates” ( only needed post vsan 6.6)
Steps to be taken on new vCenter Server:
- Create a vSAN Cluster and enable vSAN, HA, DRS, Deduplication & Encryption ( Same features as previous cluster)
- Import DVSwitch from existing vCenter Server
- Create custom storage policies with reference to existing one ( if any)
- Remove KMS from existing vCenter and add it to new vCenter with same cluster name ( details can be fetched from esx.conf file)
- Configure the stretch cluster with same witness node
- Add the nodes into DVS without connectivity loss
We will migrate Metro Cluster and in this screenshot you can that this is a stretched mode is enabled. Hosts only contain one vmkernel port with management,vsan,vmotion enabled and is on vDS
In this screenshot, you can see that Encryption is enabled and is using KMS Cluster “KMSSecurity”
All the disks in this cluster are healthy and mounted in existing cluster
Created one tiny linux machine and started connectivity test
As, I am using vSAN default storage policy there is no need to create a policy as it exists by default. If you have any custom VM storage policies, in this step you need to create the same.
Now, let’s remove the KMS server from the existing environment and add it to new vCenter Server. This is going to be the crucial step as you have to add the KMS with same cluster ID and other details.
Disable the stretch cluster configuration in order to remove the nodes successfully.
The next plan is to add the hosts one by one at the data-center object first and check the VM connectivity then add the witness host onto the witness data-center object.
Once these nodes are added add them into the newly created cluster with same properties. Now, you need to configure the stretch cluster again and add the witness node.
Once the cluster is configured, you may have add the hosts into imported vDS configuration.
No connectivity loss of VM and VMkernel management network
Now, check the vSAN health and revert the advance values to “esxcfg-advcfg -s o /VSAN/IgnoreClusterMemberListUpdates” on the datanodes.
You may get a vSAN health error “vCenter state is authoritative” shows error as we just moved cluster from another setup and hosts are out of sync. You can run remediation on this health check safely to get vCenter and hosts configurations to be in sync.
Please note, if KMS is added after the cluster nodes are added in the vSAN cluster the you will face issues with Disk locked and VMs will be inaccessible. You need to configure the KMS in vSAN cluster and then it will run shallow key to generate new HostKey and KEK key to rewrap DEK. Eventually, disks will be able to mount but this process carries downtime of complete vSAN Cluster.
I hope this procedure simplifies the cluster movements between vCenter Servers.
Thank you for reading!!