Migrating Encrypted vSAN Stretched Cluster from one vCenter Server to another vCenter Server

In this blog post, we are going to discuss how to migrate encrypted vSAN cluster from one vCenter server to another vCenter server. There is a reason to write this post as with encrypted vSAN cluster steps are different and if we miss any step, it may cause vSAN disks to be locked and downtime for vSAN cluster. Hence, it is imperative to follow the step by step procedure.

Below environment details are must to be captured from existing vCenter Server before going for migration process.

Check Version:

[root@localhost:~] vmware -vl
VMware ESXi 6.7.0 build-13644319
VMware ESXi 6.7.0 Update 2

Cluster Information:

[root@localhost:~] localcli vsan cluster get
Cluster Information:
Enabled: true
Current Local Time: 2019-05-30T00:43:59Z
Local Node UUID: 5cedfe44-ccc1-b8e2-88cc-00505601e17a
Local Node Type: NORMAL
Local Node State: BACKUP
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 5cedfe78-e686-c30d-4769-00505601e17b
Sub-Cluster Backup UUID: 5cedfe44-ccc1-b8e2-88cc-00505601e17a
Sub-Cluster UUID: 52bab0ab-74b3-bb9d-4cb6-6937787c5aa1
Sub-Cluster Membership Entry Revision: 2
Sub-Cluster Member Count: 3
Sub-Cluster Member UUIDs: 5cedfe78-e686-c30d-4769-00505601e17b, 5cedfe44-ccc1-b8e2-88cc-00505601e17a, 5cee2ed1-0d66-73a4-d660-0050569a7c66
Sub-Cluster Member HostNames: localhost.vhabit.com, localhost.vhabit.com, SmallESXi.vhabit.com
Sub-Cluster Membership UUID: 4433ee5c-2b7d-dd80-5040-00505601e17b
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Config Generation: 31ad6051-5228-4676-a75c-a153fd36cf6b 5 2019-05-29T07:31:47.121

vSAN Storage Information:

[root@localhost:~] localcli vsan storage list | grep -i cmmds
In CMMDS: true
In CMMDS: true

Encryption Information:

[root@localhost:~] localcli vsan storage list | grep -i diskkey
DiskKeyLoaded: true
DiskKeyLoaded: true
[root@localhost:~] cat /etc/vmware/esx.conf | grep -i kek
/vsan/kekId = "ab091586-81e3-11e9-bd13-0050569a293a"

[root@localhost:~] cat /etc/vmware/esx.conf | grep -i hostkey
/vsan/hostKeyId = "ab1ade4b-81e3-11e9-bd13-0050569a293a"
[root@localhost:~] cat /etc/vmware/esx.conf | grep -i kmip
/vsan/kmipServer/child[0000]/kmskey = "KMSSecurity/KMS2Encrypt"
/vsan/kmipServer/child[0000]/kmipClusterId = "KMSSecurity"
/vsan/kmipServer/child[0000]/name = "KMS2Encrypt"
/vsan/kmipServer/child[0000]/address = "192.168.2.123"
/vsan/kmipServer/child[0000]/port = "5696"
/vsan/kmipServer/child[0000]/old = "false"
/vsan/kmipClusterId = "KMSSecurity"

VMKernel Port Information:

[root@localhost:~] esxcli network ip interface tag get -i vmk0
Tags: VSAN, Management, VMotion

[root@localhost:~] esxcli network ip interface tag get -i vmk0
Tags: VMotion, Management, VSAN

In my lab I have encrypted stretched cluster with tiny linux vm running on it to check connectivity issues while migrating to new vCenter Server. Let’s consider some prerequisites which are mentioned in my previous blog post here  https://bit.ly/2VeeLRs

Important step to take in existing setup:

  • To maintain cluster membership set these values on all the hosts in the cluster “esxcfg-advcfg -s 1 /VSAN/IgnoreClusterMemberListUpdates” ( only needed post vsan 6.6)

Steps to be taken on new vCenter Server:

  • Create a vSAN Cluster and enable vSAN, HA, DRS, Deduplication & Encryption ( Same features as previous cluster)
  • Import DVSwitch from existing vCenter Server
  • Create custom storage policies with reference to existing one ( if any)
  • Remove KMS from existing vCenter and add it to new vCenter with same cluster name ( details can be fetched from esx.conf file)
  • Configure the stretch cluster with same witness node
  • Add the nodes into DVS without connectivity loss

We will migrate Metro Cluster and in this screenshot you can that this is a stretched mode is enabled. Hosts only contain one vmkernel port with management,vsan,vmotion enabled and is on vDS

In this screenshot, you can see that Encryption is enabled and is using KMS Cluster “KMSSecurity”

All the disks in this cluster are healthy and mounted in existing cluster

Created one tiny linux machine and started connectivity test

We have created a cluster on new vCenter Server and enabled features ( vSphere HA, vSphere DRS, vSAN)

As, I am using vSAN default storage policy there is no need to create a policy as it exists by default. If you have any custom VM storage policies, in this step you need to create the same.

Now, let’s remove the KMS server from the existing environment and add it to new vCenter Server. This is going to be the crucial step as you have to add the KMS with same cluster ID and other details.

Disable the stretch cluster configuration in order to remove the nodes successfully.

 

The next plan is to add the hosts one by one at the data-center object first and check the VM connectivity then add the witness host onto the witness data-center object.

Once these nodes are added add them into the newly created cluster with same properties. Now, you need to configure the stretch cluster again and add the witness node.

 

Once the cluster is configured, you may have add the hosts into imported vDS configuration.

No connectivity loss of VM and VMkernel management network

 

Now, check the vSAN health and revert the advance values to “esxcfg-advcfg -s o /VSAN/IgnoreClusterMemberListUpdates” on the datanodes.

You may get a vSAN health error “vCenter state is authoritative” shows error as we just moved cluster from another setup and hosts are out of sync. You can run remediation on this health check safely to get vCenter and hosts configurations to be in sync.

 

Please note, if KMS is added after the cluster nodes are added in the vSAN cluster the you will face issues with Disk locked and VMs will be inaccessible. You need to configure the KMS in vSAN cluster and then it will run shallow key to generate new HostKey and KEK key to rewrap DEK. Eventually, disks will be able to mount but this process carries downtime of complete vSAN Cluster.

 

 

 

I hope this procedure simplifies the cluster movements between vCenter Servers.

 

Thank you for reading!!

Be the first to comment

Leave a Reply

Your email address will not be published.


*