Changing KMS Server in vSAN Encryption Cluster

In this blog post we will discuss how to change KMS server for vSAN Encryption. I have vSAN 6.7U1 stretched cluster environment and have configured vSAN Encryption with Hytrust KMS server. It is a automated process & pretty simple.

In my lab environment, I have existing setup of Stretched cluster with Encryption enabled. We will go through the process of changing KMS server smoothly to avoid downtime.

In the below screenshot you can see that encryption is enabled and KMS server “Encryption” is configured.

In this screenshot, vSAN health test shows KMS cluster status and mutual trust established between vCenter & KMS server.

Now, question would pop-up as to why do we need to change KMS server when existing configuration is working fine. The answer depends on the use case for e.g if other vendor is more cheaper than existing one.

 

Let’s us deploy new KMS server in the environment and begin with the process of adding the same into vCenter Server. In this case I am not showing you deployment & configuration process of KMS server.

 

Once the KMS configuration is complete, add the KMS server in vCenter. The new cluster name is “Encryption2” and server name is “KMSReload” and click on Add.

 

Click on Trust ( This wizard will show server certificate of the KMS server which would like to add). You can confirm the server certificate by logging into the KMS web-browser.

Once vCenter trusts KMS, It will be added into the KMS options. In the below screenshot you can see that only way trust is established. KMS trust to vCenter is still pending. For that there are different ways to get certificates generated based on the vendor’s request.

 

Based on the vendor’s capability I have selected the option and click on next

I have uploaded KMS certificate and private key downloaded from the KMS server. Click on “Establish Trust”

After clicking on establish trust, KMS to vCenter trust has been created and vCenter is successfully connected to the KMSReload.

 

Now, vCenter and KMS connection has been established and need to re-encrypt the stretched cluster. Select Cluster ⇒ Configure ⇒ Services ⇒ Encryption ⇒ Edit and you can see drop down shows two KMS servers. Encryption is the existing one and Encryption2 is the new one. Select Encryption2 and click on apply.

The moment vCenter trusts new KMS, it will request KEK_ID, HEK_ID, KMS Certs & settings. It will push all the configurations to all the hosts in the cluster which will overwrite existing KMIP configuration. This process is very quick and will initiate shallow rekey with in vSAN Encryption cluster. Health test also show all green.

Now, let’s take a look at the configuration change in the ESXi hosts:

 

Existing configuration with KMS Server “Encryption”

[root@blr9sc:~] grep kmip /etc/vmware/esx.conf
/vsan/kmipServer/child[0001]/old = "false"
/vsan/kmipServer/child[0001]/port = "5696"
/vsan/kmipServer/child[0001]/address = "192.168.2.122"
/vsan/kmipServer/child[0001]/name = "KMSHytrust2"
/vsan/kmipServer/child[0001]/kmipClusterId = "Encryption"
/vsan/kmipServer/child[0001]/kmskey = "Encryption/KMSHytrust2"
/vsan/kmipServer/child[0000]/kmskey = "Encryption/KMSHytrust1"
/vsan/kmipServer/child[0000]/kmipClusterId = "Encryption"
/vsan/kmipServer/child[0000]/name = "KMSHytrust1"
/vsan/kmipServer/child[0000]/address = "192.168.2.121"
/vsan/kmipServer/child[0000]/port = "5696"
/vsan/kmipServer/child[0000]/old = "false"
/vsan/kmipClusterId = "Encryption"

[root@blr9sc:~] grep -i /vsan/hostk /etc/vmware/esx.conf
/vsan/hostKeyId = "0d10f815-124a-11e9-a645-0050569ac1ea"

 

New configuration with KMS server “Encryption2”

[root@blr9sc:~] grep -i /vsan/k /etc/vmware/esx.conf
/vsan/kmipClusterId = "Encryption2"
/vsan/kmipServer/child[0002]/old = "false"
/vsan/kmipServer/child[0002]/port = "5696"
/vsan/kmipServer/child[0002]/address = "192.168.2.123"
/vsan/kmipServer/child[0002]/name = "KMSReload"
/vsan/kmipServer/child[0002]/kmipClusterId = "Encryption2"
/vsan/kmipServer/child[0002]/kmskey = "Encryption2/KMSReload"
/vsan/kekId = "48fbb7f6-1320-11e9-958d-0050569af6e0"

[root@blr8sc:~] grep -i /vsan/hostk /etc/vmware/esx.conf
/vsan/hostKeyId = "49163c3e-1320-11e9-958d-0050569af6e0"

With the same configuration the old certificates for e.g vsan_kms_castore.pem, vsan_kms_client.key, vsan_kms_client.crt will be overwritten with new certificates.

Post changing the KMS server to “Encryption2” and checking the configuration, I rebooted one of the ESXi node to check if we see any issues with Disk group mount. And as expected disk groups mounted with out any issues.

[root@blr10sc:~] localcli vsan storage list | grep -i cmmds
In CMMDS: true
In CMMDS: true

As I said, this process is very straight forward and simple.

 

Happy learning!!

Hope you have enjoyed reading the article!!

 

4 Comments

  1. What KMS solutions (besides the big guys, Hytrust, Townsend) do you recommend. I have a very small environment and need to encrypt a few machines but most vendors are asking around 14K for a site setup (unlimited VMs), which is out of my range. I wish VMware had there own small basic KMS setup.

Leave a Reply

Your email address will not be published.


*