Fixing Service Account Duplication in VMware VCF: Resolving vCenter / NSX Connection Issues

Environment Overview

We have a couple of workload domains which have the next scheme of domain names:

Workload_Domain_ID-Workload_Domain_Type.regionX.domain.local

For example:

v1-01.regionA.domain.local v1-02.regionA.domain.local

Where v1-01 - our Management domain and v1-02 - Workload domain names.

vCenter servers has DNS FQDNs:

vc01.v1-01.regionA.domain.local for v1-01 Workload Domain
vc01.v1-02.regionA.domain.local for v1-02 Workload Domain

Issue Overview

In VMware VCF, when managing multiple workload domains (e.g., v1-01, v1-02), with DNS FQDNs which contains 4 subdomains (it is my assumption) SDDC Manager doesn't create a new service accounts. Instead of that, it is using Management domain’s vCenter (e.g., vc01.v1-01.domain.local) account. This leads to service account duplication inside database when adding new workload domains, rewriting a password for service account and causing problems for connecting SDDC Manager to vCenter and NSX components.

Identifying the Issue

When running the command to list service account credentials, you'll see multiple entries with identical usernames, despite them belonging to different vCenters. For example:

TOKEN=$(curl -d '{"username" : administrator@vsphere.local, "password" : "$PSC_PASSWORD_FROM_LOOKUPPASSWORD_IN_SDDC_MANAGER"}' -H "Content-Type: application/json" -X POST http://127.0.0.1/v1/tokens | jq -r '.accessToken')
 
curl -k -X GET -H "Authorization: Bearer "$TOKEN"" --insecure 'https://localhost/v1/system/credentials/service' | json_pp | less
 
   {
      "creationTime" : 1724069003300,
      "credentialType" : "SSO",
      "entityId" : "8b82cb02-f752-4d66-a33e-81d63caeeabf",
      "entityType" : "VCENTER",
      "id" : "9f997cc4-9f8a-4ce3-a06b-d7c4d1c79858",
      "modificationTime" : 1727285744350,
      "secret" : "just_a_super_secure_password",
      "serviceId" : "ae094afe-1b8d-4359-92a6-fc1426654990",
      "serviceType" : "SDDC_MANAGER",
      "targetType" : "VCENTER",
      "username" : "svc-vcf01-v1-01-vc01@vsphere.local" 
   },
   {
      "creationTime" : 1724877651231,
      "credentialType" : "SSO",
      "entityId" : "b4b9e13b-649d-4b4f-8bcb-9de828af3972",
      "entityType" : "VCENTER",
      "id" : "ca7fdb72-10d8-4321-95a7-2334cfa8938f",
      "modificationTime" : 1727285040737,
      "secret" : "just_another_super_secure_password ",
      "serviceId" : "ae094afe-1b8d-4359-92a6-fc1426654990",
      "serviceType" : "SDDC_MANAGER",
      "targetType" : "VCENTER",
      "username" : "svc-vcf01-v1-01-vc01@vsphere.local"
   }

If we check info about vCenter and compare them with entityID we will see that different vCenters are using different entries, but username in those records are the same:

root@vcf01-nbg6-m01 [ / ]#  psql --host=localhost -U postgres
psql (13.14)
Type "help" for help.
postgres=# \c platform
You are now connected to database "platform" as user "postgres".
platform=# select id,vm_hostname from vcenter;
                  id                  |        vm_hostname
--------------------------------------+----------------------------
8b82cb02-f752-4d66-a33e-81d63caeeabf | vc01.v1-01.nec.noris.de 
b4b9e13b-649d-4b4f-8bcb-9de828af3972 | vc01.v1-02.nec.noris.de
(2 rows)

This conflict can lead to vCenter authentication failures, especially when doing operations within the Workload Domain (adding hosts, creating clusters, etc.).

Example of errors, if service account is broken

Emergency workaround: If we need to work in Workload domain v1-01 we need to set the PW with ID 9f997cc4-9f8a-4ce3-a06b-d7c4d1c79858, otherwise if we are working in v1-02 - set the PW with ID ca7fdb72-10d8-4321-95a7-2334cfa8938f.

Procedure for Resolution

Create New Accounts in vCenter
On the respective vCenter (e.g., vc01.v1-02.domain.local), create the following accounts:
svc-vcf01-vc01-v1-02@vsphere.local svc-nsx01-vc01-v1-02@vsphere.local
Assign these accounts the global Administrator role.

Add Users to Groups
Add the created users to the following groups:
Users:
svc-vcf01-vc01-v1-02@vsphere.local
Groups:
Administrators, CAAdmins, SystemConfiguration.BashShellAdministrators

Update Usernames in SDDC Manager
SSH into the SDDC Manager database and run the following commands to update the usernames:

root@sddc-manager [ ~ ]# psql -U postgres -h localhost -d platform
psql (13.14)
Type "help" for help.
 
update credential set username='svc-vcf01-vc01-v1-02@vsphere.local' where username='svc-vcf01-vc01-v1-01@vsphere.local' and entityid in (select id from vcenter where vm_hostname='vc01.v1-02.nec.noris.de'); 

update credential set username='svc-nsx01-vc01-v1-02@vsphere.local' where username='svc-nsx01-vc01-v1-01@vsphere.local' and entityid in (select id from vcenter where vm_hostname='vc01.v1-02.nec.noris.de');

Update Service Account Passwords
After updating the usernames, log in to the SDDC Manager UI and navigate to the password manager screen to update the passwords for the service accounts.

Conclusion

This procedure helps ensure that VMware VCF services work correctly across multiple workload domains and prevents authentication issues due to service account duplication.