Unable to retrieve the complete inventory of vCenter objects (vCenter DOS alerts)

Alexey Koznov -

VMware Aria Operations (formerly vRealize Operations) is a powerful tool for managing and monitoring virtualized environments. It provides comprehensive insights into various aspects of your infrastructure, including hosts and virtual machines (VMs) managed by vCenter. However, users may sometimes encounter a frustrating issue where they are unable to retrieve the complete inventory of vCenter objects through vRealize Operations. This article delves into a common cause of this problem and provides a step-by-step guide to fixing it.

When attempting to gather information about hosts and VMs from vCenter through vRealize Operations and not all objects are available at interface you can check logs of the vAPI endpoint to troubleshoot communications.

You will see the couple of messages that are showing some Over Limit connections and delaying of that requests:

2023-08-07T14:10:16.951Z | WARN | jetty-default-2195 | DoSFilter | DOS ALERT: Request delayed=100ms, ip=127.0.0.1, overlimit=OverLimit@43c9916[type=IP, id=127.0.0.1, duration=PT0.977020579S, count=1500], session=null, user=null
2023-08-07T14:10:16.951Z | WARN | jetty-default-1799 | DoSFilter | DOS ALERT: Request delayed=100ms, ip=127.0.0.1, overlimit=OverLimit@3c8ebf50[type=IP, id=127.0.0.1, duration=PT0.977192879S, count=1500], session=null, user=null
2023-08-07T14:10:16.953Z | WARN | jetty-default-1943 | DoSFilter | DOS ALERT: Request delayed=100ms, ip=127.0.0.1, overlimit=OverLimit@dd73aff[type=IP, id=127.0.0.1, duration=PT0.979394814S, count=1500], session=null, user=null
2023-08-07T14:10:16.955Z | WARN | jetty-default-2069 | DoSFilter | DOS ALERT: Request delayed=100ms, ip=127.0.0.1, overlimit=OverLimit@2dabfb64[type=IP, id=127.0.0.1, duration=PT0.980555122S, count=1500], session=null, user=null
2023-08-07T14:10:16.955Z | WARN | jetty-default-2197 | DoSFilter | DOS ALERT: Request delayed=100ms, ip=127.0.0.1, overlimit=OverLimit@58f223e[type=IP, id=127.0.0.1, duration=PT0.980456018S, count=1500], session=null, user=null
2023-08-07T14:10:16.957Z | WARN | jetty-default-1939 | DoSFilter | DOS ALERT: Request delayed=100ms, ip=127.0.0.1, overlimit=OverLimit@2409bba0[type=IP, id=127.0.0.1, duration=PT0.982484841S, count=1500], session=null, user=null

This error indicates that there is an overload or throttling issue in the communication between vRealize Operations and the vAPI endpoint.

Fortunately, addressing this issue is relatively straightforward, involving a configuration change and a service restart. Follow these steps to resolve the incomplete inventory problem:

Step 1: Edit the vAPI Endpoint Configuration

The vAPI endpoint configuration file is located at /etc/vmware-vapi/endpoint.properties within the vCenter Server Appliance.

Open the endpoint.properties file in a text editor of your choice. Find and change the settings that are related to throttling (limitation of the number of incoming requests to prevent overloading the system).

Make a copy of configuration file and make a commented line copy for all parameters that you will change. Snapshot is also good to have.

You might find settings like and http.request.rate.count , http.request.rate.interval , http.request.maxSize and http.request.inflight.maxCount in this file.

Step 2: Adjust the Settings

Set the values of http.request.rate.count , http.request.rate.interval , http.request.maxSize and http.request.inflight.maxCount to 0 (zero) to allow for an unlimited volume of incoming requests. These values determine how many requests the system can handle concurrently and when to start throttling requests. Increasing these values can help accommodate larger inventory retrieval requests.

http.request.rate.count=0
http.request.rate.interval=0
http.request.maxSize=0
http.request.inflight.maxCount=0

Also, we have increased heap size for allocated memory for vAPI endpoint service to 1024mb. For checking current settings you can use command cloudvm-ram-size -l | grep vmware-vapi-endpoint . For our installation size heap size was 1024mb.

We've increased that number to 512mb (using article https://kb.vmware.com/s/article/83135):

cloudvm-ram-size -C 1024 vmware-vapi-endpoint

Step 3: Save the Changes

After making the necessary adjustments, save the changes to the endpoint.properties file.

Step 4: Restart the vAPI Endpoint Service

To apply the changes, restart the vAPI endpoint service. You can do this using the appropriate commands from ssh session or using VAMI web interface.

You might use the following commands to stop and start service:

service-control --stop vmware-vapi-endpoint
service-control --start vmware-vapi-endpoint

Step 5: Verify the Fix

Once the service is restarted, attempt to retrieve the vCenter inventory from vRealize Operations again. Monitor the logs for any recurrence of the 'DOS ALERT' error message. If the error no longer appears, and you're able to retrieve the complete inventory, then the issue is successfully resolved.