Insufficient disk space during NSX-V upgrade

Alexey Koznov -

During planned upgrade from NSX-V version 6.4.4 to 6.4.6 we stucked with error in NSX Manager:

Insufficient disk space. Database disk usage is at 72%, but it should be less than 70%.
NSX Manager error - Insufficient disk space with info from NSX Manager

Cause of issue

  • When inflow of number of jobs / tasks was very high, purge tasks was not able to keep pace with the inflow.
  • This eventually resulted in growth in job task tables, causing high disk space consumption on NSX manager.

Steps to resolve issue

WARNING: Before you start create a DB backup. For making any changes in the DB of products you should contact support and create SR to get proper assistance! Use this post at your own risk. We don’t take responsibility and/or give any warranty if you reuse this content.

  • Take a backup of the NSX Manager.
  • Login and root to the NSX manager (you can use KB2149630)
  • Get back disk space using full vacuum. Follow below steps:
  • Stop NSX Manager services:
/etc/rc.d/init.d/bluelane-manager stop
  • Log into psql database:
psql -U secureall
  • Go to postgreSQL prompt and run following commands:
VACUUM (FULL) job_instance_task_instances;
VACUUM (FULL) task_instance_task_data;
VACUUM (FULL) task_instance_task_output;
VACUUM (FULL) task_instance;
VACUUM (FULL) task_task_init_data;
VACUUM (FULL) task;
VACUUM (FULL) task_policy;
VACUUM (FULL) task_target;
VACUUM (FULL) job_instance_job_output;
VACUUM (FULL) job_instance;
VACUUM (FULL) job_data_task_dependency_map;
VACUUM (FULL) task_dependency_tasks;
VACUUM (FULL) dependent_task;
VACUUM (FULL) job_data;
VACUUM (FULL) job_schedule;
VACUUM (FULL) task_dependency;

Note: For large data these commands will take time

  • Truncate the NSX tables by running command below:
TRUNCATE TABLE job_instance_task_instances, task_instance_task_data, task_instance_task_output, task_instance, task_task_init_data, task, task_policy, task_target, job_instance_job_output, job_instance, job_data_task_dependency_map, task_dependency_tasks, dependent_task, job_data, job_schedule, task_dependency, housekeeping_module;
  • Exit psql database:
\q
  • Start NSX Manager services:
/etc/rc.d/init.d/bluelane-manager start

Conclusion

  • By vacuuming the postgres database tables we were able to free up space on the NSX manager common partition from 72% to 62%
  • Also, we truncated tables on NSX manager to being the NSX manager common partition from 62% to 57%
  • This free freed up enough space to continue with NSX upgrade.