NSX-T - Disk Space Requirements Not Met

Upgrading NSX-T in the lab, the Host Pre-Check came back with:

Disk space requirements not met. Please ensure 180MB free space is available in tmp partition to proceed with upgrade.

Disk Space Requirements Not Met

The usual reaction is to check disk usage on ESXi:

[root@esxi:~] df -h
Filesystem Size   Used Available Use% Mounted on
NFS        7.0T   5.2T      1.8T  74% /vmfs/volumes/NFS2019
NFS        7.0T   5.2T      1.8T  74% /vmfs/volumes/NFS1
vfat       4.0G 208.9M      3.8G   5% /vmfs/volumes/BOOTBANK1
vfat       4.0G 202.9M      3.8G   5% /vmfs/volumes/BOOTBANK2
vsan       1.8T   1.3T    532.9G  71% /vmfs/volumes/vsanDatastore
[root@esxi:~]

But where is /tmp mounted?

I’m booting from a USB stick, which sets /tmp to ramdisk.

To view the ramdisk, use vdf -h

[root@esxi:~] vdf -h
[...Output omitted...]
-----
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        6M       25M  21% --
etc                        28M      760K       27M   2% --
opt                        32M        1M       30M   5% --
var                        48M       13M       34M  28% --
tmp                       256M       88M      167M  34% --
iofilters                  32M        0B       32M   0% --
shm                      1024M        0B     1024M   0% --
crx                      1024M        0B     1024M   0% --
configstore                32M      132K       31M   0% --
configstorebkp             32M      132K       31M   0% --
hostdstats                553M        5M      547M   0% --
nestdb                    512M       34M      477M   6% --
nsx-idps                   64M        2M       61M   3% --
[root@esxi:~]

And here we are. /tmp has 167M available.

For more verbose output:

[root@esxi:/var/log] esxcli system visorfs ramdisk list
Ramdisk Name    System  Include in Coredumps   Reserved      Maximum       Used   Peak Used   Free  Reserved Free  Maximum Inodes  Allocated Inodes  Used Inodes  Mount Point
--------------  ------  --------------------  ---------  -----------  ---------  ----------  -----  -------------  --------------  ----------------  -----------  -----------
root              true                  true  32768 KiB    32768 KiB   6916 KiB    6920 KiB   78 %           78 %           18944              6848         6837  /
etc               true                  true  28672 KiB    28672 KiB    760 KiB     828 KiB   97 %           97 %            4096              1024          848  /etc
opt               true                  true      0 KiB    32768 KiB   1656 KiB    1656 KiB   94 %            0 %            8192              1024          751  /opt
var               true                  true   5120 KiB    49152 KiB  13888 KiB   13952 KiB   71 %            0 %            8192              1312         1286  /var
tmp              false                 false   2048 KiB   262144 KiB  90948 KiB  111736 KiB   65 %            0 %            8192               288           99  /tmp
iofilters        false                 false      0 KiB    32768 KiB      0 KiB       0 KiB  100 %            0 %           10240                32            1  /var/run/iofilters
shm              false                 false      0 KiB  1048576 KiB      0 KiB       0 KiB  100 %            0 %             512                32            1  /var/run/shm
crx              false                 false      0 KiB  1048576 KiB      0 KiB       0 KiB  100 %            0 %            8192                32            2  /var/run/crx
configstore      false                 false  32768 KiB    32768 KiB    132 KiB     160 KiB   99 %           99 %            8192                32            3  /etc/vmware/configstore
configstorebkp   false                 false  32768 KiB    32768 KiB    132 KiB     136 KiB   99 %           99 %            8192                32            3  /var/lib/vmware/configstore/backup
hostdstats       false                 false      0 KiB   566272 KiB   5200 KiB    5200 KiB   99 %            0 %            8192                32            4  /var/lib/vmware/hostd/stats
nestdb           false                 false      0 KiB   524288 KiB  35512 KiB   44556 KiB   93 %            0 %            8192                32           10  /var/lib/vmware/nsx/nestdb/db
nsx-idps         false                 false      0 KiB    65536 KiB   2324 KiB    2324 KiB   96 %            0 %            8192                32            2  /etc/nsx-idps/rules
[root@esxi:/var/log]

Simply by deleting files from /tmp you could fix the problem. But which files?

For ESXi 7.x installed on a USB stick, you’ll have the scratch partition at: /tmp/_osdataxxxxxxx/. But don’t delete files that you don’t know what they do. To find out what folder is using up disk space, run [root@esxi:~] du -h

In my case, the NSX nestdb had grown quite large. Rather than attempt to delete enough benign files, a reboot resolved the issue.

As a comparison in disk space usage:

Before Reboot After Reboot
[root@esxi:/var] du -h
34.7M ./lib/vmware/nsx/nestdb/db
34.7M ./lib/vmware/nsx/nestdb
34.7M ./lib/vmware/nsx
39.9M ./lib/vmware
51.9M ./lib
13.5M ./log
68.3M .
[root@esxi:/var]
[root@esxi:/var] du -h
20.0K ./lib/vmware/nsx/nestdb/db
24.0K ./lib/vmware/nsx/nestdb
28.0K ./lib/vmware/nsx
2.8M ./lib/vmware
14.8M ./lib
708.0K ./log
18.3M .
[root@esxi:/var]

And to confirm /tmp:

[root@esxi:~] vdf -h
[...Output omitted...]
Ramdisk                   Size      Used Available Use% Mounted on
root                       32M        6M       25M  20% --
etc                        28M      728K       27M   2% --
opt                        32M        1M       30M   5% --
var                        48M      696K       47M   1% --
tmp                       256M        3M      252M   1% --
iofilters                  32M        0B       32M   0% --
shm                      1024M        0B     1024M   0% --
crx                      1024M        0B     1024M   0% --
configstore                32M      136K       31M   0% --
configstorebkp             32M      136K       31M   0% --
hostdstats                553M        2M      550M   0% --
nestdb                    512M       16K      511M   0% --
nsx-idps                   64M        0B       64M   0% --
[root@esxi:~]

Now there’s enough space to continue the NSX-T upgrade.

Andrew Dauncey
Andrew Dauncey
Senior Consultant at VMware PSO

Every day I’m shuffling

Previous

Related