I maintain over a dozen xubuntu (16.04.6) linux servers for my lab. All of them are interconnected via nfs/autofs/ldap services to form a distributed cluster. It has been working fine, but two days ago, my central server providing ldap+home dir nfs services went down. Without having to make a trip to manually reboot the machines, I managed to log in each machine using a maintenance account, and switched ldap/nfs mounts to a backup server. Most of them were brought alive again without a reboot. I forced unmounting the dead nfs mounts (including user home dir) via umount -f -l /mount/point.
However, I noticed that, because the previous nfs mounts were gone, leaving some processes sleep/halt. When I log in these computers, I noticed an unusually high load average number, in the range of 20-100 when typing top, for example:
top - 18:30:46 up 50 days, 23:35, 10 users, load average: 17.59, 17.85, 23.97
Tasks: 494 total, 1 running, 396 sleeping, 0 stopped, 19 zombie
%Cpu(s): 3.0 us, 0.5 sy, 0.4 ni, 95.1 id, 1.0 wa, 0.0 hi, 0.1 si, 0.0 st
KiB Mem : 16382324 total, 1159748 free, 7052016 used, 8170560 buff/cache
KiB Swap: 8458236 total, 7108860 free, 1349376 used. 8296156 avail Mem from the above output, you can see that the load average is high, but CPU is mostly idle. There are 19 zombie programs, and 90% of the processes are sleeping.
another one looks like this
top - 18:56:40 up 18 days, 57 min, 9 users, load average: 25.23, 25.16, 28.13
Tasks: 494 total, 1 running, 377 sleeping, 0 stopped, 40 zombie
%Cpu(s): 1.5 us, 0.4 sy, 0.0 ni, 49.0 id, 49.2 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 32612980 total, 2231320 free, 10522468 used, 19859192 buff/cache
KiB Swap: 8458236 total, 8451836 free, 6400 used. 19531452 avail Mem For one of the machines, the load average grew overnight, from 40ish to over 130 this morning. I found it was due to some cron jobs.
I tried to kill the zombies, but it did not work. I saw this post and successfully killed some of the 'D' (uninterruptible sleep) status processes, that brought the load to around 20-30 range, but still it looks quite high.
I am wondering if there is something I can do to take down these dead/zombie/halt programs. I don't want to do a reboot because it may have the risk not to come back, especially for the alternative ldap/nfs server that I just switch to. These machines are smooth to use despite the high load average. The CPU usage is low, and cpu temperature is low. I am hoping to bring the load back to normal, usually 1 or below 1. Is this possible without a reboot?
Reset to default
