Running out of disk space

I have Ubuntu machine, version 14.04 running on AWS (EC2). Yesterday, my disk space run out of space. Running with user root

df -h

Bring this result:

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 148G 142G 106M 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 3.7G 8.0K 3.7G 1% /dev
tmpfs 748M 332K 748M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.7G 0 3.7G 0% /run/shm
none 100M 0 100M 0% /run/user
overflow 1.0M 32K 992K 4% /tmp

But running this command:

du -ch /

Gave this result:

....
1.5G /
1.5G total

Reboot gave save results. This morning, when I came back to the office AND NO ONE TOUCH THE MACHINE (and there is no batch tasks or processes that write and / or delete files), I run again the commands. I got:

Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 148G 1.6G 140G 2% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 3.7G 8.0K 3.7G 1% /dev
tmpfs 748M 332K 748M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 3.7G 0 3.7G 0% /run/shm
none 100M 0 100M 0% /run/user
overflow 1.0M 32K 992K 4% /tmp

Which make much more sense.

Anyone has explanation why this happened? and why after a night space was free?

Please note that I saw the used disk space grow steadily. And it is the third time it happens.

3 Answers

This is just a glitch/bug.

One possible cause is that a temp file/folder/dir was taking up lots of space, but was not cleared yet. When you checked it in the morning, some process cleared the tempspace.

It's nothing to worry about, and probably was just a glitch in the system.

Based on your comment that it was a "hack attempt", that is probably the best answer. During the hack, some glitch or bug was introduced. It also may have just been a large amount of temp files (think memory overflow). Nothing to worry about in the long run.

Are you the only person who can login this box?

Why I ask this question is, it looks you are running an application which generate huge log file or other files (for example, turn on the debug option in applications, it fills the space very quickly). Then one guy in your team who delete that file directly, but the space does not released, until you restart the application or reboot the box.

Does that make sense for you?

I guess you have already deleted files but they're still open by some program.

Like nginx access.log file. You can delete it, but if the nginx isn't rotated, or restarted. The file will remain.

Try to find all deleted files which is still open.

sudo lsof | grep '(deleted)'

If you find files that are marked as '(deleted)', just restart the program using that files.