Notepad

Quick note on how to fix an issue with Linux machine that has 100% disk usage. After doing it more than 20 times for the last 3 years I decided to put something here:

So, in my case I had CentOS with 100% disk usage for root partition:

[root@hostname ~]# df -h
Filesystem                              Size  Used Avail Use% Mounted on
/dev/mapper/centos_hostname-root   17G   17G   20K 100% /
devtmpfs                                903M     0  903M   0% /dev
tmpfs                                   915M     0  915M   0% /dev/shm
tmpfs                                   915M   89M  826M  10% /run
tmpfs                                   915M     0  915M   0% /sys/fs/cgroup
/dev/xvda1                             1014M  181M  834M  18% /boot
tmpfs                                   183M     0  183M   0% /run/user/0
[root@hostname ~]#

I was luck and quick search for files that are greater than 1GB revealed an issue - standard log messages occupied too much space:

[root@hostname ~]# find /var/log -size +1G -print0 2>/dev/null | xargs -0 ls -lhS
-rw------- 1 root root 8.4G Sep 14 15:08 /var/log/messages-20200914
-rw------- 1 root root 4.8G Sep  6 03:26 /var/log/messages-20200906
[root@hostname ~]#

Quickest solution is to make “logrotate” rules a bit tighter: start compressing historical files, compress more often OR if a file reaches certain size maybe keep less number of them. All I did is modified “/etc/logrotate.conf” by adding the following lines to enable file compression and max file size:

vi /etc/logrotate.conf
compress
maxsize 50M

Then forcibly run logrotate to see the result:

logrotate -fv /etc/logrotate.conf

Next step is to see if/when something changed by looking at the file sizes (files are sorted by date, “-t” option):

[root@hostname ~]# ls -laht /var/log/messages*
-rw-------   1 root    root    8.4G Sep 14 15:07 messages
-rw-------   1 root    root    4.8G Sep  6 03:26 messages-20200906
-rw-------   1 root    root    822K Aug 30 03:49 messages-20200830
-rw-------   1 root    root    807K Aug 23 03:28 messages-20200823
-rw-------   1 root    root    822K Aug 16 03:29 messages-20200816
[root@hostname ~]#

It was easy to notice that something changed after Aug 30th when files size changed drastically. I found what happened by looking at file content and modified “/etc/rsyslog.conf” to fix the root cause, but it’s something too specific that doesn’t apply to everyone, but I hope you learned something.

Good luck!