Linux CentOS - 100% disk usage. [SOLVED]
Quick note on how to fix an issue with Linux machine that has 100% disk usage. After doing it more than 20 times for the last 3 years I decided to put something here:
So, in my case I had CentOS with 100% disk usage for root partition:
[root@hostname ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/centos_hostname-root 17G 17G 20K 100% / devtmpfs 903M 0 903M 0% /dev tmpfs 915M 0 915M 0% /dev/shm tmpfs 915M 89M 826M 10% /run tmpfs 915M 0 915M 0% /sys/fs/cgroup /dev/xvda1 1014M 181M 834M 18% /boot tmpfs 183M 0 183M 0% /run/user/0 [root@hostname ~]#
I was luck and quick search for files that are greater than 1GB revealed an issue - standard log messages occupied too much space:
[root@hostname ~]# find /var/log -size +1G -print0 2>/dev/null | xargs -0 ls -lhS -rw------- 1 root root 8.4G Sep 14 15:08 /var/log/messages-20200914 -rw------- 1 root root 4.8G Sep 6 03:26 /var/log/messages-20200906 [root@hostname ~]#
Quickest solution is to make “logrotate” rules a bit tighter: start compressing historical files, compress more often OR if a file reaches certain size maybe keep less number of them. All I did is modified “/etc/logrotate.conf” by adding the following lines to enable file compression and max file size:
vi /etc/logrotate.conf compress maxsize 50M
Then forcibly run logrotate to see the result:
logrotate -fv /etc/logrotate.conf
Next step is to see if/when something changed by looking at the file sizes (files are sorted by date, “-t” option):
[root@hostname ~]# ls -laht /var/log/messages* -rw------- 1 root root 8.4G Sep 14 15:07 messages -rw------- 1 root root 4.8G Sep 6 03:26 messages-20200906 -rw------- 1 root root 822K Aug 30 03:49 messages-20200830 -rw------- 1 root root 807K Aug 23 03:28 messages-20200823 -rw------- 1 root root 822K Aug 16 03:29 messages-20200816 [root@hostname ~]#
It was easy to notice that something changed after Aug 30th when files size changed drastically. I found what happened by looking at file content and modified “/etc/rsyslog.conf” to fix the root cause, but it’s something too specific that doesn’t apply to everyone, but I hope you learned something.
Good luck!