Raspberry Pi logging issue
Yesterday my Raspberry Pi NAS server.music playout system just stopped working. Logging in all seemed OK, 'ps' showed the processes running, the webserver running the music playout app was serving pages, it just wouldn't play out music...
While typing a command I pressed ‘tab’ to autocomplete, and rather than the expected response I got a message from bash saying it couldn’t create the necessary temporary file. ‘df’ revealed the root file system was 100% used!
Poking around with ‘du’ I tracked it down to the /var/log directory, which was over 2Gb in size! The syslog, kern.log and debug files where massive, and had lines like:
Sep 12 01:21:25 raspberrypi kernel: [ 1137.132839] delay: estimated 0, actual 132 Sep 12 01:21:25 raspberrypi kernel: [ 1137.140846] delay: estimated 0, actual 132 Sep 12 01:21:25 raspberrypi kernel: [ 1137.148843] delay: estimated 0, actual 133 Sep 12 01:21:25 raspberrypi kernel: [ 1137.156843] delay: estimated 0, actual 133 Sep 12 01:21:25 raspberrypi kernel: [ 1137.164835] delay: estimated 0, actual 133
Using tail on kern.log I verified that the lines only appeared when I was playing music. And producing lines at a rate of ~100 per second, which was why my SD card was full! Looking at the archived logs, it seemed that this had only started recently, and I had done an update with apt-get recently! Something had obviously been upgraded with code in to produce these messages.
The easiest fix for this is to suppress the messages. Since I’m not debugging the kernel then I switched off all debug messages by using 'sudo vi /etc/rsyslog.conf' and adding:
just before the first of the ‘catch-all’ rules. The tilde tells the system logger to just discard the messages, and takes effect immediately. I then deleted all the huge logfiles and rebooted. Messages gone!
My only concern is have I killed any useful messages? I’m not sure, but I prefer that to filling up the SD card!
(The playout system failed to play out because I was logging the output from the player, and it didn’t have the space to do that).
Version 3 updated Sept. 12, 2012, 8:46 a.m.