/etc/logrotate.d
-------------------------------------------------------------------------------- Setting up and verifying system logging: syslog and klog; /etc/syslog.conf; remote logging; monitoring logs using swatch; managing logs using log rotate. It is possible to force log rotation to test new configuration. { dbw comment: invalid entries in logrotate.conf - such as logs for customer web directories since deleted, can cause logrotate to fail completely! You _can_ force a log rotation checking your logrotate.conf file for validity. syntax: logrotate -f /etc/logrotate.conf Test without actually rotating and show details: logrotate -vd /etc/logrotate.conf } man logrotate -------------------------------------------------------------------------------- #note: endscript should appear only in the last entry if in logrotate.d # and should not appear at all if in logrotate.conf -------------------------------------------------------------------------------- How logrotate is invoked from cron. On one of my RedHat 5.2 systems, `crond' is started from the script /etc/rc.d/init.d/crond. The manual `man cron' says that cron searches for /etc/crontab and the files in the /etc/cron.d/ directory. My /etc/crontab file contains the following. SHELL=/bin/bash PATH=/sbin:/bin:/usr/sbin:/usr/bin MAILTO=root # run-parts 01 * * * * root run-parts /etc/cron.hourly 02 4 * * * root run-parts /etc/cron.daily 22 4 * * 0 root run-parts /etc/cron.weekly 0 0 1 * * root run-parts /etc/cron.monthly And the /etc/cron.d directory contains nothing. So clearly, any logrotate commands are being run out of one or more of the /etc/cron.* files. The manual `man 5 crontab' indicates that the above commands mean that the program `run-parts' will be run as user `root' for each of the scripts. The command `run-parts' turns out to be the script /usr/bin/run-parts, which is very short, as follows. #!/bin/bash # run-parts - concept taken from Debian # keep going when something fails set +e if [ $# -lt 1 ]; then echo "Usage: run-parts <dir>" exit 1 fi if [ ! -d $1 ]; then echo "Not a directory: $1" exit 1 fi for i in $1/* ; do [ -d $i ] && continue if [ -x $i ]; then $i fi done exit 0 In essence, this just runs all of the scripts in the specified directory. E.g. each hour, the executable plain files in /etc/cron.hourly are run. It turns out that the `logrotate' program is invoked from /etc/cron.daily. This is the contents of the file /etc/cron.daily/logrotate. #!/bin/sh /usr/sbin/logrotate /etc/logrotate.conf This means that the only file that `logrotate' gets its instructions from directly is /etc/logrotate.conf, which contains the following lines. # see "man logrotate" for details # rotate log files weekly weekly # keep 8 weeks worth of backlogs rotate 8 # send errors to root errors root # create new (empty) log files after rotating old ones create # uncomment this if you want your log files compressed #compress # RPM packages drop log rotation information into this directory include /etc/logrotate.d # no packages own lastlog or wtmp -- we'll rotate them here /var/log/wtmp { monthly rotate 1 } # system-specific logs may be configured here This is all explained in the manual `man logrotate'. Operation of logrotate. The directory /etc/logrotate.d contains the following files. -rw-r--r-- 1 root root 354 Oct 13 1998 apache -rw-r--r-- 1 root root 108 Aug 28 1999 cron -rw-r--r-- 1 root root 188 Oct 14 1998 linuxconf -rw-r--r-- 1 root root 156 Oct 13 1998 mgetty -rw-r--r-- 1 root root 327 Aug 12 1998 syslog -rw-r--r-- 1 root root 457 Sep 10 1998 uucp The file /etc/logrotate.d/apache is the one I'm interested in for this exercise. This file contains the following. /var/log/httpd/access_log { postrotate /usr/bin/killall -HUP httpd endscript } /var/log/httpd/agent_log { postrotate /usr/bin/killall -HUP httpd endscript } /var/log/httpd/error_log { postrotate /usr/bin/killall -HUP httpd endscript } /var/log/httpd/referer_log { postrotate /usr/bin/killall -HUP httpd endscript } When I installed the latest version of Apache to get the PHP3 and PostgreSQL to work (around 20 March 2000) on my web server machine, I installed Apache so that the log files were in /home2/apache/logs instead of /var/log/httpd. Therefore what I need to do now is to modify the /etc/logrotate.d/apache file so that the files referred to are all in the directory /home2/apache/logs instead. My new /etc/logrotate.d/apache script is as follows, and I saved the old one in directory /etc/logrotate.d/old1. # The new improved logrotate script for apache on fox. /home2/apache/logs/*-access_log { rotate 9 monthly errors akenning@fox.topology.org create ifempty olddir /home2/apache/logs/oldlogs postrotate /usr/bin/killall -HUP httpd endscript } /home2/apache/logs/*-combref_log { rotate 9 monthly errors akenning@fox.topology.org create ifempty olddir /home2/apache/logs/oldlogs postrotate /usr/bin/killall -HUP httpd endscript } /home2/apache/logs/*-error_log { rotate 9 monthly errors akenning@fox.topology.org create ifempty olddir /home2/apache/logs/oldlogs postrotate /usr/bin/killall -HUP httpd endscript } -------------------------------------------------------------------------------- Now on Sunday 30 September 2001, it's time to get `logrotate' going on my new SuSE 7.1 web server `dog'. The problem is that SuSE 7.1 does not come with logrotate software on it. So I had to go and look for it. The link at redhat was wrong, but by looking around a bit, I finally found it at ftp://ftp.redhat.com/pub/redhat/linux/code/logrotate/ I downloaded this file: logrotate-3.3.tar.gz. This can be compiled very simply with `make' and `make install'. There's a manual logrotate.8 with it too. The binary is installed as /usr/sbin/logrotate. So now all I have to do is write a configuration file for the logrotate program and then write a cron script for it. (Note that since my main initial consideration is to rotate the httpd logs for the beginning of October 2001, I could just use the Apache tool for this purpose, which is /usr/local/apache/bin/rotatelogs on my system. However, I can't understand the documentation for this. So I'm playing safe and using the more flexible redhat `logrotate' tool instead.) On my SuSE 7.1 machine `dog', the manual says that after reading the per-user crontab files in /var/spool/cron/tabs on start-up, the cron process reads the file /etc/crontab. On my machine as configured, I find the following. root@dog# more /etc/crontab SHELL=/bin/sh PATH=/usr/bin:/usr/sbin:/sbin:/bin:/usr/lib/news/bin MAILTO=root #-* * * * * root test -x /usr/sbin/atrun && /usr/sbin/atrun 0 21 * * * root test -x /usr/sbin/faxqclean && /usr/sbin/faxqclean 5 22 * * * root test -x /usr/sbin/texpire && /usr/sbin/texpire 25 23 * * * root test -e /usr/sbin/faxcron && sh /usr/sbin/faxcron | mail FaxMaster # # check scripts in cron.hourly, cron.daily, cron.weekly, and cron.monthly # -*/15 * * * * root test -x /usr/lib/cron/run-crons && /usr/lib/cron/run-crons 59 * * * * root rm -f /var/spool/cron/lastrun/cron.hourly 14 0 * * * root rm -f /var/spool/cron/lastrun/cron.daily 29 0 * * 6 root rm -f /var/spool/cron/lastrun/cron.weekly 44 0 1 * * root rm -f /var/spool/cron/lastrun/cron.monthly There's nothing here to help me with initiating my daily script. (By the way, the fax commands are a bit worrying. I'll get rid of those when I understand exactly what they do. They obviously produce many meaningless message which root receives every day!) So my next step is to look at the files in /etc/cron.d, because the cron manual says that all scripts in this directory are read next. On my machine the only file in /etc/cron.d is a script `seccheck', which produces copious useless messages to root every day and week. (I'll see if I can get rid of that some day too!) The directory /etc/cron.daily contains a script `aaa_base_rotate_logs' which contains a complex set of rotation rules, but how are the scripts in this directory invoked? Hmmm... Maybe they're invoked from that `/usr/lib/cron/run-crons' script. Yes!! That's where it's invoked from. Yet another big, incomprehensible script. The core of that script is the following Bourne-shell loop. SPOOL=/var/spool/cron/lastrun for CRONDIR in /etc/cron.{hourly,daily,weekly,monthly} ; do test -d $CRONDIR || continue BASE=${CRONDIR##*/} test -e $SPOOL/$BASE && { case $BASE in cron.hourly) TIME="-cmin +60 -or -cmin 60" ;; cron.daily) TIME="-ctime +1 -or -ctime 1" ;; cron.weekly) TIME="-ctime +7 -or -ctime 7" ;; cron.monthly) TIME="-ctime +`date -u +%d`" ;; esac eval find $SPOOL/$BASE $TIME | xargs -r rm -f } if test ! -e $SPOOL/$BASE ; then touch $SPOOL/$BASE # keep going when something fails set +e for SCRIPT in $CRONDIR/*[^~,] ; do test -d $SCRIPT && continue test -x $SCRIPT || continue case "$SCRIPT" in *.rpm*) continue ;; *.swap) continue ;; *.bak) continue ;; *.orig) continue ;; \#*) continue ;; esac /sbin/checkproc $SCRIPT && continue nice -15 $SCRIPT done fi done Now what does this mean? The command `BASE=${CRONDIR##*/}' means that BASE is set to the `longest substring of $CRONDIR which matches pattern "*/"', according to my Bash reference card. This just means that the leading path components are removed. (This is done more simply in C-shell!) In the case of the daily cron job, if there is a file /var/spool/cron/lastrun/cron.daily (which is true), then the following command is run. eval find /var/spool/cron/lastrun/cron.daily -ctime +1 -or -ctime 1 | xargs -r rm -f The `xargs' command (which I have never seen before) builds and executes a command line from standard input. Wierd! The `xargs' manual says this. If the standard input does not contain any non-blanks, do not run the command. Normally, the command is run once even if there is no input. So in this case, the command `rm -f' is executed for each of the files with modification times within 24 hours of the current time. I don't really understand this. It looks like the file removal commands in the file /etc/crontab are designed to synchronise the operation of the quarter-hour operations. The commands are only executed if the files in /var/spool/cron/lastrun have been removed. What a convoluted way of achieving a simple objective!! Next any file in the directory /etc/cron.daily which do not end in the characters `~' or `,' (presumed to be edited files) are executed if they are executable and do not have the endings .rpm, .swap, .bak or .orig as follows. /sbin/checkproc $SCRIPT && continue nice -15 $SCRIPT The loop continues is the process is already running. Otherwise it is run with nice level 15. What this all means finally is that any script in the directory /etc/cron.daily will be run at 00:00 on each day. More to the point, the /etc/cron.monthly scripts are run at 00:00 at the beginning of each month. It all looks a bit dodgy because the `date' test is `date -u', which gives the current UTC day. But this should work, although the motivation for the `-u' is not quite clear. All I have to do now is write a script and put it in /etc/cron.monthly. No problem! -------------------------------------------------------------------------------- sample methods: # system-specific logs may be configured here ## rotate apache logs weekly, keeping 8 rotations /www/apache/var/logs/*log { rotate 8 postrotate kill -HUP `cat /www/apache/var/logs/httpd.pid` endscript } ## rotate rembo logs weekly, keeping 4 rotations /opt/rembo/logs/*.log { rotate 4 postrotate /etc/rc.d/init.d/rembo stop /etc/rc.d/init.d/rembo start endscript } -------------------------------------------------------------------------------- the roach method (for virtual hosting): I have many virtual hosts. I feared that apache would be start/stop frequently, once per virtual host, during the rotation process. Therefore, I decided to use the 'copytruncate' option. With this option the apache web server does not require a restart. -roach /usr/apache/log/*log { weekly rotate 52 notifempty missingok copytruncate } /usr/apache/log/dynahost.dbw.org/*log { weekly rotate 52 notifempty missingok copytruncate } -------------------------------------------------------------------------------- Configure the new /etc/logrotate.d/apache file Now Apache logs files residing in the /chroot/var/log/httpd directory instead of /var/log/httpd and for this reason we need to modify the /etc/logrotate.d/httpd file to point to the new chrooted directory. Also, we've compiled Apache with mod_ssl, so we'll add one more line to permit the logrotate program to rotate the ssl_request_log and ssl_engine_log files. Configure your /etc/logrotate.d/apache file to rotate your log files each week automatically. Create the apache file, touch /etc/logrotate.d/apache and add: /chroot/httpd/var/log/httpd/access_log { missingok postrotate /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd endscript } /chroot/httpd/var/log/httpd/error_log { missingok postrotate /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd endscript } /chroot/httpd/var/log/httpd/ssl_request_log { missingok postrotate /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd endscript } /chroot/httpd/var/log/httpd/ssl_engine_log { missingok postrotate /usr/bin/killall -HUP /chroot/httpd/usr/sbin/httpd endscript } -------------------------------------------------------------------------------- Credits and Quasi-bibliography: The majority of this text is from: http://www.topology.org/linux/logrotate.html Reference: Alan Kennington's Some information was gathered from: http://misc.epfl.ch/rembo/logrotate_conf.html Reference: Unknown Securing and Optimizing Linux: RedHat Edition -A Hands on Guide http://www.tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap29sec258.html Anything else was created by: roach date: Sun May 12 18:03:33 CDT 2002 -------------------------------------------------------------------------------- Logrotate: error reading top line of /var/lib/logrotate.status If you get an error saying /etc/cron.daily/logrotate: error: error reading top line of /var/lib/logrotate.status delete /var/lib/logrotate.status, and then run logrotate once with -f flag, like #> logrotate -f /etc/logrorate.d/syslog This should initialize the status file and the error should not be repeating.
Virtual Host Log Rotation for Apache
Many linux distributions include a utility called logrotate.d which is installed and configured by default. It is configured to rotate the main Apache log file typically weekly or monthly. Yet, Apache configuration for virtual hosts (virtual_host) suggests creating separate logs for each virtual host, and in a directory named for the site. This is typically under the primary Apache log directory. Consider a server where /var/log/apache2/access.log is the primary apache log, but there is a virtual host which a log under /var/log/apache2/www.smokinghotbabes.com/access.log. Now looking in /etc/logrotate.d/httpd the default config is:
/var/log/apache2/*.log { missingok notifempty postrotate /usr/bin/killall -HUP httpd endscript }
This default installed configuration for logrotate.d/httpd will only rotate the primary web site. It will not rotate virtual host sites. This can be amended by adding another part to the log rotation configuration script:
/var/log/apache2/*.log { missingok notifempty postrotate /usr/bin/killall -HUP httpd endscript }
/var/log/apache2/*/*.log { missingok notifempty postrotate /usr/bin/killall -HUP httpd endscript }
configuration testing and some important flags
Verbose
-v tells logrotate to say what it’s doing while it’s doing it. It’s very useful when trying to find out why logrotate doesn’t rotate a log when you want it to.
Debug
-d tells logrotate to go through the motions of rotating logs but not actually rotate them. Good for checking that the config file is formatted properly and that logrotate can find the log files it would rotate. It doesn’t actually run the rotations and it doesn’t test some parts of the process like the postrotate scripts.
Force
-f forces logrotate to rotate all logs, whether or not they would normally need to be rotated at that time.
External Resources
- HowTo: The Ultimate Logrotate Command Tutorial with 10 Examples
- Apache logs and logrotate.d
- Understanding logrotate - part 2