queue max run limit reached

We had a recent issue were cron jobs suddenly faied to run. In troubleshooting the issue we see the following in /var/cron/log.

! c queue max run limit reached Fri Nov 12 19:40:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:40:00 GMT 2004 
! c queue max run limit reached Fri Nov 12 19:40:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:40:00 GMT 2004 
! c queue max run limit reached Fri Nov 12 19:41:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:41:00 GMT 2004 
! c queue max run limit reached Fri Nov 12 19:41:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:41:00 GMT 2004 
! c queue max run limit reached Fri Nov 12 19:42:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:42:00 GMT 2004 
! c queue max run limit reached Fri Nov 12 19:42:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:42:00 GMT 2004 
! c queue max run limit reached Fri Nov 12 19:43:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:43:00 GMT 2004 
! c queue max run limit reached Fri Nov 12 19:43:00 GMT 2004 
! rescheduling a cron job Fri Nov 12 19:43:00 GMT 2004

This indicates that cron attempted to start too many concurrent jobs.

Resolution

You may need to modify the cron queuedefs entry to enable more concurrent jobs or modify your crontab entry to reduce the number of concurrent jobs.

The queuedefs file

The queuedefs file is read by the clock daemon, cron, and controls how jobs submitted using at, batch, and crontab are executed. Every job submitted by one of these commands is placed in a certain queue, and the behavior of these queues is defined by queuedefs.

<q>.#<njob>#<nice>#<nwait>

The fields in this line are:

qThe name of the queue, such that a is the default queue for jobs started by at (see at(1)), b is the queue for jobs started by batch (see at(1)), and c is the queue for jobs run from a crontab file (see crontab(1)). Queue names d through y designate user-defined queues.
njobThe maximum number of jobs that can be run simultaneously in that queue. Although any number can be specified here, cron (see cron(1M)) by default limits the number of jobs that can be run on all the queues to 100. This limitation can be removed by setting the variable DISABLE_MAXJOB_LIMIT to 1 in the /etc/default/cron file.
niceThe nice value to give to all jobs in that queue that are not run with a user ID of super-user (see nice(1)). The default value is 2.
nwaitThe number of seconds to wait before rescheduling a job that was deferred because more than njob jobs were running in that job's queue, or because more than 100 jobs were running in all the queues (see njob above).

Queues are designated by a single lower-case letter. The following queues have special significance:

# cat /etc/cron.d/queuedefs
a.4j1n
b.2j2n90w
c.200j2n90w

This file indicates:

  • The a queue, for at jobs, can have up to 4 jobs running simultaneously; those jobs will be run with a nice value of 1. As no wait value was given, if a job cannot be run because too many other jobs are running cron will wait 60 seconds (default value) before trying again to run it.
  • The b queue, for batch jobs, can have up to 2 jobs running simultaneously; those jobs will be run with a nice(1) value of 2. If a job cannot be run because too many other jobs are running, cron(1M) will wait 90 seconds before trying again to run it.
  • The c queue for cron queue, can have up to 200 jobs running simultaneously; those jobs will be run with a nice value of 2. If a job cannot be run because too many other jobs are running, cron will wait 90 seconds before trying again to run it.

File location

  • Solaris -- /etc/cron.d/queuedefs
  • IBM -- /var/adm/cron/queuedefs
  • HP-UX -- /var/adm/cron/queuedefs
  • SCO OpenServer -- /usr/lib/cron/queuedefs
  • Red Hat -- /var/adm/cron/queuedefs