Python: How to Create Rotating Logs

Python’s logging module has lots of options. In this article, we will looks at the logging module’s ability to create Rotating Logs. Python supports two types of rotating logs:

Rotate logs based on size (RotatingFileHandler)
Rotate the logs based on some time interval (TimedRotatingFileHandler)

Let’s spend some time learning how each of these two types of loggers are implemented and used.

The RotatingFileHandler

The RotatingFileHandler class within the logging module allows the developer to create a logging handler object that gives them the ability to rotate their logs based on the size of the log. You can use the maxBytes parameter to tell it when to rotate the log. This means that when the log reaches a certain number of bytes, it gets “rolled over”. This occurs when the file size is about to be exceeded. The handler will close the file and silently open a new one. If you pass in a number for the backupCount parameter, then it will append “.1”, “.2”, etcetera to the end of the log files. Let’s take a look at a simple example:

import logging
import time

from logging.handlers import RotatingFileHandler

#----------------------------------------------------------------------
def create_rotating_log(path):
    """
    Creates a rotating log
    """
    logger = logging.getLogger("Rotating Log")
    logger.setLevel(logging.INFO)
    
    # add a rotating handler
    handler = RotatingFileHandler(path, maxBytes=20,
                                  backupCount=5)
    logger.addHandler(handler)
    
    for i in range(10):
        logger.info("This is test log line %s" % i)
        time.sleep(1.5)
        
#----------------------------------------------------------------------
if __name__ == "__main__":
    log_file = "test.log"
    create_rotating_log(log_file)

This code is based on an example from the Python Logging Cookbook. Here we create a rotating log with a logging level of INFO. Then we set up the handler to rotate the log whenever the log file is 20 bytes in length. Yes, that’s an absurdly low number, but it makes demonstrating what happens easier. Next we create a loop that will create 10 lines in our log file with a sleep in between each call to log. If you run this code, you should end up with six files: the original test.log and 5 backup logs.

Now let’s look at how to use a TimedRotatingFileHandler.

The TimedRotatingFileHandler

The TimedRotatingFileHandler allows the developer to create a rotating log based on how much time has elapsed. You can set it to rotate the log on the following time conditions:

second (s)
minute (m)
hour (h)
day (d)
w0-w6 (weekday, 0=Monday)
midnight

To set one of these conditions, you just pass it in the when parameter, which is the 2nd argument. You will also want to set the interval parameter too. Let’s take a look at an example:

import logging
import time

from logging.handlers import TimedRotatingFileHandler

#----------------------------------------------------------------------
def create_timed_rotating_log(path):
    """"""
    logger = logging.getLogger("Rotating Log")
    logger.setLevel(logging.INFO)
    
    handler = TimedRotatingFileHandler(path,
                                       when="m",
                                       interval=1,
                                       backupCount=5)
    logger.addHandler(handler)
    
    for i in range(6):
        logger.info("This is a test!")
        time.sleep(75)

#----------------------------------------------------------------------
if __name__ == "__main__":
    log_file = "timed_test.log"
    create_timed_rotating_log(log_file)

This example will rotate the log every minute with a back up count of 5. A more realistic rotation would probably be on the hour, so you would set the interval to 60 or the when to “h”. When this code is run, it too will create 6 files, but instead of appending integers to the log file name, it will append a timestamp using the strftime format %Y-%m-%d_%H-%M-%S.