HOWTO: Python logging

A flexible logging solution is crucial for a FastAPI backend, especially when you need to handle various destinations like files and the ELK stack. Here’s a good approach, along with explanations and code examples, combining best practices for flexibility and maintainability:

1. Use Python’s Built-in logging Module with Configuration:

• Python’s logging module is powerful and well-suited for this task.
• Avoid hardcoding logging settings directly into your application code. Instead, use a configuration file (e.g., YAML, JSON, or even a .conf file) to define your logging setup. This makes it incredibly easy to change logging behavior without modifying your source code.

2. Define a Flexible Logging Configuration:

Here’s a sample logging configuration in YAML format (logging_config.yaml) that demonstrates how to handle multiple handlers (file and ELK stack):

YAML

version: 1
disable_existing_loggers: false
formatters:
  simple:
    format: "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
  json:
    format: '{"timestamp": "%(asctime)s", "logger": "%(name)s", "level": "%(levelname)s", "message": "%(message)s"}'
    # You might need additional fields for ELK like thread, process, etc.

handlers:
  console:
    class: logging.StreamHandler
    level: DEBUG
    formatter: simple
    stream: ext://sys.stdout

  file:
    class: logging.handlers.RotatingFileHandler
    level: INFO
    formatter: simple
    filename: app.log
    maxBytes: 10485760 # 10MB
    backupCount: 5
    encoding: utf8

  logstash:
    class: logstash_async.handler.AsynchronousLogstashHandler
    formatter: json
    level: WARNING
    host: logstash-host # Replace with your Logstash host
    port: 5959 # Replace with your Logstash port
    database_path: logstash_app.db # Path for the handler's internal queue

loggers:
  # Default logger
  "":
    level: DEBUG
    handlers: [console, file, logstash]
    propagate: true

  uvicorn:
    level: INFO
    handlers: [console, file, logstash]
    propagate: false

  uvicorn.error:
    level: WARNING
    handlers: [console, file, logstash]
    propagate: false

  uvicorn.access:
    level: INFO
    handlers: [console, file, logstash]
    propagate: false

Explanation of the Configuration:

• version: 1: Indicates the schema version of the logging configuration file format.
• disable_existing_loggers: false: Prevents this configuration from disabling loggers that might be configured elsewhere (e.g., by libraries you’re using).
• formatters:
  • simple: A basic formatter for human-readable logs (useful for console and file output).
  • json: A formatter that produces JSON output, which is ideal for structured logging and ingestion by systems like the ELK stack. You can customize the fields to include relevant information.
• handlers:
  • console: Sends logs to the console (standard output).
  • file: Writes logs to a file (app.log) with rotation.
    • RotatingFileHandler rotates the log file when it reaches a certain size (maxBytes) and keeps a number of backup files (backupCount).
  • logstash: Sends logs asynchronously to a Logstash instance using the logstash_async library.
    • host and port: Replace with your Logstash instance’s address and port.
    • database_path: The AsynchronousLogstashHandler uses an internal SQLite database to queue messages. Specify a path for this database.
• loggers:
  • "" (root logger): The default logger. Logs from all parts of your application will go through this logger unless a more specific logger is configured.
  • uvicorn, uvicorn.error, uvicorn.access: Specific loggers for different parts of uvicorn logs.
    • propagate: false is used to prevent uvicorn logs from being processed by other loggers. It ensures that specific uvicorn logs are handled independently.

3. Install Required Libraries:

Bash

pip install pyyaml python-logstash-async

4. Load the Configuration and Use the Logger in Your FastAPI App:

Python

import logging.config

import yaml
from fastapi import FastAPI

# Load logging configuration from file
with open("logging_config.yaml", "r") as f:
    config = yaml.safe_load(f.read())
    logging.config.dictConfig(config)

# Get a logger for your application
log = logging.getLogger(__name__)

app = FastAPI()

@app.get("/")
async def read_root():
    log.debug("This is a debug message.")
    log.info("This is an info message.")
    log.warning("This is a warning message.")
    log.error("This is an error message.")
    log.critical("This is a critical message.")
    return {"message": "Hello World"}

5. Docker Compose for ELK (Optional but Recommended):

If you want to test with an ELK stack, using Docker Compose is an easy way to set it up locally:

YAML

# docker-compose.yml (for ELK)
version: '3.7'

services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.17.0 # Choose a version
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false # Disable security for local testing
    ports:
      - 9200:9200

  logstash:
    image: docker.elastic.co/logstash/logstash:7.17.0
    container_name: logstash
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
    ports:
      - 5959:5959
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:7.17.0
    container_name: kibana
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

logstash.conf (for the logstash service):

# logstash.conf
input {
  tcp {
    port => 5959
    codec => json
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "my-fastapi-app-%{+YYYY.MM.dd}"
  }
}

To start the ELK stack:

Bash

docker compose up -d

Key Advantages of This Approach:

• Flexibility: You can easily add, remove, or modify handlers and formatters without changing your application code.
• Centralized Configuration: Manage all your logging settings in a single place.
• Environment-Specific Logging: You can create different configuration files for development, staging, and production environments.
• Structured Logging: The JSON formatter makes your logs easily parseable by machines, which is essential for analysis with the ELK stack.
• Asynchronous Logging to Logstash: The logstash_async handler prevents logging from becoming a bottleneck in your application.

Further Considerations:

• Error Handling: Add error handling around your logging configuration loading to catch potential issues.
• Security: If you are sending logs to a remote ELK stack in a production environment, make sure to secure the connection (e.g., using TLS/SSL) and consider authentication for Logstash.
• Performance: For high-volume logging, monitor the performance of the logstash_async handler and the Logstash instance. You may need to adjust resources or implement more sophisticated queuing mechanisms if necessary.
• Contextual Information: Enrich your logs with additional contextual information (e.g., request IDs, user IDs) to make debugging and analysis easier.

This comprehensive approach provides a solid foundation for managing logs in your FastAPI application, regardless of your current or future logging needs. Please let me know if you have any other questions.