There’s nothing quite like waking up to alerts only to realise your app isn’t down because of a code bug or a failed deployment—
It’s the logs. Again. Just sitting there. Eating your disk like it’s an all-you-can-eat buffet.

If you’ve ever been locked out of your VM or had a Kubernetes pod crash in a puff of “Disk full” smoke, this one’s for you.

Logs are a great way to determine the application health locally, but they are great until they are not. So, let’s talk about the one thing that could’ve prevented this entire mess: Log rotation.

What is Log Rotation?

Log rotation is the practice of automatically compressing, moving, or deleting old log files to free up disk space. This prevents uncontrolled growth of log files that could otherwise fill up the disk, causing:

• Application crashes

• Inaccessible systems

• Failed deployments

• Corrupted databases or services

What it does is archive, compress, and delete the ancient logs based on the configuration.

Real-world Use Case

Scenario: Imagine you run a Python Flask or Node.js app, writing logs to /var/log/myapp/app.log. If you forget to rotate logs, it can grow to multiple GBs, eventually eating up your entire disk (Happens with lots of folks !)

Implementing Log Rotation with logrotate

Install logrotate

Most Linux systems come with it. If not:

sudo apt install logrotate    # Debian/Ubuntu
sudo yum install logrotate    # RHEL/CentOS

Create a Logrotate Config for Your App

Let’s say your log file is at /var/log/myapp/app.log.

Create a new file:

sudo nano /etc/logrotate.d/myapp

Paste this:

/var/log/myapp/app.log {
    daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        systemctl reload myapp.service > /dev/null 2>/dev/null || true
    endscript
}

What does the above gibberish mean?!

• daily: Rotate logs daily , which can also be set to weekly, monthly

• rotate 7: Keep 7 days of logs, and can be adjusted to any number of days as per demand.

• compress: Gzip old logs, as it's better to keep them compressed to save disk space

• notifempty: Don’t rotate if the log is empty.

• postrotate: Reload your service if it needs to reopen the log file

Test It Manually

Force rotate to test:

sudo logrotate -f /etc/logrotate.d/myapp

Check if:

• The current log is rotated to .1

• A new empty app.log is created

• Old logs are compressed as app.log.1.gz

What If It's Already Too Late?

You're staring at a “no space left on device” error. SSH is hanging. App is dead. You are, effectively, cooked.

Quick Steps to resolve:

# Check who the hoarder is
du -sh /var/log/* | sort -rh | head -n 10

# Shrink it before it kills again
truncate -s 0 /var/log/myapp/app.log

# Clear out the compressed junk
rm -rf /var/log/*.gz /var/log/*.1

# Bonus: Docker log file cleansing ritual
truncate -s 0 /var/lib/docker/containers/*/*-json.log

If SSH is dead? Time to pray to your cloud provider and crack open that web console or serial shell.

Still stuck?

Try mounting a temporary disk or using a rescue disk via the cloud console.

Bonus: Add to Cron (if needed)

Logrotate usually runs via cron daily, but you can confirm or add a custom cron:

0 2 * * * /usr/sbin/logrotate /etc/logrotate.conf

Real-World Scenarios Where Log Rotation can be the saviour

Web Server Crashes on a Friday Night

Nginx logs grew unchecked. Access logs are now bigger than your production database. Service won’t restart. You just wanted one quiet weekend.

Fix: Rotate Daily; Kepps 7 Copies, Compress => Done

CI/CD Pipeline Exploded

GitLab/Jenkins pipeline fails at the log collection step because the disk is full. Logs about logs causing failure. Inception, but sad.

Fix: Rotate build logs, keep only the last few runs. Bonus points for compressing.

Kubernetes Pod Died Mid-Deployment

The logs filled up /var/lib/docker. Pod now says “OOMKilled” and refuses to come back like your ex.

Fix: Configure Docker log rotation via daemon.json.

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m", # Size to be 10MB
    "max-file": "5" 
  }
}

Best Practices for Log Rotation

• Always use logrotate For apps with persistent logs

• Compress logs to save space

• Don’t log excessively in debug mode in prod

• Monitor disk space using tools like ncdu, df, or Prometheus + Grafana

• Use centralized logging solutions like ELK, Loki, or CloudWatch for large-scale apps

Conclusion

Log Rotation Isn’t Sexy. It’s Survival.

Uncontrolled logs are like cholesterol: silent, deadly, and preventable.

Setting up logrotate takes minutes. Ignoring it? That could take down your entire infrastructure.

So next time your app starts acting weird, check the logs. But also check the disk.

EzyInfra.dev – Expert DevOps & Infrastructure consulting! We help you set up, optimize, and manage cloud (AWS, GCP) and Kubernetes infrastructure—efficiently and cost-effectively. Need a strategy? Get a free consultation now!