A production logging strategy balances critical observability with system performance by implementing efficient log management, retention policies, and optimized performance configurations.
In this guide from PerLod Hosting, you will learn how to design a logging architecture that captures essential data, simplifies troubleshooting, and maintains system stability without compromising your disk space or I/O performance.
Design an Efficient Logging Architecture
The main goal of a production logging strategy is to maintain system observability while preserving server resources. Effective logging enables troubleshooting and monitoring without decreasing application performance or exhausting storage capacity.
Here is the most effective way to design a production logging strategy:
Define Appropriate Log Levels
Production environments should use INFO, WARN, and ERROR log levels as the baseline threshold.
You can use INFO to track key events that help resolve issues and keep DEBUG and TRACE off in production because they create a huge volume of data; only enable them when you need to fix a specific bug.
You can configure log levels in your application or syslog configuration. For rsyslog in the /etc/rsyslog.conf file, you can use:
This setup sends general info to the main log but saves warnings and errors separately so you can find and fix problems faster.
Implement Structured Logging Formats
Structured logging using JSON format makes logs easy for tools to read. This structure lets systems search and analyze data much faster than plain text.
Sending all logs to one central system improves monitoring and reduces disk I/O on your servers. You can use tools like rsyslog or syslog-ng to forward logs to a central logging server.
For example, in /etc/rsyslog.conf file, you can forward all logs to a central server with:
Bash
*.* @@central-log-server.example.com:514
The @@ option uses TCP for reliable transmission, while @ uses UDP for lower overhead.
Production Logging Strategy Retention
Log retention policies determine how long logs are stored and when they are archived or deleted. This prevents your disk from filling up while ensuring you still have the history needed for debugging and compliance.
Configure Log Rotation Frequency
Log rotation frequency depends on log volume and disk I/O capacity. Web servers and high-traffic applications typically require daily rotation, while lower-volume services can use weekly rotation.
For example, create a logrotate configuration in /etc/logrotate.d/application:
This setup rotates logs daily and keeps one week of history. If a file hits 100MB, it rotates immediately, which prevents it from filling disk space before the next scheduled rotation run.
Set Retention Periods by Log Type
Different log types require different retention periods. Access logs require 7 to 30 days of retention, and audit logs require 90 days or more for compliance.
Here is an example for multi-tier retention strategy:
Bash
# Short retention for high-volume access logs/var/log/nginx/access.log{dailyrotate14compressdelaycompresspostrotatenginx-sreopen>/dev/null2>&1endscript}# Extended retention for security logs/var/log/auth.log{weeklyrotate12compressdelaycompress}# Long-term retention for audit logs/var/log/audit/audit.log{weeklyrotate52compress}
Enable Compression for Archived Logs
Compression reduces storage requirements by 70 to 90 % for text-based logs. The compress directive in logrotate uses gzip by default, and delaycompress skips compression for the most recent rotation to avoid conflicts with applications still writing to the file.
This setup utilizes xz compression at the highest level to maximize disk space savings:
Performance Optimization for Production Logging Strategy
Performance optimization prevents logging from disk I/O bottlenecks or CPU contention, especially on VPS Hosting, where storage throughput may be shared. Several strategies minimize I/O impact on application performance, including asynchronous logging, buffering, and proper filesystem configuration.
1. Implement asynchronous logging: Asynchronous logging separates writing logs from running your app. It saves logs to memory first, so your application doesn't stop and wait for the disk to write.
For rsyslog, you can enable queue-based asynchronous processing:
Bash
# /etc/rsyslog.conf$ModLoadimuxsock$ModLoadimjournal# Main message queue configuration$MainMsgQueueTypeLinkedList$MainMsgQueueFileNamemainqueue$MainMsgQueueMaxDiskSpace 1g$MainMsgQueueSize100000$MainMsgQueueDiscardMark90000$MainMsgQueueWorkerThreads4# Action queue for file output$ActionQueueTypeLinkedList$ActionQueueFileNamefilequeue$ActionResumeRetryCount-1$ActionQueueSaveOnShutdownon
This setup holds 100,000 messages in memory and moves to disk only if full. It uses 4 threads to handle the load and drops less important logs if the queue hits 90%. Also, the ActionResumeRetryCount -1 keeps retrying failed writes forever.
2. Optimize buffer sizes: Larger buffers reduce the frequency of disk writes but increase memory usage and potential data loss during crashes. The default rsyslog buffer is 256KB, while application frameworks like Log4j2 use ring buffers of 256KB in standard mode.
You can increase buffer size for high-throughput logging:
Bash
# For rsyslog$MainMsgQueueSize500000# For Log4j2, set in JVM arguments-Dlog4j2.asyncLoggerRingBufferSize=524288
Set buffer sizes to powers of 2, like 128, 256, and 512, for better performance. Monitor your queue; if it stays full, your buffer is too small and needs to be increased.
3. Reduce disk I/O overhead: Disk I/O is the main bottleneck in logging systems. Several strategies minimize I/O impact on application performance, including:
Separate log storage: You can mount /var/log on a separate disk or partition to isolate logging I/O from application data access:
Bash
# Create separate partition for logsmkfs.ext4/dev/sdb1mkdir-p/var/logecho"/dev/sdb1 /var/log ext4 defaults,noatime 0 2">>/etc/fstabmount-a
Use appropriate filesystems: XFS and ext4 perform well for logging workloads. XFS handles large files efficiently and provides better performance for sequential writes.
Monitor I/O performance: Track disk I/O metrics to identify bottlenecks:
Bash
# Monitor disk I/O statisticsiostat-xm1# Identify processes causing high I/Oiotop-oPa
Batch log writes: Configure applications to write logs in batches rather than individual entries:
Bash
# For rsyslog, group messages before writing$ActionFileEnableSyncoff$OMFileFlushInterval10
4. Tune for high-volume environments: High-throughput production environments require additional optimization to handle tens of thousands of messages per second.
You can enable rsyslog's high-performance modules and increase worker threads:
Bash
$ModLoadimuxsock$ModLoadimjournal$IMUXSockRateLimitInterval0$MainMsgQueueWorkerThreads8$MainMsgQueueSize1000000# Use faster output format$ActionFileDefaultTemplateRSYSLOG_FileFormat
Disabling rate limiting with $IMUXSockRateLimitInterval 0 allows rsyslog to process all incoming messages without artificial throttling. Increasing worker threads to 8 enables parallel processing on multi-core systems.
For centralized log servers receiving logs from multiple sources, you can configure input buffering:
This enables TCP input on port 514 with support for 1000 concurrent connections.
Conclusion
A good production logging strategy balances visibility and speed. By setting correct log levels, automating rotation, and using asynchronous processing, you can monitor your servers without slowing them down. Start with these settings to protect your disk performance, then adjust your retention rules as needed.
We hope you enjoy this guide. Subscribe to our X and Facebook channels to get the latest updates and articles.
Use INFO, WARN, and ERROR in production. Only turn on DEBUG when you are actively fixing a problem.
Rotate high-volume logs daily, and lower-volume application logs weekly. Use maxsize to force immediate rotation if a file gets too big before its scheduled time.
Keep access logs for 7 to 30 days, error logs for 30 to 90 days, and audit logs for 90+ days. Archive compressed logs to cheaper storage for extended retention.
Enable asynchronous logging with memory buffers, use separate storage for logs with the noatime mount option, and configure batch writes instead of synchronous I/O.
We use cookies to run PerLod and to understand how the site is used. You can accept everything, or open Manage to turn individual categories off. Cookie policy.