Comprehensive Guide to Logging in Linux Environments

calendar Oct 20, 2024 · 4 min read · linux aws  ·
Share on: linkedin copy

Rsyslog - Logrotate - AWS Cloudwatch Logs

Rsyslog is a powerful and flexible logging service that is included by default in most Linux distributions. It is used to manage log files generated by various system components, providing the ability to collect, filter, and forward logs to different destinations.

Rsyslog can handle logs from various sources and applications, and it plays an important role in monitoring, troubleshooting, and ensuring system reliability.

In this article, we will cover the basics of Rsyslog, log rotation, and forwarding logs to AWS CloudWatch.

Key Concepts in Rsyslog

Rsyslog uses a configuration file (/etc/rsyslog.conf) where you can define how and where logs should be handled. Logs can be directed to files, databases, or even forwarded to remote servers.

Rsyslog Configuration

Rsyslog configuration is typically handled through the /etc/rsyslog.conf file. The configuration follows a simple format where you specify:

  • Facility: Defines which type of logs to handle (e.g., auth, cron, mail).
  • Priority: Specifies the severity of the logs to handle (e.g., info, crit, debug).
  • Destination: Specifies where the logs should be sent (e.g., a log file, a remote server, etc.).

Example Rsyslog Configuration

 1# Log anything of level info or higher (except mail)
 2*.info;mail.none;authpriv.none;cron.none /var/log/messages
 3
 4# Authpriv logs to secure file
 5authpriv.* /var/log/secure
 6
 7# Log all cron messages to cron file
 8cron.* /var/log/cron
 9
10# Emergency messages to all users
11*.emerg :omusrmsg:*
12
13# Log boot messages to boot.log
14local7.* /var/log/boot.log
15
16# Custom logging for critical messages
17local4.crit /var/log/critandabove.log
18local4.=crit /var/log/crit.log
  • *.info: Logs all facilities (*) at the info level or higher.
  • mail.none: Excludes mail logs.
  • authpriv.*: Logs all authpriv messages to /var/log/secure.
  • *.emerg: Sends emergency messages to all logged-in users using omusrmsg.

Viewing Logs with Rsyslog

You can view log messages stored by Rsyslog using cat or less commands:

1sudo cat /var/log/messages
2sudo less /var/log/secure

Log Rotation with logrotate

Log files grow over time and need to be rotated to prevent them from consuming too much disk space. Logrotate is a utility that automates log rotation, compression, and deletion.

Logrotate Configuration

The global Logrotate configuration is stored in /etc/logrotate.conf. By default, log files are rotated weekly, and four weeks of logs are kept. Below is an example of the Logrotate configuration:

 1# Rotate log files weekly
 2weekly
 3
 4# Keep four weeks of backlogs
 5rotate 4
 6
 7# Create new empty log files after rotation
 8create
 9
10# Date as suffix for rotated files
11dateext
12
13# Include all configuration files in /etc/logrotate.d
14include /etc/logrotate.d
15
16# Specific configuration for wtmp logs
17/var/log/wtmp {
18    monthly
19    minsize 1M
20    create 0664 root utmp
21    rotate 1
22}
  • weekly: Rotates logs on a weekly basis.
  • rotate 4: Keeps four rotations (4 weeks) of logs.
  • create: Creates a new empty file after each rotation.
  • dateext: Adds the current date as a suffix to rotated logs.

Logrotate Example for Firewalld

Firewalld logs can be rotated using the following configuration file in /etc/logrotate.d/firewalld:

1/var/log/firewalld {
2    weekly
3    missingok
4    rotate 4
5    copytruncate
6    minsize 1M
7}
  • missingok: Ignores the log file if it’s missing.
  • copytruncate: Copies the log file and truncates the original file to prevent service interruption.

Forwarding Logs to AWS CloudWatch

You can forward logs from an EC2 instance to AWS CloudWatch using the CloudWatch agent. This is useful for centralized log management and real-time monitoring.

Steps to Forward Logs to CloudWatch

  1. Create IAM Role: Assign an IAM role to the EC2 instance with CloudWatch write permissions.
  2. Install CloudWatch Agent: Install the CloudWatch agent on the instance.

For RedHat-based systems, install the CloudWatch agent as follows:

1curl https://s3.amazonaws.com/amazoncloudwatch-agent/redhat/amd64/latest/amazon-cloudwatch-agent.rpm -O
2sudo rpm -U amazon-cloudwatch-agent.rpm
  1. Configure CloudWatch Agent: You can configure the CloudWatch agent using the wizard:
1sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

This wizard helps set up log collection from various system components, such as system logs, application logs, and custom logs.

Example CloudWatch Agent Configuration for Nginx Logs

Here’s an example configuration for forwarding Nginx logs to CloudWatch:

1[/var/log/nginx/access.log]
2datetime_format = %b %d %H:%M:%S
3file = /var/log/nginx/access.log
4buffer_duration = 5000
5log_stream_name = web-server-01
6initial_position = start_of_file
7log_group_name = webserver-logs
  • file: Specifies the log file path.
  • log_stream_name: The CloudWatch log stream to which logs are sent.
  • log_group_name: The CloudWatch log group for storing logs.

After making changes to the configuration, restart the CloudWatch agent:

1sudo service awslogs restart

Best Practices for Logging

  1. Log Rotation: Use Logrotate to manage and rotate logs, ensuring they don’t consume excessive disk space.
  2. Centralized Logging: Forward logs to a centralized logging service like AWS CloudWatch for better analysis, monitoring, and alerting.
  3. Review Logs Regularly: Check logs for any anomalies or issues, especially for security and performance monitoring.

Conclusion

Rsyslog is a highly efficient and configurable logging utility that allows you to manage logs effectively. Paired with Logrotate, you can ensure your system logs remain under control. Additionally, by forwarding logs to CloudWatch, you can leverage AWS's powerful monitoring and alerting tools for centralized log management.