Syslog

Overview

Syslog, also known as legacy-syslog or BSD-syslog, is a logging framework started in 1980 and first defined in RFC3164. The current and up-to-date version of syslog is often referred to as IETF-syslog and is defined in RFC5424. While journald (part of systemd) is a more modern approach to logging, that address some of syslog shortcomings, syslog (or one of its variation) is still in use today on many Linux distributions.

syslog logs events / messages from various system components, notably on:

• Successful authentication and failed authentication attempts.
• sudo usage.
• Kernel and hardware component activities.
• cron jobs and mail activity (sendmail).
• …

The syslog message format differs between RFC3164 and RFC5424, with both versions still in use today.

syslog-ng and rsyslog

syslog-ng, started in 1998, extends the initial syslog protocol with new features, such as transport over TCP (instead of UDP only initially), TLS encryption, advanced filtering, etc. syslog-ng uses a different configuration file format than syslog.

rsyslog, started in 2004, also extends the initial syslog protocol with additional features, such as rule based filtering, incoming messages filtering and restrictions, etc. rsyslog uses the same configuration file format as syslog.

Syslog, syslog-ng, and rsyslog continue to be developed as three separate projects, growing separately and in parallel.

syslog Facility, Severity, and Priority

syslog relies on Facility to classify and separate log messages from their originating subsystem / components (kernel, cron, user, mail, …). The Facility are numbered, from 0 and 23 inclusive, with the levels ranging from 16 to 22 reserved for locally used facilities.

The facilities are described in RFC5424:

Note however that the mapping between a facility code and a keyword is not uniform in different Linux distributions or syslog implementations.

The syslog.h interface only allows specific Facilities: user (1) and local0 to local7 (16 to 23). However, as syslog text-based log files are not protected against tampering, it is possible to manually edit the log files to add arbitrary messages in a log file normally linked to a specific Facility.

syslog messages are associated with a Severity level, ranging from 0 (emergency) to 7 (debug). Messages can be filtered by their Severity level, with usually all lower levels kept when filtering for a given level (i.e with levels 0, 1, 2, 3, and 4 kept if filtering on level 4 Warning).

Facility and Severity levels can be mapped to log files in syslog configuration files to log syslog messages to disk. A number of mappings are usually configured by default depending on the Linux distribution. Those default mappings can however change depending on the Linux distribution and syslog version in use.

The Priority value of a message is based on the message Facility and Severity level: Priority = Facility * 8 + Severity. Priority values thus range between 0 and 191 included. The Priority is included in (BSD) syslog: <PRI>HEADER MESSAGE. The Priority is however often omitted in the messages in log files on disk.

Configuration files

syslog configuration file locations differ depending on the Linux distribution and syslog implementation. syslog.conf is usually the main configuration file for syslog.

The following files and directories are common syslog configuration locations:

• /etc/syslog.conf
• /etc/rsyslog.conf
• /etc/rsyslog.d/*.conf
• /etc/syslog-ng.conf
• /etc/syslog-ng/*

Each line in a syslog configuration file defines a rule, with each rule being composed of one or multiple selector fields (each separated by a semicolon (‘;’)) and an action field. The selector and action fields are separated by one or more spaces or tabs.

The selector field consists of two parts: a facility and a priority, separated by a period (‘.’). The facility and priority can be replaced by an asterisk, to match all facilities or all priorities respectively. Multiple facilities and priorities can be specified for in a single selector, using a comma-separated list (example: auth,authpriv.* ). All messages of the specified priority, and higher, for a given facility are logged according to the given action. The keyword none stands for no priority of the given Facility, i.e to prevent logging of messages associated with the Facility (example: *.*;auth,authpriv.none).

The action field of a rule defines how and where the messages matching the rule selector are logged. The action field is usually a file path, but may also be a remote host (@<HOSTNAME>), a named pipe, a console, or a user / users list (to notify users using the wall feature). The file path can be prefixed with the minus (‘-‘) sign to omit syncing the file after every new message, for performance reasons.

Example of a syslog configuration file:

auth,authpriv.*			 /var/log/auth.log
*.*;auth,authpriv.none		-/var/log/syslog

cron.*				/var/log/cron.log
kern.*				-/var/log/kern.log
mail.*				-/var/log/mail.log

Default log files

The syslog log file locations are fully dependant on the syslog configuration, and the configuration files should thus first be reviewed. Additionally, Linux distribution and syslog implementation leverage different default configurations, with different default log file locations.

Information of interest

Syslog message formats

The syslog message format depend on the syslog / rsyslog version in use:

• RFC3164 (legacy-syslog / BSD-syslog) format:

  <Mmm dd hh:mm:ss> <HOSTNAME> <PROCESS>[<PID>]: <MESSAGE>

  Example:

  Jan 29 08:07:28 hostname-01 sshd[1715]: Accepted publickey for username from 10.1.2.3 port 33830 ssh2: RSA SHA256:XXX

• RFC5424 (IETF-syslog) format:

  <VERSION> <TIMESTAMP_ISO_8601> <HOSTNAME> <APP_NAME> <PID> <MSG_ID> [<MESSAGE_STRUCTURED_DATA>] <MESSAGE>

  The MESSAGE_STRUCTURED_DATA field contains a list of key-value pairs, for easier parsing.

In both formats, the <MESSAGE> is freely specified by the originating process and does not follow a standard.

auth facility

Jan 31 18:07:28 hostname sshd[1715]: Accepted publickey for user from 1.2.3.4 port 33830 ssh2: RSA SHA256:XXX
Jan 31 18:07:28 hostname sshd[1715]: pam_unix(sshd:session): session opened for user user by (uid=0)

Aug  4 08:33:44 hostname sshd[2498]: Invalid user admin from 1.2.3.4 port 45843
Aug  4 08:33:44 hostname sshd[2498]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=1.2.3.4

2024-07-23T23:08:09.264429-01:00 hostname sshd[1748729]: Failed password for root from 1.2.3.4 port 35190 ssh2

authpriv facility

Aug  3 10:22:41 hostname sudo[33630]: user : TTY=pts/0 ; PWD=/home/user ; USER=root ; COMMAND=/bin/bash
Aug  3 10:22:41 hostname sudo[33630]: pam_unix(sudo-i:session): session opened for user root by (uid=0)
Aug  3 10:36:35 hostname sudo[33630]: pam_unix(sudo-i:session): session closed for user root

Aug  4 02:59:18 hostname useradd[2979]: new user: name=new_user, UID=1001, GID=1001, home=/home/new_user, shell=/bin/bash

Aug  4 02:58:05 hostname useradd[2979]: new group: name=new_group, GID=1001

Aug  4 05:53:11 hostname passwd[2934]: pam_unix(passwd:chauthtok): password changed for user

Aug  4 02:59:40 hostname usermod[2999]: add 'tuser' to group 'wheel'
Aug  4 02:59:40 hostname usermod[2999]: add 'tuser' to shadow group 'wheel'

cron facility

Aug  4 08:32:07 hostname run-parts[2473]: (/etc/cron.daily) starting logrotate
Aug  4 08:32:07 hostname run-parts[2473]: (/etc/cron.daily) finished logrotate

Aug  4 10:30:01 hostname CROND[25488]: (root) CMD (/root/.script.sh)

kern facility

The kern Facility is used by the Linux kernel to emit messages. While the kernel messages are usually of limited forensics interest (depending on the investigation type), the following information may be found in kern logs:

• Kernel version and command line.

• init parameters.

• Details on hardware components (CPU, physical RAM, PCI bus and devices, etc.) and storage devices (SATA, NVMe, etc.).

• Basic information on Ethernet interfaces (notably including name and MAC address).

• Details on USB devices plugged (notably including vendor and product identifiers, serial number).

• …

For an easier analysis, it is possible to separate kernel messages generated during the system boot from the messages generated during system operations (that can indicate changes to the initial system configuration).

References

• RFC3164

• RFC5424

• serverfault - What is the difference between syslog, rsyslog and syslog-ng?

• Quest - Syslog message formats (4282913)

• Solarwinds - The syslog protocol - Legacy

• Open Group Base Specifications Issue 6 - syslog.h

• Linux die - syslog.conf

• AskUbuntu - Difference between /var/log/messages, /var/log/syslog, and /var/log/kern.log?