Setup central syslog server

Problem:

How can I configure several host to forward their syslog messages to a central logging server?

Solution:

rsyslogd supports several transport protocols, which have different benefits and requirements:

User Datagram Protocol (UDP)

This is the traditional protocol, which is standardized by RFC 3195 and used by many systems. It is fast, but messages can get lost in congested networks. By default UDP port 514 is used.

Transmission Control Protocol (TCP)

This is no standardized yet, but implemented by several vendors. Messages normally do not get lost, but this still can occur in certain crash scenarios. Due to network problems or loaded systems the syslog daemon may block and stop processing messages until the problem is resolved. TCP port 10514 is often used.

Reliable Event Logging Protocol (RELP)

This is a rsyslog specific protocol, which is more reliable than TCP and prevents message loss. It still has the drawback that it can block the syslog daemon. Often used TCP ports include 2514 and 20514. The package rsyslog-relp must be installed.

Since UCS-4.1-4 Erratum 353 this can be configured thru several UCR variables.

The instructions in this description doesn’t work with UCS 5.x until the Bug 56055 is fixed, but you could patch the UCR template manually as described.

Sink configuration

On the receiving hosts one (or more) protocols must be enabled. This is done by specifying the port to open:

UDP

ucr set syslog/input/udp=514

TCP

ucr set syslog/input/tcp=10514
RELP (you may need to install the package rsyslog-relp in order to use relp)
ucr set syslog/input/relp=2514

Afterwards the rsyslog daemon and the firewall must be restarted to activate the new configuration:

systemctl restart rsyslog.service
systemctl restart univention-firewall.service

Source configuration

On the sending hosts the receiving host and protocol must be named:

UDP

ucr set syslog/remote=@master.domain.tld

TCP

ucr set syslog/remote=@@master.domain.tld:10514

RELP

ucr set syslog/remote=:omrelp:master.domain.tld:2514

Afterwards the rsyslog daemon must be restarted to activate the new configuration:

systemctl restart rsyslog.service
systemctl restart univention-firewall.service

Advanced Options

Separation by date

Without further instructions, the default path at the Sink configuration will save the syslog messages to:

/var/log/rsyslog/%HOSTNAME%/RemoteLog.log

If you want to be more specific about the syslog messages date, you may add a new config file below /etc/rsyslog.conf.d containing

$template RemoteHost,"/var/syslog/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"

Filtering

By default rsyslog forwards all messages to the remote host. On a busy network this can create too many messages. Therefore the so-called selector can be adjusted to only sent a subset of all messages to the remote server. This can be configured through the UCR variable syslog/remote/selector, which defaults to .. For example

ucr set syslog/remote/selector='mail,auth,authpriv.none;*.warn' 

would filter out all messages from the facilities mail, auth, and authpiv and would further restrict the messages to those of severity warn and above (err, crit, alert, emerg). Messages of priority debug, info, and notice would not get forwarded. See man 5 rsyslog.conf for details on Selectors.

Fallback

If TCP or RELP are used and the receiving server cannot be reached, the sending rsyslog daemon may block. It is possible to configure one (or more) fallback servers, which would then be contacted. This is done through the UCR variable syslog/remote/fallback, which takes a space separated list of servers similar to syslog/remote. For example

 ucr set syslog/remote/fallback='@@backup01.domain.rtld:10514 @@backup02.domain.tld:10514 /var/log/fallback' 

would try the master first, then the two backups through TCP and finally would fall back to writing the message into a local file if none of the servers can be reached.

Encryption

By default messages are forwarded unencrypted! Especially messages from the facilities auth and authpriv may contain sensitive information. rsyslog can be configured to transmit messages encrypted - they are still stored unencrypted in the log files. This only works with TCP, not with UDP and RELP.

This setup is compatible with RFC 5425, which uses TCP port 6514. The package rsyslog-gnutls (currently only available in the unmaintained section of the UCS software repository) must be installed on both sending and receiving systems. The host certificates are used for authentication, so this procedure works on UCS systems out-of-the-box. For other systems certificates can be created on the UCS DC Master using the univention-certificate command - the exact details are out of scope for this article.

On the receiving host create a configuration fragment file in the directory /etc/rsyslog.d/, for example /etc/rsyslog.d/tlsin.conf:


  # make gtls driver the default
  $DefaultNetstreamDriver gtls

  # certificate files
  $DefaultNetstreamDriverCAFile /etc/univention/ssl/ucsCA/CAcert.pem
  $DefaultNetstreamDriverCertFile /etc/univention/ssl/master.domain.tld/cert.pem
  $DefaultNetstreamDriverKeyFile /etc/univention/ssl/master.domain.tld/private.key

  $ModLoad imtcp # load TCP listener

  $InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode

  # Verify the certificate of the sender:
  $InputTCPServerStreamDriverAuthMode x509/name
  $InputTCPServerStreamDriverPermittedPeer *.domain.tld
  # Alternative: do NOT verify client certificate
  #$InputTCPServerStreamDriverAuthMode anon

  $InputTCPServerRun 6514 # finally start up listener at port 6514

As setting up TLS is incompatible with unencrypted TCP, the UCR variable syslog/remote/tcp must be unset; otherwise the encrypted messages will not be decrypted! TCP port 6514 needs to be opened in the firewall by setting the following UCR variables:

  ucr set security/packetfilter/package/rsyslog/tcp/6514/all{=ACCEPT,/en=syslog}

On all sending hosts create a configuration fragment file in the directory /etc/rsyslog.d/, for example /etc/rsyslog.d/tlsout.conf:

  # certificate files
  $DefaultNetstreamDriverCAFile /etc/univention/ssl/ucsCA/CAcert.pem
  $DefaultNetstreamDriverCertFile /etc/univention/ssl/client.domain.tld/cert.pem
  $DefaultNetstreamDriverKeyFile /etc/univention/ssl/client.domain.tld/private.key

  # set up the action
  $DefaultNetstreamDriver gtls # use gtls netstream driver
  $ActionSendStreamDriverMode 1 # require TLS for the connection

  # Verify the certificate of the receiver:
  $ActionSendStreamDriverAuthMode x509/name
  $ActionSendStreamDriverPermittedPeer master.domain.tld
  # Alternatively: do NOT verify server certificate
  #$ActionSendStreamDriverAuthMode anon

The receiving server can be configured normally through the UCR variable syslog/remote, but it is advisable to change the so-called framing-mode to octet-based, e.g.

 ucr set syslog/remote=@@(o)master.domain.tld

Note: Make sure to change all referenced to *.domain.tld to match your domain!

Older UCS releases

For UCS systems before UCS-4.1-4 rsyslog must be configured manually. This can be easily done by adding the following configuration file fragment on the receiving server in a new file located in the directory /etc/rsyslog.d/, for example /etc/rsyslog.d/remote.conf:

  $ModLoad imudp
  $UDPServerRun 514

The firewall must be opened to allow access to UDP port 514, which can be done through two UCR variables:

  ucr set security/packetfilter/package/rsyslog/udp/514/all{=ACCEPT,/en=syslog}

Afterwards rsyslogd and the firewall have to be restarted to activate the new configuration:

  /etc/init.d/rsyslog restart
  /etc/init.d/univention-firewall restart

For all other systems, which should forward their messages to that server, create the following new configuration file fragment in a new file located in the directory /etc/rsyslog.d/, for example /etc/rsyslogd.d/remotelog.conf:

*.* @master.domain.tld

Replace master.domain.tld with the fully qualified name of the host the messages should be forwarded to. Afterward restart the rsyslogd:

  /etc/init.d/rsyslog restart
3 Likes
Mastodon