Problem:
How can I configure several host to forward their syslog messages to a central logging server?
Solution:
rsyslogd supports several transport protocols, which have different benefits and requirements:
User Datagram Protocol (UDP)
This is the traditional protocol, which is standardized by RFC 3195 and used by many systems. It is fast, but messages can get lost in congested networks. By default UDP port 514 is used.
Transmission Control Protocol (TCP)
This is no standardized yet, but implemented by several vendors. Messages normally do not get lost, but this still can occur in certain crash scenarios. Due to network problems or loaded systems the syslog daemon may block and stop processing messages until the problem is resolved. TCP port 10514 is often used.
Reliable Event Logging Protocol (RELP)
This is a rsyslog specific protocol, which is more reliable than TCP and prevents message loss. It still has the drawback that it can block the syslog daemon. Often used TCP ports include 2514 and 20514. The package rsyslog-relp must be installed.
Since UCS-4.1-4 Erratum 353 this can be configured thru several UCR variables.
The instructions in this description doesn’t work with UCS 5.x until the Bug 56055 is fixed, but you could patch the UCR template manually as described.
Sink configuration
On the receiving hosts one (or more) protocols must be enabled. This is done by specifying the port to open:
UDP
ucr set syslog/input/udp=514
TCP
ucr set syslog/input/tcp=10514
ucr set syslog/input/relp=2514
Afterwards the rsyslog daemon and the firewall must be restarted to activate the new configuration:
systemctl restart rsyslog.service
systemctl restart univention-firewall.service
Source configuration
On the sending hosts the receiving host and protocol must be named:
UDP
ucr set syslog/remote=@master.domain.tld
TCP
ucr set syslog/remote=@@master.domain.tld:10514
RELP
ucr set syslog/remote=:omrelp:master.domain.tld:2514
Afterwards the rsyslog daemon must be restarted to activate the new configuration:
systemctl restart rsyslog.service
systemctl restart univention-firewall.service
Advanced Options
Separation by date
Without further instructions, the default path at the Sink configuration will save the syslog messages to:
/var/log/rsyslog/%HOSTNAME%/RemoteLog.log
If you want to be more specific about the syslog messages date, you may add a new config file below /etc/rsyslog.conf.d containing
$template RemoteHost,"/var/syslog/hosts/%HOSTNAME%/%$YEAR%/%$MONTH%/%$DAY%/syslog.log"
Filtering
By default rsyslog forwards all messages to the remote host. On a busy network this can create too many messages. Therefore the so-called selector can be adjusted to only sent a subset of all messages to the remote server. This can be configured through the UCR variable syslog/remote/selector, which defaults to .. For example
ucr set syslog/remote/selector='mail,auth,authpriv.none;*.warn'
would filter out all messages from the facilities mail, auth, and authpiv and would further restrict the messages to those of severity warn and above (err, crit, alert, emerg). Messages of priority debug, info, and notice would not get forwarded. See man 5 rsyslog.conf for details on Selectors.
Fallback
If TCP or RELP are used and the receiving server cannot be reached, the sending rsyslog daemon may block. It is possible to configure one (or more) fallback servers, which would then be contacted. This is done through the UCR variable syslog/remote/fallback, which takes a space separated list of servers similar to syslog/remote. For example
ucr set syslog/remote/fallback='@@backup01.domain.rtld:10514 @@backup02.domain.tld:10514 /var/log/fallback'
would try the master first, then the two backups through TCP and finally would fall back to writing the message into a local file if none of the servers can be reached.
Encryption
By default messages are forwarded unencrypted! Especially messages from the facilities auth and authpriv may contain sensitive information. rsyslog can be configured to transmit messages encrypted - they are still stored unencrypted in the log files. This only works with TCP, not with UDP and RELP.
This setup is compatible with RFC 5425, which uses TCP port 6514. The package rsyslog-gnutls (currently only available in the unmaintained section of the UCS software repository) must be installed on both sending and receiving systems. The host certificates are used for authentication, so this procedure works on UCS systems out-of-the-box. For other systems certificates can be created on the UCS DC Master using the univention-certificate command - the exact details are out of scope for this article.
On the receiving host create a configuration fragment file in the directory /etc/rsyslog.d/, for example /etc/rsyslog.d/tlsin.conf:
# make gtls driver the default
$DefaultNetstreamDriver gtls
# certificate files
$DefaultNetstreamDriverCAFile /etc/univention/ssl/ucsCA/CAcert.pem
$DefaultNetstreamDriverCertFile /etc/univention/ssl/master.domain.tld/cert.pem
$DefaultNetstreamDriverKeyFile /etc/univention/ssl/master.domain.tld/private.key
$ModLoad imtcp # load TCP listener
$InputTCPServerStreamDriverMode 1 # run driver in TLS-only mode
# Verify the certificate of the sender:
$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.domain.tld
# Alternative: do NOT verify client certificate
#$InputTCPServerStreamDriverAuthMode anon
$InputTCPServerRun 6514 # finally start up listener at port 6514
As setting up TLS is incompatible with unencrypted TCP, the UCR variable syslog/remote/tcp must be unset; otherwise the encrypted messages will not be decrypted! TCP port 6514 needs to be opened in the firewall by setting the following UCR variables:
ucr set security/packetfilter/package/rsyslog/tcp/6514/all{=ACCEPT,/en=syslog}
On all sending hosts create a configuration fragment file in the directory /etc/rsyslog.d/, for example /etc/rsyslog.d/tlsout.conf:
# certificate files
$DefaultNetstreamDriverCAFile /etc/univention/ssl/ucsCA/CAcert.pem
$DefaultNetstreamDriverCertFile /etc/univention/ssl/client.domain.tld/cert.pem
$DefaultNetstreamDriverKeyFile /etc/univention/ssl/client.domain.tld/private.key
# set up the action
$DefaultNetstreamDriver gtls # use gtls netstream driver
$ActionSendStreamDriverMode 1 # require TLS for the connection
# Verify the certificate of the receiver:
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer master.domain.tld
# Alternatively: do NOT verify server certificate
#$ActionSendStreamDriverAuthMode anon
The receiving server can be configured normally through the UCR variable syslog/remote, but it is advisable to change the so-called framing-mode to octet-based, e.g.
ucr set syslog/remote=@@(o)master.domain.tld
Note: Make sure to change all referenced to *.domain.tld to match your domain!
Older UCS releases
For UCS systems before UCS-4.1-4 rsyslog must be configured manually. This can be easily done by adding the following configuration file fragment on the receiving server in a new file located in the directory /etc/rsyslog.d/, for example /etc/rsyslog.d/remote.conf:
$ModLoad imudp
$UDPServerRun 514
The firewall must be opened to allow access to UDP port 514, which can be done through two UCR variables:
ucr set security/packetfilter/package/rsyslog/udp/514/all{=ACCEPT,/en=syslog}
Afterwards rsyslogd and the firewall have to be restarted to activate the new configuration:
/etc/init.d/rsyslog restart
/etc/init.d/univention-firewall restart
For all other systems, which should forward their messages to that server, create the following new configuration file fragment in a new file located in the directory /etc/rsyslog.d/, for example /etc/rsyslogd.d/remotelog.conf:
*.* @master.domain.tld
Replace master.domain.tld with the fully qualified name of the host the messages should be forwarded to. Afterward restart the rsyslogd:
/etc/init.d/rsyslog restart