syslog-ng configuration

Last Updated: Nov 16, 2013 02:31PM PST

 

About syslog-ng

syslog-ng can be used to collect local syslog messages & monitor log files on your servers and then forward them to Loggly.

syslog-ng OSE is an open source alternative to the standard syslog daemon that's commonly found on UNIX and UNIX-like (*nix) systems. It uses the basic syslog protocol, but extends it with content-based filtering, flexible configuration options and adds important features, such as using TCP (as well as TLS), which is much more reliable than UDP. syslog-ng OSE is developed by Balabit. Great documentation for advanced configuration is available on their web site, but we'll share a standard config here.

Installation

Using a Package Manager

Depending on your Linux distribution you can use yum or APT (do this with root or sudo privileges):
# apt-get install syslog-ng
You will most likely need to enable Extra Packages for Enterprise Linux (EPEL)
# yum install syslog-ng
Configure syslog-ng, either using our Loggly Syslog Configuration script or manually.

Compiling From Source

Download syslog-ng source code & eventlog source code. Install both eventlog & syslog-ng. Eventlog is a generic event logging library developed by Balabit. Once you've unzipped both packages (evenlog_x.x.xx.tar.gz & syslog-ng-x.xx.tar.gz), do this in each of those directories:
$ ./configure
$ make
$ sudo make install
Configure syslog-ng, either using our Loggly Syslog Configuration script or manually.

Check your syslog-ng version

You'll need to know which version of syslog-ng you've got installed. We recommend running on the latest, but at least version 3.2 for best results.
$ syslog-ng -V

Config for version 3.2 and above

Open your /etc/syslog-ng/syslog-ng.conf file and look for a source with an internal() directive, if it already exists, take note of the source's name. If it doesn't exist, then add the following lines at the bottom of the file:
 ### Syslog Logging Directives for Loggly (myaccount.loggly.com) ###
	source s_loggly {
		system();     # Check which OS & collect system logs
		internal();     # Collect syslog-ng logs
		udp(ip(0.0.0.0) port(514));   # Collect logs sent over UDP
	};
If you'd like to monitor files (for example, your Apache access logs), add as many file() directives as you'd like to the source:
 ### Syslog Logging Directives for Loggly (myaccount.loggly.com) ###
	source s_loggly {
		system();      # Check which OS & collect system logs
		internal();      # Collect syslog-ng logs
		udp(ip(0.0.0.0) port(514));     # Collect logs sent over UDP
		file("/path/to/your/file" follow_freq(1) flags(no-parse));
	};
Continue to the All Versions section.

Below version 3.2

Open your /etc/syslog-ng/syslog-ng.conf file and look for a source with an internal() directive, if it already exists, take note of the source's name. If it doesn't exist, then add the following lines at the bottom of the file:
### Syslog Logging Directives for Loggly (myaccount.loggly.com) ### 
   source s_loggly { 
     internal();                            # Collect syslog-ng logs
     udp(ip(0.0.0.0) port(514));     # Collect logs sent over UDP
     unix-stream("/dev/log");   # Collect Linux system logs
   };
If you'd like to monitor files, the same file directive shown previously can be used here. Continue to the All Versions section.

All versions

Once you've got your source in place, you'll need to specify where to send the logs & how to send them. The template that's shown here makes sure that your logs are RFC5424 compliant, which Loggly requires. Your Customer Token, which needs to be included, is embedded in the template & it's sent with every log event. Create a file called 22-loggly.conf within /etc/syslog-ng/conf.d/ and add the following lines. (Be sure to use your own Customer Token.)
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [012308d8-2b63-4225-8fe9-e12394b6e472@41058] $MSG\n");
template_escape(no);
};


destination d_loggly {
    tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};

log { 
    source(s_loggly); 
    destination(d_loggly); 
};
### END Syslog Logging Directives for Loggly (myaccount.loggly.com) ###
To find your own Customer Token, visit your Source Setup --> Customer Tokens page. We also recommend using "tags" within your syslog header to help with segmentation during searching. Read more about tag usage in our documentation.

Global Options

The default message size for syslog-ng is 8192 bytes, which may not be large enough for some log events, especially if you're logging stack traces or large JSON. In order to increase the maximum message size, add the following within the options block:

options { 
        ...
        log_msg_size(65536); 
        ...
}; 

Restart syslog-ng

$ /etc/init.d/syslog-ng restart

Test your configuration

From your command line, send a test log message:
$ logger "loggly is better than a bee in your aunt's bonnet"
Troubleshooting tips are available.

Quick TLS setup

There are going to be times when you're going to want your logs encrypted during transport. This is where TLS comes in. The quick setup will ensure that your logs go to Loggly encrypted, but it will skip the step where Loggly validates you (which prevents man-in-the-middle attacks). The configuration is similar to above, but for a couple changes within the destination.
template LogglyFormat { template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID} ${MSGID} [026308d8-2b63-4225-8fe9-e01294b6e472@41058] $MSG\n");};

destination d_loggly {
    tcp("logs-01.loggly.com" port(6514) 
    tls(peer-verify(required-untrusted) ca_dir('/opt/syslog-ng/keys/ca.d/'))
    template(LogglyFormat));
};

log { 
    source(s_loggly); 
    destination(d_loggly); 
};
### END Syslog Logging Directives for Loggly (myaccount.loggly.com) ###
You'll need to download Loggly's SSL certificate and the intermediate certificate from Starfield, called sf_bundle.crt. The certificates can be obtained by:
$ wget https://logdog.loggly.com/media/loggly.com.crt
$ wget https://certs.starfieldtech.com/repository/sf_bundle.crt
To verify, go ahead and double check the sha1sum or md5:
$ sha1sum sf_bundle.crt
9f4b50011bdeabda276c9dd08f32f545218ea1b7  sf_bundle.crt
$ md5sum sf_bundle.crt
f742e64a892167bb5b4a10da5a380425  sf_bundle.crt
The sha1 hash is displayed on the Starfield web page where you can also obtain the intermediary cert via the browser. You then need to concatenate both and put it into syslog-ng's CA directory:
$ cat {sf_bundle.crt,loggly.com.crt} > loggly_full.crt
$ mv loggly_full.crt /opt/syslog-ng/keys/ca.d/
Of course, you'll need to restart syslog-ng to see your changes take effect. You may want to start syslog-ng with the -d flag (for debug) so you can get an idea of what's happening. TLS can be a little tricky to get just right.

Parsing Logs

On Linux, one can use patterndb to parse log messages and generate name value pairs from them. These can also be forwarded to Loggly using the JSON output. In the following example, we use the ssh pattern from the BalaBit pattern repository available at https://github.com/balabit/syslog-ng-patterndb It parses username, authentication method, source host, etc. information out from sshd syslog messages and even adds a few more extra information.

Here is the configuration snippet, as usual, please use your own “Customer Token”:

parser p_db { 

db-parser(file("/etc/syslog-ng/ssh.xml")); 

};
template LogglyFormat {
template("<${PRI}>1 ${ISODATE} ${HOST} ${PROGRAM} ${PID}
${MSGID} [abcd1234-aaaa-bbbb-1234-1234abcd1234@41058] $MSG
$(format_json --scope nv_pairs)\n");};
destination d_loggly {
tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log { source(src); parser(p_db);
destination(d_loggly); };

As not all logs are known by patterndb, $MSG is left in the template and JSON output is appended to the end. Once syslog-ng is restarted, it will parse SSH messages and the extra information will be available in the Loggly web interface.

Enter the following query in the search field to find root logins:

json.usracct.username:root

If you want to receive an alert on root logins, first save this query by clicking on the star on the left hand side and choose “Save this search as…”. Once ready, click on the “Alerts” menu at the top of the page, and choose “Add new”. Here you can set different parameters, including which saved search to run.

Sending Through A Central Server

To send logs to a central server before sending them to Loggly, configure your source servers to send to the central one using port 601 as a destination. It can accept logs in a IETF-syslog protocol (as described in RFC5424-28).
destination d_centralserver {
tcp("server.yourdomain.com" port(601) template(LogglyFormat));
};
log { source(s_source);
destination(d_centralserver); };
Then in the central server add the below configuration to accept syslog as a source, and add the Loggly destination.
source s_syslog {
syslog();
};
destination d_loggly {
tcp("logs-01.loggly.com" port(514) template(LogglyFormat));
};
log { source(s_syslog);
destination(d_loggly); };
logglyassistly@zoho.com
https://cdn.desk.com/
false
@loggly
Loading
seconds ago
a minute ago
minutes ago
an hour ago
hours ago
a day ago
days ago
about
false
Invalid characters found
/customer/en/portal/articles/autocomplete