Syslog & network devices & firewalls

By /

I want to go over some basic logging [for network devices] & syslog. Why it’s important and how easy it is to set up.
No, I’m not talking about your SIEM of choice, but a simple ubuntu based syslog for networking devices.

The post is going to be split into two sections:

  • Why – Why do we need syslog? Why is it important? Why is it set to debug? Why not just gather logs when you need them?
  • How – A very basic example using Ubuntu Linux

The WHY – Why why why do I think it’s important for everyone to at least have basic syslog & text-based logs available.

I’m sometimes told one or all of the following items:

  • if you don’t have a solution that actively goes through your logs and highlights and reports potential issues, or can detect anomalies, there isn’t any point in having syslogs at all.
  • “if I need logs, I can just view the logs live or enable the logs when I need them”
  • I should not log connection events from the firewall, having only denied connections is sufficient.

While there’s rarely a “right or wrong” opinion on anything in tech, for many environments these viewpoints are somewhat shortsighted.
Don’t get me wrong, if you have the budget to put all of your logs in a commercial SIEM or a log-analytics solution, by all means do so.
But even large companies can be selective about which logs go into their SIEM of choice due to pricing.

Having text-based logs is better than to have no logs at all!

How often do you…

  • Analyze a problem before it happens?
  • Troubleshoot a connectivity issue before it happens or anyone complains?
  • Try to find the root cause of an issue before it happens?
  • List up all connections that a server made in the future?

There are all kinds of systems that can predict failure and monitor and alert when something happens.
But there will always be incidents, big or small, and you will want to have logs available both to troubleshoot and scope the issue.
Let’s look a few examples:

  • Looking at logs from network equipment moments before it crashes (and buffer clears)
  • Looking at logs from switches to view issues with link flaps or ports disabling.
  • Looking at logs from firewalls to determine if a connection was made, and if the server did respond.
    • (syn timeout, were there received bytes, was the connections topped by IPS, was it denied?)
    • Quite often the question you get may be along the lines of:
      • “yesterday at 13:15 a server with IP 192.168.6.8 tried to connect to 192.0.4.6 on tcp port 443 but it didn’t work, did the firewall stop this request?”
      • Can you verify that this connection was indeed made? Can you verify the correct port was used? Did the firewall block or allow? Was the traffic NAT-ed? Was the connection successful? Is the source-ip of the server correct or did the connection come from another server?
  • A security incident happened with server DMZSRV01 at IP address 172.16.3.2, and we need to know all connections made from this server in the last 90 days to try to establish the scope of the incident and if the attacker was able to pivot to internal resources.

There are also a number of things you can get from logs, that you might not easily get otherwise.
For example, if you have a Cisco ASA VPN gateway, and someone asks you to provide a list of everyone that has authenticated using VPN in the last 90 days.
Now, ideally the VPN gateway should be able to provide this, but the ASA doesn’t, and the authentication source (AD, NPS, ISE, LDAP, etc) should also be able to provide this, but this information might be difficult to extract depending on what the backend authentication server is.

There are plenty of companies that live happily without having any logs.
But when I work with those companies, I can see that they’re getting sub-standard service from their internal IT or VAR.
Troubleshooting takes 4x longer and requested information is never available.

The HOW – A simple practical example, with an optional addon.

There are many types of syslog servers, and depending on your environment you might op for a windows based service or program.
While the capabilities between Windows and Linux get a bit blurry with the use of Windows Subsystem for Linux (WSL) on Windows, my personal preference is to have a linux server for this purpose.

A few years ago my Linux flavor of choice was CentOS, but with the changes RedHat made to CentOS, and now IBM to RedHat, my choice today would be Ubuntu.
You only need the latest, “minimal” install option to get started.
(23.04 at the time of writing)
And here’s how I would start out

  • Install latest ubuntu minimal LTS
  • decide that all my logs will be put into /var/log/network/YYYY/MM/DD/file
    • filename is based on the source IP address for the syslog messages without any DNS lookup.
    • I’ll say this again for those in the back, no DNS lookup.
  • in /etc/rsyslog.conf I’ll uncomment or add the following lines:
# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

$FileOwner syslog
$FileGroup syslog
  • I’ll create a new file, called “10-network.conf” in /etc/rsyslog.d/ with the following content:
$template DynaFile,"/var/log/network/%$YEAR%/%$MONTH%/%$DAY%/%fromhost-ip%"
*.* -?DynaFile

Now, what’s going to happen is that if a network device with the IP address of 172.16.3.1 sends a syslog to my server on the 23rd of July 2023, the log message will be put into the following file:
/var/log/network/2023/07/23/172.16.3.1

Since some logs may get a bit large during the day, especially from next gen firewalls, I create a new file in the /usr/local/bin directory named “logzip.sh”, so I have /usr/local/bin/logzip.sh with the following content:

#!/bin/sh

/usr/bin/find /var/log/network/ ! -name "*bz2" ! -path "*`/bin/date +%Y/%m/%d`*" -exec /usr/bin/bzip2 {} \;

And I add it into cron, using the command “crontab -e” as a user that has write privileges to /var/log/network, so the crontab entry looks like:

19 0 * * * /bin/bash /usr/local/bin/logzip.sh

What the above config does is it will search for all filenames in the /var/log/network directory, but it will exclude the path of the current date, and will exclude filenams that end with *bz2, and it will the compress them using bzip2, so that 172.16.3.1 ends up as 172.16.3.1.bz2

The bz2 script will run once a day, at 19 minutes past midnight.
There are plenty of online resources on how to work with crontabs, for example crontab.guru

If, at a later stage someone asks that all logs should be sent to a SIEM solution, you can choose between modifying the networking equipment, or simply adding the following line to the /etc/rsyslog.d/10-network.conf file, if in this example the SIEM solution has an IP address of 192.168.5.14:

*.* action(type="omfwd" target="192.168.5.14" port="514" protocol="tcp")

Now you will get all of your syslogs in text files to work with and to give internal or external network admins access to, but also have the logs sent to a SIEM solution for be used for detection, compliance and reporting purposes.

Other items to consider

  • If you’re within Europe and your logs contain personally identifiable information, you need to let someone know to prevent GDPR compliance issues.
  • Some people like to have the syslog directory structure different, such as separate folder for each device.
    • Just think about how you will be working with the logs.
    • I’ve tried multiple approaches, both on file and directory level, and the above structure is what works best for me.
    • Plus, makes removing old data easy.
    • When someone suggests that the directory or filename should be based on the DNS name, remember that not only will the syslog server have to make additional DNS requests just for the logs, but any fault or a change in DNS is going to change where the log is saved.
  • If you have multiple firewalls, you may need to tweak the UDP buffer or get professional assistance to make sure your server can handle the logs.