In this article, basically explained log collection for SIEM. It contains a basic log, time, source system and a message. For example, when we look at the content of the "/var/log/auth.log" file on an Ubuntu server, we can see the source, time and message information.
Logs are generally collected in the following 2 ways:
Also, if you want to know how to parse logs you can check the Log Parsing blog post
In order to implement this method, a log agent software is required. Agents often have parsing, log rotation, buffering, log integrity, encryption, conversion features. In other words, this agent software can take action on the logs it collects before forwarding them to the target.
For example, with the agent software, we can divide a log with "username: LetsDefend; account: Administrator" into 2 parts and forward it as:
It is a very popular network protocol for log transfers. It can work with both UDP and TCP, and can optionally be encrypted with TLS. Some devices that support syslog: Switch, Router, IDS, Firewall, Linux, Mac, Windows devices can become syslog supported with additional software.
If you want to forward your log with Syslog, you will need to parsing in syslog format.
Syslog Format:
Timestamp - Source Device - Facility - Severity - Message Number - Message Text
Also, the maximum packet size that can be sent with Syslog UDP is 1024 bytes. For TCP it is 4096 bytes.
Most SIEM products have their own agent software. 3rd party agents have more capabilities than syslog because of the features they support. Some agents:
Splunk: universal forwarder
ArcSight: ArcSight Connectors
These agents are easy to integrate into SIEM and have parsing features.
Open Source Agents
They are generally agents that provide basic needs comfortably. However, it may not be as effective as the agent of the SIEM product itself. (Ease of installation, integration, additional features etc.)
Popular open source agents:
Agentless log sending process is sometimes preferred as there is no installation and update cost. Usually, logs are sent by connecting to the target with SSH or WMI.
For this method, the username and password of the log server are required, therefore there is a risk of the password being stolen.
Easier to prepare and manage than the agent method. However, it has limited capabilities and credentials are wrapped in the network.
Sometimes there are logs that you cannot collect with existing agent software. For example, if you cannot read the logs of a cloud-based application with the agent, you may need to write your own script.
As you can see, there are various ways to collect logs. These are agents and agentless. In cases where the agents on the market are not sufficient, you should write your own scripts.
