Collecting DataΒΆ
Log data is being collected from all the log files in /var/log and all of its subdirectories as well as from certain listeners that can be configured on each host in the server farm to collect additional data from applications without piping them through the system log facilities first.
All of that collected data gets forwarded to a central log server within the server farm where it will be stored and indexed by ElasticSearch.
The collection and forwarding is performed by FluentD. Each host has a FluentD client and the central log server also has a FluentD server instance installed. Each of the clients is collecting data from system log files in /var/log and from optionally configured additional listeners. The data is queued and forwarded to the FluentD server. This forwarding is protected the following way:
- Shared Key: a string only known to all the hosts in the server farm
- SSL Certificate: used to encrypt the data in transit
- username and password: to authenticate the communication, these credentials are only known to hosts within the server farm
All the communication is done over port 24284.
The fluentd server is also queueing all the received data and forwards it locally on to ElasticSearch which is configured to only accept data from that fluentd instance and not from any other local or remote process.
Depending on configuration all the data can be indexed in different indexes which makes querying the data easier down the road. Also, data retention can be configured seperately for each index. Though this is not possible in this server farm as yet.