Filebeat is not sending a continuous stream to Logstash

[surfingjoe]

I’m running a development set of servers (not production). I have an ELK server, a web server, and a reverse proxy server. Data from the web server and the reverse proxy have successfully been sent into Logstash on the ELK server. Kibana’s filebeat discovery does show the hostname of all three servers. So I know the configuration is correct.

Running the following command: sudo filebeat -e -c /etc/filebeat/filebeat.yml test output on both remote servers (the web server and the reverse proxy server) has the following response: logstash: 192.168.1.6:5044... connection... parse host... OK dns lookup... OK addresses: 192.168.1.6 dial up... OK TLS... WARN secure connection disabled talk to server... OK

Running the command to see the status of the filebeat service (on both remote servers) reflects the service is active and has a recent timestamp of log data that is being collected. I did this a few times over a span of 15 minutes, and I can continuously see new time stamps of log collection by file beat on the remote servers.

But for some reason, the only time filebeat actually sends data to Logstash on the ELK server is upon reboot of the web server and reverse proxy server. And only just once. Filebeat data is not a continuous stream of data into Logstash.

So my question is: How is it possible that testing filebeat output shows a successful connection, that filebeat data is successfully seen in elasticsearch, and that Kibana discovery has valid data for my servers? But, filebeat is not sending continuously into logstash.

Oh, one more thing. The ELK server does, in fact, have a continuous stream of filebeat data from itself (the ELK server’s filebeat is continuously streaming data to logstash). It’s the other servers that are not feeding continuously.

Running Elasticstack version 7.17.13 on Ubuntu servers. ELK server is Ubuntu server 20.04 (minimal), web server is Ubuntu server 20.04 (minimal), and the reverse proxy server is Ubuntu server 22.04 (minimal). All are on the same subnet. Running as virtual servers in an ESXi VMware platform.

I’ve tried purging filebeat from the web server and the reverse proxy server and ensuring the filebeat configurations are deleted.

Then, reinstalling filebeat and putting in a fresh configuration to send all data to logstash on the ELK server and test the filebeat outputs.

Everything looks OK, but I see only one stream of data sent to logstash immediately upon rebooting the servers! Would like to see a continuous stream of filebeat data from other servers. Ughhhhh!

[surfingjoe]

The problem was caused by installing Filebeat on a minimized Ubuntu LTS server.
Solution: Install rsyslog so the logs are created for Fillebeat to send.
I have determined why the Filebeat service is not streaming to ELK Logstash. It is as simple as the fact the logs do not exist to be sent. Apparently, a Ubuntu server (minimized) doesn’t have rsyslog installed, and therefore "auth.log, kern.log, and syslog don’t exist. Therefore there are no logs that would change over time in the path /var/log/. This makes sense, I suppose, because a minimized system is not expecting users to log into the system.

However, I must make sure that no rogue users are getting into the system, and therefore I must see user logins, user sudo commands, etc., especially on the reverse proxy.

So I ran sudo apt-get install rsyslog on the web server and the reverse proxy server. Now I see a regular stream of data into logstash from those servers because now the data actually exists.

Still don’t see sudo commands yet, hopefully, I can figure that one out. at least I have streaming data from remote servers into Logstash now.