Describe the issue:
I have a log ingestion pipeline configured with fluentbit, dataprepper and opensearch. I am able to create index pattern in opensearch dashboard and it shows log lines for 15 minutes and then disappears but while extending the range to one day, it shows the hits count increasing. What could be the reason that hits not showing for 15 minutes time range? I can confirm that log lines being sent to the index continuously as count is increasing continuously in 1 day time range. Thanks for you help in advance.
do you mean Discover shows them?
Yes, index pattern from Discover shows the log lines.
does your time filter say “Last 15 minutes”? if so, that is telling Discover to only look at the last 15 minutes and not beyond that.
Yes, filter says Last 15 minutes. When I start log ingestion pipeline, data shows for 10-15 minutes but then it disappears from 15 minutes time range but that does not mean logs are not being sent to the dashboard. If I select 1 hour, it shows logs are being accumulated as counts are increasing. After 1 hour, logs get disappear again but if I extend time range to one day, it shows the log lines with the continuous increment of log lines count.
Today, I had to extend the time range to one week, so that I can see the logs in the dashboard. I am not sure what is exact issue. Your help is highly appreciated. Thanks.