Versions
Fluent-Bit 3.1.3
Logstash OSS 8.9
OpenSearch 2.15.0
OpenSearch Dashboards 2.15.0
RHEL 7.9
Describe the issue:
I just cannot query IP based fields only when using PPL queries. DQL, SQL are able to query for the same field properly.
For example, srcaddr is an IP field present in my Palo Alto firewall logs index. The index mapping for the field looks as follows:
“srcaddr”: {
“type”: “ip”
}
My PPL query: source=paloaltologs-2024.07.30 and srcaddr=‘10.xx.xx.xx’
When I run this on Query Workbench I get ‘Events: Bad Request, this query is not runnable.’
When run on Events Explorer, it just says no results found.
The error captured in OpenSearch logs is as follows:
[2024-07-30T02:27:10,212][ERROR][o.o.s.p.r.RestPPLQueryAction] [dc1ddcpplcxx.dxx.locxx] Error happened during query handling
org.opensearch.sql.exception.ExpressionEvaluationException: = function expected {[BYTE,BYTE],[SHORT,SHORT],[INTEGER,INTEGER],[LONG,LONG],[FLOAT,FLOAT],[DOUBLE,DOUBLE],[STRING,STRING],[BOOLEAN,BOOLEAN],[DATE,DATE],[TIME,TIME],[DATETIME
,DATETIME],[TIMESTAMP,TIMESTAMP],[INTERVAL,INTERVAL],[STRUCT,STRUCT],[ARRAY,ARRAY]}, but get [IP,STRING]
at org.opensearch.sql.expression.function.DefaultFunctionResolver.resolve(DefaultFunctionResolver.java:63) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.expression.function.BuiltinFunctionRepository.resolve(BuiltinFunctionRepository.java:153) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.expression.function.BuiltinFunctionRepository.lambda$resolve$1(BuiltinFunctionRepository.java:139) ~[core-2.15.0.0.jar:?]
at java.base/java.util.Optional.or(Optional.java:313) ~[?:?]
at org.opensearch.sql.expression.function.BuiltinFunctionRepository.resolve(BuiltinFunctionRepository.java:139) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.expression.function.BuiltinFunctionRepository.compile(BuiltinFunctionRepository.java:112) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.expression.function.BuiltinFunctionRepository.compile(BuiltinFunctionRepository.java:102) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.ExpressionAnalyzer.visitCompare(ExpressionAnalyzer.java:307) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.ExpressionAnalyzer.visitCompare(ExpressionAnalyzer.java:78) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.ast.expression.Compare.accept(Compare.java:32) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.ExpressionAnalyzer.analyze(ExpressionAnalyzer.java:96) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.Analyzer.visitFilter(Analyzer.java:237) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.Analyzer.visitFilter(Analyzer.java:111) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.ast.tree.Filter.accept(Filter.java:41) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.Analyzer.visitProject(Analyzer.java:376) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.Analyzer.visitProject(Analyzer.java:111) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.ast.tree.Project.accept(Project.java:65) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.analysis.Analyzer.analyze(Analyzer.java:136) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.analyze(QueryService.java:91) ~[core-2.15.0.0.jar:?]
at org.opensearch.sql.executor.QueryService.execute(QueryService.java:41) [core-2.15.0.0.jar:?]
at org.opensearch.sql.executor.execution.QueryPlan.execute(QueryPlan.java:65) [core-2.15.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$submit$0(OpenSearchQueryManager.java:31) [opensearch-2.15.0.0.jar:?]
at org.opensearch.sql.opensearch.executor.OpenSearchQueryManager.lambda$withCurrentContext$1(OpenSearchQueryManager.java:45) [opensearch-2.15.0.0.jar:?]
at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:882) [opensearch-2.15.0.jar:2.15.0]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) [?:?]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) [?:?]
at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
It works absolutely fine with other languages on OpenSearch Dashboards as seen below-
DQL ==> srcaddr: 10.xx.xx.xx
SQL ==> select * from paloaltologs-2024.07.30 where srcaddr = ‘10.xx.xx.xx’;
Even in Detection Rules, I can query the field using Sigma YAML using cidr modifier.
On Dev Tools, I can query the same using
GET paloaltologs/_search
{
“query”: {
“term”: {
“srcaddr”: “10.0.0.0/8”
}
}
}
Please help figure out what is the issue with PPL in my setup so that I could have a good night’s sleep!