Logstash prune filter with json data
Logstash isn’t the most intuitive system at times. I found myself banging my head against a wall for hours trying to solve what felt like a simple issue.
I was importing JSON data structures with sub-fields but wanted to prune some out. You would expect to be able to write something like the following:
It seems the logstash-filter-prune plugin doesn’t like this configuration. After digging through the source code I found this lovely note:
NOTE: This filter currently only support operations on top-level fields, i.e. whitelisting and blacklisting of subfields based on name or value does not work.
GitHub. A solution is to copy the required fields out of the nested structure like so:
Hopefully this will help a few people who try the same type of configuration.