Logstash isn’t the most intuitive system at times. I found myself banging my head against a wall for hours trying to solve what felt like a simple issue.

I was importing JSON data structures with sub-fields but wanted to prune some out. You would expect to be able to write something like the following:

JSON Example

{
    "user" {
        "created_at": "..."
        etc...
    }
}

Logstash Configuration

if [user] {
    prune {
        whitelist_names => [
            "user.created_at",
            "[user][created_at]",
            "[user].[created_at]"
        ]
    }
}

It seems the logstash-filter-prune plugin doesn’t like this configuration. After digging through the source code I found this lovely note:

NOTE: This filter currently only support operations on top-level fields, i.e. whitelisting and blacklisting of subfields based on name or value does not work.

GitHub. A solution is to copy the required fields out of the nested structure like so:

Logstash Configuration

if [user] {
    mutate {
        add_field => {
            "user_created_at" => "%{[user][created_at]}"
        }
    }
    prune {
        whitelist_names => [
            "^user\_"
        ]
    }
}

Hopefully this will help a few people who try the same type of configuration.