Why Hunk's field extractor behaves differently in Smart Mode vs Fast Mode?

[jimjh]

My data files are in Avro, and I have a props.conf that looks like

[source::/logs/...]
sourcetype = api

[api]
KV_MODE = json
TIME_PREFIX = "timestamp"
TIME_FORMAT = %10s%3N

For a given time range,

• in Smart Mode, Hunk tells me that some large n events were found
• in Fast Mode, Hunk tells me that no events were found

How can I resolve this?

[Ledion_Bitincka]

In Hunk 6.2 the recommended way of solving this would be to tell Hunk to always return the timestamp field

[my-virtual-index]
....
# required fields (6.2 or later)
vix.input.[N].required.fields = <comma delimited, optionally wildcarded, list of fields to always output for this input>

In Hunk 6.1.x the recommended way of solving this issue is to disable the column projection optimization that leads to this problem:

[my-provider]
...
# disable column projection
vix.splunk.search.column.filter = false

[Ledion_Bitincka]

The reason for the difference in behavior is due to Hunk's optimization based on required fields - which in "Fast mode" are whatever the search requires, while "Smart mode", which for an event search (e.g. search index=avro) is the same as "Verbose mode", all fields are required. The avro record reader honors the required fields and since Hunk needs/expects "_time" while the data contains "timestamp" the time related field is omitted and thus causing the problem.

[jimjh · Accepted solution]

I was given a workaround.

EXTRACT-_time = strptime('timestamp', "%s%3N")

[martin_mueller]

Stick to Smart Mode 😛 there rarely is a reason to not use Smart Mode.

What search are you running?