1

I'm using following system/package:

$ cat /etc/redhat-release 
CentOS Linux release 7.2.1511 (Core) 
$ rpm -q filebeat
filebeat-1.3.0-1.x86_64
$ 

with /etc/filebeat/filebeat.yml:

$ cat /etc/filebeat/filebeat.yml 
filebeat:
  prospectors:
    -
      paths:
        - /var/log/*.log
      input_type: log
  registry_file: /var/lib/filebeat/registry
output:
  elasticsearch:
    hosts: ["localhost:9200"]
shipper:
logging:
  to_syslog: true
  files:
$ 

kibana

"message": "Sep 8 10:20:01 X CROND[11586]: (root) CMD (/usr/lib64/sa/sa1 1 1)",

  • How can I use timestamp from message as @timestamp in Kibana?
  • How can I separate out rest of message (daemon, etc.) into separate fields?
1
  • Please bear with me, if I didn't ask my question properly, I'm only learning)
    – alexus
    Sep 8, 2016 at 18:14

2 Answers 2

0

The message field is text, not something Kibana knows how to use as a timestamp. You need to add some additional parsing in order to convert the timestamp from your log file into a date data type. You can learn more about Elasticsearch data types by reading the relevant documentation.

You've configured Filebeat to output directly to Elasticsearch. In order to parse the timestamp (and potentially other fields) from your log file, you'll need to configure Filebeat to output to Logstash instead. Logstash can then be used to modify your log data, parsing the timestamp as well as doing other pattern matching.

Having said all that, it looks like you might be using Filebeat to read a syslog file. That's fine, but it's also possible for Logstash to directly receive syslog data, which could simplify your overall setup.

1
  • I think the OP's question could be generalized as follows: Let's say he had a time field defined above and it contained some time stamp value. He wants Kibana to use that field as the official "timestamp" rather than defaulting to Elasticsearch's automatically-generated timestamp field.
    – Howiecamp
    Jul 16, 2017 at 2:38
0

Case 1: If you have time stamp in the message as @timestamp then change it to root level by changing json.keys_under_root: true then change "json.overwrite_keys: true

Case 2: Add @timestamp to your app for example if your log has data as {"@timestamp":"2017-01-18T11:41:28.753Z","message":"Some log"}

Then change "json.overwrite_keys: false" to true in filebeat.yml and now the @timestamp matches

2
  • Hi, do you have any reference to this from the documentation?
    – m26a
    Dec 3, 2021 at 12:17
  • @Sidney you can refer the example here https://discuss.elastic.co/t/in-filebeat-what-is-the-difference-between-json-keys-under-root-and-decode-json-fields/202732/2 For documentation, you can refer this page https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html
    – Sahit
    Dec 22, 2021 at 6:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .