Questions tagged [fluentd]

Fluentd is an open source log aggregator.

Filter by
Sorted by
Tagged with
0
votes
0answers
14 views

Fluentd flitering - concat, can't join lines

I can't join multiple-line logs into one-line log. All logs are sended to syslog, and next parsed by fluentd. From fluentd I am sending logs to loggly. All works perfectly, but as separate lines. I ...
0
votes
2answers
77 views

Kubernetes DaemonSet Pod exceeds memory limit but is not killed

We use kubernetes (specifically: openshift) to run our infrastructure in our team. One of the daemonsets, fluentd, is currently causing a lot of trouble, frequently taking full nodes down with huge ...
0
votes
1answer
59 views

Disable fluentd on on Container-Optimized OS (GCE)

I would like to disable fluentd on a VM instance running Container-Optimized OS on Google compute engine. The reason being that it is eating up a lot of RAM for functionalities I do not require. $ ...
0
votes
0answers
22 views

Sending custom metrics to cloud watch

I want to execute the commands : java -XX:+PrintFlagsFinal -version | grep HeapSize free -m collect the output and send the collected output with instance id of the instance to cloud watch. I ...
0
votes
1answer
41 views

installing google-fluentd agent on docker produce an error

I am trying to install google-fluentd agent on a docker and get the following error: Restarting google-fluentd: google-fluentd ... failed! The OS is Debian GNU/Linux 9 (stretch) Docker file snippet:...
0
votes
1answer
61 views

Parsing or Reformatting Logs before feeding them to Splunk or Elastic Search

I have very complex log messages, that I want to reduce to the most important fields in order to save quota. The log messages are multiline and there is a lot of redundant information in them. A ...
1
vote
0answers
16 views

Is there a preferred way to migrate from logcheck to fluentd?

Logcheck uses a negative filter approach, you specify the log messages that you consider harmless and you will see the rest. Looking at fluentd it is the opposite, it will filter messages that are ...
0
votes
0answers
170 views

Collecting lambda logs from cloudwatch using fluentd

can someone please help me in collecting logs from cloudwatch and send them to elasticsearch. My configuration is <source> @type cloudwatch_logs region us-east-1 # You must supply a ...
0
votes
0answers
248 views

fluentd error parsing auth.log

I have fluentd setup for Kubernetes which is working just fine but when add more sources it fails. I'm trying to get auth.log like this: <source> @id authlog @type tail format none ...
0
votes
1answer
353 views

Send Kube logs to Cloudwatch using fluentd

My cluster is on AWS and I've used kops to build by cluster. I'm trying to send the kubernetes logs to AWS Cloud-watch using fluentd. I was able to stand-up the fluentd pods. I am seeing the below ...
1
vote
0answers
51 views

Routing Fluent Bit tcp event doesn't match to Fluent

i’m new to Fluent and Fluent Bit. Fluent Bit (0.13) listens to tcp and forwards to a Fluent instance (td-agent 3.2). Fluent will push it to rabbitmq. The basic process is working. Fluent Bit ...
0
votes
2answers
87 views

systemd and launching docker containers

So I am new to docker and wanted to start up a logspout docker container on server start up so I thought i would just chuck in a systemd start up file: [Unit] Description=Logspout GELF Container ...
0
votes
1answer
1k views

How to retrieve logs from AWS EC2 and Docker containers fail-proof?

Currently, we have the following setup: Multiple AWS EC2 machines, some which have a Docker Container running, some are dockerless. To retrieve Syslogs, webserver logs (Apache) and application logs, ...
0
votes
1answer
163 views

td-agent shared key mismatch between servers

I am trying to set encrypted logs with td-agent between two machines. I have followed the examples from the official documentation of fluentd however i am stuck in a very weird situation. The client ...
0
votes
2answers
920 views

How to config Zabbix item read data from port and create graphs?

I am new to Zabbix. I send data from fluentd client to zabbix server port 10050. My question is How should I config Zabbix item to read this data from port and generate graph for it. My data is like ...
0
votes
0answers
330 views

fluentd services die immediately after they start up

Please excuse the naivety in my question, but this is not a subject I know much about at present. My company is currently running kubernetes-managed fluentd processes to push logs to logstash. These ...
4
votes
0answers
914 views

Consuming multiline json in fluentd

I am new to fluentd. I have applications that run in Docker containers. They are Java apps that log in JSON format. The JSON messages are usually split over multiple lines. I would like to use the ...
0
votes
0answers
634 views

using google fluentd to log from compute engine instance in google cloud

I got a problem with the logging agent from Google Cloud. What i wanted was to use the google-fluentd (based on fluentd) on a webserver to collect the logs and sent it to Stackdriver Logging Api. ...
2
votes
1answer
947 views

How to specify the log tag from docker compose on google logging?

In my docker compose file, I can specify the logging driver as a "fluentd" and by default, it pushes the log to the listener of google-fluentd on port 24224 I can see the log appear on the logging ...
2
votes
0answers
228 views

Monitoring Cassandra Server Logs?

I've done some searching and have not found any examples of people using something like FluentD to forward Cassandra log file data (system.log) to a log indexer. Is this something any is doing? If ...
1
vote
0answers
51 views

Microsoft OMSAgent Listens on TCP port 16001

I have installed the Microsoft OMSAgent (on Linux). Now I see that it listens on TCP port 16001. Neither the documentation nor grepping through the Sourcecode gave some hints why. Anyone got an ...
1
vote
1answer
1k views

How to address container for logging using docker-compose

A container described in docker-compose.yml uses logging extension to send log to a fluentd container. version: "2" services: fluentd: image: fluent/fluentd:v0.14.8 container_name: ...
0
votes
1answer
100 views

Fluentd elasticsearch comprssion

I forward logs from app servers with fluentd (type tail) to elasticsearch server. Do not understand the following: Can I enable for reducing traffic the compression transmitted logs with type tail ...
0
votes
1answer
462 views

How to split a single record to multiple records in fluentd?

I would like to ask regarding to fluentd. My fluentd versions is below. td-agent-2.1.5-0.x86_64 fluentd 0.10.61 I now have a tail input plugin using multiple line format which parses multiple lines ...
1
vote
0answers
87 views

fluentd missing region error

i want to fetch cloudwatch logs. I installed cloudwatch input plugin and configured the config file of td-agent. but im getting the error as "unexpected error error_class=aws::errors::...
5
votes
1answer
395 views

How to maximize utilization of fluentd server?

I have a fluentd server that is processing an average of 1000 req/s. This server is composed by 32 cpus. In this server are running 32 fluentd docker containers with the same configuration. The entry ...
4
votes
1answer
198 views

logstash-forward equivalent for fluentd?

Is there something equivalent to logstash-forwarder that can ship logfiles to fluentd? I am trying to send log files from an application to a remote fluentd but have not seen whether this is ...
2
votes
0answers
367 views

fluentd modular configuration

Is there any way to setup fluentd/td-agent in a way that it's configuration will be modular? I know there is @include directive but this works only if every time I add something new I modify main td-...
1
vote
1answer
628 views

Fluentd not writing apache access log to mongodb

I'm following the instructions from this link to write apache access logs to mongo db- http://docs.fluentd.org/articles/apache-to-mongodb. But nothing is getting written to mongo DB. I'm checking ...
2
votes
2answers
286 views

How to collect logs for statistics in site?

I want to collect logs from nginx ( several servers, 1 000 000 lines in logs for minute for all servers ) to central stat server for statistics processing. I see 2 variants: write logs to local log ...
0
votes
2answers
4k views

Configuring Logrotate for Fluentd files. Necessary?

I have the following fluent.conf <source> type forward </source> <source> type monitor_agent port 24220 </source> # Listen DRb for debug <source> type ...
2
votes
1answer
3k views

Forward linux logs to fluentd on linux

On one VM I have this in /etc/rsyslog.d/50-default.conf *.* @192.168.29.1:42185 # Default rules for rsyslog. On the vm with fluentd I have this: I have this in /etc/td-agent/td-agent.conf <...
0
votes
1answer
3k views

fluent configuration not working

I have setup fluent with elasticsearch and Kibana with nginx. I followed the instruction and when I go to http:/192.168.10.25:8888 I get the following message 400 Bad Request 'json' or 'msgpack' ...
1
vote
1answer
2k views

Installing fluentd / Kibana / Elastic Search on Suse Enterprise Linux 11

I have successfully installed the recommended fluentd stack ( fluentd, Kibana, Elastic Search ) on a Centos 6 virtual machine. But our production environment is all Suse Enterprise Linux 11. I am ...