Introduction
Fluent Bit is a lightweight log processor and forwarder — it collects, parses, filters, and routes logs with minimal resource usage (typically ~450KB memory). It's the CNCF-graduated log collector for Kubernetes and edge environments. Ansible automates deployment: install Fluent Bit, configure input/parser/filter/output pipelines, and manage the service across your fleet.
Install Fluent Bit
---
- name: Deploy Fluent Bit
hosts: all
become: true
vars:
fluentbit_version: "3.1"
tasks:
- name: Add Fluent Bit GPG key
ansible.builtin.get_url:
url: https://packages.fluentbit.io/fluentbit.key
dest: /usr/share/keyrings/fluentbit-keyring.gpg
mode: '0644'
when: ansible_os_family == 'Debian'
- name: Add Fluent Bit repository
ansible.builtin.apt_repository:
repo: "deb [signed-by=/usr/share/keyrings/fluentbit-keyring.gpg] https://packages.fluentbit.io/ubuntu/{{ ansible_distribution_release }} {{ ansible_distribution_release }} main"
filename: fluent-bit
when: ansible_os_family == 'Debian'
- name: Install Fluent Bit
ansible.builtin.package:
name: fluent-bit
state: present
- name: Deploy Fluent Bit config
ansible.builtin.template:
src: fluent-bit.conf.j2
dest: /etc/fluent-bit/fluent-bit.conf
mode: '0644'
notify: restart fluent-bit
- name: Deploy parsers
ansible.builtin.template:
src: parsers.conf.j2
dest: /etc/fluent-bit/parsers.conf
mode: '0644'
notify: restart fluent-bit
- name: Start Fluent Bit
ansible.builtin.service:
name: fluent-bit
state: started
enabled: true
handlers:
- name: restart fluent-bit
ansible.builtin.service:
name: fluent-bit
state: restarted
Pipeline Configuration
# templates/fluent-bit.conf.j2
[SERVICE]
Flush 5
Daemon Off
Log_Level info
Parsers_File /etc/fluent-bit/parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
# ── INPUTS ──
[INPUT]
Name systemd
Tag syslog.*
Systemd_Filter _SYSTEMD_UNIT=sshd.service
Systemd_Filter _SYSTEMD_UNIT=nginx.service
[INPUT]
Name tail
Tag syslog.messages
Path /var/log/syslog,/var/log/messages
Parser syslog-rfc3164
Mem_Buf_Limit 5MB
Skip_Long_Lines On
{% if 'webservers' in group_names %}
[INPUT]
Name tail
Tag nginx.access
Path /var/log/nginx/access.log
Parser nginx
Mem_Buf_Limit 10MB
[INPUT]
Name tail
Tag nginx.error
Path /var/log/nginx/error.log
Mem_Buf_Limit 5MB
{% endif %}
{% if 'app_servers' in group_names %}
[INPUT]
Name tail
Tag app.logs
Path /var/log/myapp/*.log
Parser json
Mem_Buf_Limit 10MB
Multiline On
Parser_Firstline multiline_java
{% endif %}
[INPUT]
Name cpu
Tag metrics.cpu
Interval_Sec 30
[INPUT]
Name mem
Tag metrics.mem
Interval_Sec 30
# ── FILTERS ──
[FILTER]
Name record_modifier
Match *
Record hostname {{ inventory_hostname }}
Record environment {{ env | default('production') }}
[FILTER]
Name grep
Match syslog.*
Exclude log CRON
{% if 'webservers' in group_names %}
[FILTER]
Name parser
Match nginx.access
Key_Name log
Parser nginx
Reserve_Data On
{% endif %}
# ── OUTPUTS ──
{% if fluentbit_output_loki | default(false) %}
[OUTPUT]
Name loki
Match *
Host {{ loki_host }}
Port {{ loki_port | default(3100) }}
Labels job=fluent-bit, host={{ inventory_hostname }}
Auto_Kubernetes_Labels Off
{% endif %}
{% if fluentbit_output_elasticsearch | default(false) %}
[OUTPUT]
Name es
Match *
Host {{ elasticsearch_host }}
Port {{ elasticsearch_port | default(9200) }}
Index fluent-bit
Type _doc
Logstash_Format On
Logstash_Prefix fluent-bit-{{ inventory_hostname }}
{% endif %}
{% if fluentbit_output_forward | default(false) %}
[OUTPUT]
Name forward
Match *
Host {{ fluentd_host }}
Port {{ fluentd_port | default(24224) }}
{% endif %}
{% if fluentbit_output_s3 | default(false) %}
[OUTPUT]
Name s3
Match *
bucket {{ s3_bucket }}
region {{ s3_region }}
total_file_size 50M
upload_timeout 10m
s3_key_format /logs/{{ inventory_hostname }}/%Y/%m/%d/%H_%M_%S.gz
compression gzip
{% endif %}
# Always stdout for debugging
[OUTPUT]
Name stdout
Match metrics.*
Format json_lines
Parsers
# templates/parsers.conf.j2
[PARSER]
Name nginx
Format regex
Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name json
Format json
Time_Key timestamp
Time_Format %Y-%m-%dT%H:%M:%S.%L
[PARSER]
Name syslog-rfc3164
Format regex
Regex /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
Time_Key time
Time_Format %b %d %H:%M:%S
[PARSER]
Name multiline_java
Format regex
Regex /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})/
Health Check
- name: Check Fluent Bit health
ansible.builtin.uri:
url: "http://localhost:2020/api/v1/health"
status_code: 200
- name: Check metrics
ansible.builtin.uri:
url: "http://localhost:2020/api/v1/metrics/prometheus"
return_content: true
register: fb_metrics
Troubleshooting
Check Pipeline Status
- name: Check uptime and inputs
ansible.builtin.uri:
url: "http://localhost:2020/api/v1/uptime"
return_content: true
register: uptime
Buffer Issues
# If Mem_Buf_Limit is hit, Fluent Bit pauses input
- name: Check for paused inputs
ansible.builtin.command: journalctl -u fluent-bit --no-pager -n 50
register: fb_logs
changed_when: false
Related Articles
Conclusion
Fluent Bit is the lightweight alternative to Fluentd and Logstash — Ansible deploys it as a DaemonSet-equivalent across all servers with group-specific inputs (web servers get nginx parsers, app servers get JSON/multiline parsers). Route logs to Loki, Elasticsearch, S3, or Fluentd with a single config change. The built-in HTTP health endpoint and Prometheus metrics make monitoring the log pipeline itself straightforward.