Ansible Event-Driven Automation with EDA Controller

Introduction

Traditional automation is reactive — a human sees an alert, logs into a system, diagnoses the problem, and runs a playbook. Event-Driven Ansible (EDA) closes that loop: monitoring tools fire events, EDA evaluates rules, and playbooks execute automatically. Disk full? EDA cleans up logs. Service crashed? EDA restarts it and opens a ticket. Certificate expiring? EDA renews it before anyone notices. The shift from "human-in-the-loop" to "human-on-the-loop" is what separates mature automation from scripting.

Architecture

┌──────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│  Event Sources   │     │  EDA Controller  │     │  AAP Controller  │
│                  │     │                  │     │                  │
│  • Prometheus    │────▶│  Rulebooks:      │────▶│  Job Templates:  │
│  • Alertmanager  │     │  • Match events  │     │  • Remediation   │
│  • Kafka         │     │  • Filter/route  │     │  • Diagnostics   │
│  • Webhooks      │     │  • Throttle      │     │  • Escalation    │
│  • CloudWatch    │     │                  │     │                  │
│  • Dynatrace     │     └──────────────────┘     └──────────────────┘
│  • ServiceNow    │              │
└──────────────────┘              │
                           ┌──────▼──────┐
                           │  Audit Log  │
                           │  & Metrics  │
                           └─────────────┘

EDA Components

Component	Description
Event source	Plugin that receives events (webhook, Kafka, file watch, etc.)
Rulebook	YAML file defining conditions → actions
Condition	Jinja2 expression that matches events
Action	What to do: run_playbook, run_job_template, debug, post_event
EDA Controller	Web UI + API for managing rulebook activations

Installation

# Install ansible-rulebook CLI
pip install ansible-rulebook

# Install EDA collections
ansible-galaxy collection install ansible.eda
ansible-galaxy collection install dynatrace.event_driven_ansible

# For development/testing
ansible-rulebook --version

Rulebook Basics

Structure

---
- name: Auto-remediate disk space alerts
  hosts: all
  sources:
    - ansible.eda.webhook:
        host: 0.0.0.0
        port: 5000

  rules:
    - name: Disk space critical
      condition: event.payload.alert == "disk_space_critical"
      action:
        run_playbook:
          name: playbooks/cleanup-disk.yml
          extra_vars:
            target_host: "{{ event.payload.host }}"
            threshold: "{{ event.payload.threshold }}"

Running a Rulebook

# Local development
ansible-rulebook --rulebook rulebooks/disk-remediation.yml -i inventory/

# With verbose output
ansible-rulebook --rulebook rulebooks/disk-remediation.yml -i inventory/ --verbose

# With AAP Controller integration
ansible-rulebook --rulebook rulebooks/disk-remediation.yml \
  --controller-url https://aap.example.com \
  --controller-token $AAP_TOKEN

Pattern 1: Prometheus Alertmanager Integration

Alertmanager Configuration

# alertmanager.yml
route:
  receiver: ansible-eda
  routes:
    - match:
        severity: critical
      receiver: ansible-eda
      continue: true

receivers:
  - name: ansible-eda
    webhook_configs:
      - url: "http://eda-controller.internal:5000/endpoint"
        send_resolved: true

EDA Rulebook for Alertmanager

---
- name: Prometheus alert auto-remediation
  hosts: all
  sources:
    - ansible.eda.alertmanager:
        host: 0.0.0.0
        port: 5000

  rules:
    - name: High memory usage — restart application
      condition: >
        event.alert.labels.alertname == "HighMemoryUsage" and
        event.alert.status == "firing" and
        event.alert.labels.severity == "critical"
      throttle:
        once_within: 15 minutes
        group_by: event.alert.labels.instance
      action:
        run_job_template:
          name: "Restart Application Service"
          organization: "Operations"
          job_args:
            extra_vars:
              target_host: "{{ event.alert.labels.instance }}"
              service_name: "{{ event.alert.labels.service }}"
              alert_name: "{{ event.alert.labels.alertname }}"

    - name: Disk space warning — cleanup logs
      condition: >
        event.alert.labels.alertname == "DiskSpaceWarning" and
        event.alert.status == "firing"
      throttle:
        once_within: 30 minutes
        group_by: event.alert.labels.instance
      action:
        run_job_template:
          name: "Disk Cleanup"
          organization: "Operations"
          job_args:
            extra_vars:
              target_host: "{{ event.alert.labels.instance }}"
              mount_point: "{{ event.alert.labels.mountpoint }}"

    - name: Service down — attempt restart then escalate
      condition: >
        event.alert.labels.alertname == "ServiceDown" and
        event.alert.status == "firing"
      throttle:
        once_within: 10 minutes
        group_by:
          - event.alert.labels.instance
          - event.alert.labels.service
      action:
        run_job_template:
          name: "Service Recovery Workflow"
          organization: "Operations"
          job_args:
            extra_vars:
              target_host: "{{ event.alert.labels.instance }}"
              service_name: "{{ event.alert.labels.service }}"

    - name: Alert resolved — log recovery
      condition: event.alert.status == "resolved"
      action:
        debug:
          msg: "Alert {{ event.alert.labels.alertname }} resolved on {{ event.alert.labels.instance }}"

Pattern 2: Self-Healing Infrastructure

Service Recovery Playbook

---
# playbooks/service-recovery.yml
- name: Automated service recovery
  hosts: "{{ target_host }}"
  become: true
  vars:
    max_restart_attempts: 3
    escalation_email: ops-oncall@example.com

  tasks:
    - name: Check service status
      ansible.builtin.systemd:
        name: "{{ service_name }}"
      register: service_status

    - name: Attempt service restart
      ansible.builtin.systemd:
        name: "{{ service_name }}"
        state: restarted
      register: restart_result
      retries: "{{ max_restart_attempts }}"
      delay: 10
      until: restart_result is success
      when: service_status.status.ActiveState != "active"

    - name: Verify service is running
      ansible.builtin.systemd:
        name: "{{ service_name }}"
      register: post_restart
      failed_when: false

    - name: Collect diagnostics on failure
      when: post_restart.status.ActiveState != "active"
      block:
        - name: Get service logs
          ansible.builtin.command: journalctl -u {{ service_name }} --since "30 minutes ago" --no-pager
          register: service_logs

        - name: Get system resource usage
          ansible.builtin.shell: |
            echo "=== Memory ==="
            free -h
            echo "=== Disk ==="
            df -h
            echo "=== Top processes ==="
            ps aux --sort=-%mem | head -20
          register: system_info

        - name: Create ServiceNow incident
          ansible.builtin.uri:
            url: "https://instance.service-now.com/api/now/table/incident"
            method: POST
            user: "{{ snow_user }}"
            password: "{{ snow_password }}"
            body_format: json
            body:
              short_description: "Service {{ service_name }} failed on {{ target_host }} — auto-remediation unsuccessful"
              description: |
                Automated recovery failed after {{ max_restart_attempts }} attempts.

                Service Logs:
                {{ service_logs.stdout }}

                System Info:
                {{ system_info.stdout }}
              urgency: 1
              impact: 2
              assignment_group: "Infrastructure Operations"
          delegate_to: localhost

        - name: Send escalation notification
          ansible.builtin.mail:
            host: smtp.internal
            to: "{{ escalation_email }}"
            subject: "🚨 ESCALATION: {{ service_name }} on {{ target_host }} — manual intervention required"
            body: "Automated recovery failed. ServiceNow incident created. Service logs attached."
          delegate_to: localhost

Pattern 3: Kafka Event Stream Processing

---
- name: Process infrastructure events from Kafka
  hosts: all
  sources:
    - ansible.eda.kafka:
        host: kafka.internal
        port: 9092
        topic: infrastructure-events
        group_id: eda-consumer

  rules:
    - name: New server provisioned — configure baseline
      condition: event.body.event_type == "server_provisioned"
      action:
        run_job_template:
          name: "Server Baseline Configuration"
          organization: "Operations"
          job_args:
            extra_vars:
              target_host: "{{ event.body.hostname }}"
              server_role: "{{ event.body.role }}"
              environment: "{{ event.body.environment }}"

    - name: Certificate expiring — auto-renew
      condition: >
        event.body.event_type == "cert_expiring" and
        event.body.days_remaining < 30
      action:
        run_job_template:
          name: "Certificate Renewal"
          organization: "Security"
          job_args:
            extra_vars:
              cert_domain: "{{ event.body.domain }}"
              cert_type: "{{ event.body.cert_type }}"

    - name: Security vulnerability detected
      condition: >
        event.body.event_type == "vulnerability" and
        event.body.severity in ["critical", "high"]
      action:
        run_job_template:
          name: "Emergency Patching"
          organization: "Security"
          job_args:
            extra_vars:
              cve_id: "{{ event.body.cve_id }}"
              affected_hosts: "{{ event.body.affected_hosts }}"
              package: "{{ event.body.package }}"

Pattern 4: CloudWatch Integration (AWS)

---
- name: AWS CloudWatch event processing
  hosts: all
  sources:
    - ansible.eda.aws_sqs_queue:
        region: us-east-1
        queue_name: eda-cloudwatch-events
        delay_seconds: 5

  rules:
    - name: EC2 instance state change — update inventory
      condition: >
        event.body.source == "aws.ec2" and
        event.body.detail_type == "EC2 Instance State-change Notification"
      action:
        run_playbook:
          name: playbooks/update-cmdb.yml
          extra_vars:
            instance_id: "{{ event.body.detail.instance_id }}"
            new_state: "{{ event.body.detail.state }}"

    - name: Auto Scaling event — configure new instances
      condition: >
        event.body.source == "aws.autoscaling" and
        event.body.detail_type == "EC2 Instance Launch Successful"
      action:
        run_job_template:
          name: "Configure New EC2 Instance"
          organization: "Cloud Ops"
          job_args:
            extra_vars:
              instance_id: "{{ event.body.detail.EC2InstanceId }}"

Throttling and Deduplication

rules:
  - name: High CPU alert
    condition: event.alert.labels.alertname == "HighCPU"
    throttle:
      # Only fire once per 15 minutes per host
      once_within: 15 minutes
      group_by: event.alert.labels.instance
    action:
      run_job_template:
        name: "Investigate High CPU"

  - name: Multiple alerts from same host — aggregate
    condition: >
      events.alert_group << event.alert and
      events.alert_group.length >= 3
    window: 5 minutes
    action:
      run_job_template:
        name: "Host Health Investigation"
        job_args:
          extra_vars:
            target_host: "{{ events.alert_group[0].alert.labels.instance }}"
            alert_count: "{{ events.alert_group.length }}"

Troubleshooting

Rulebook Not Matching Events

# Debug mode to see all events
ansible-rulebook --rulebook my_rulebook.yml -i inventory/ --verbose --debug

# Test with a manual webhook event
curl -X POST http://localhost:5000/endpoint \
  -H "Content-Type: application/json" \
  -d '{"alert": "disk_space_critical", "host": "web-01", "threshold": 95}'

EDA Controller Connection Issues

# Verify EDA Controller is reachable
curl -k https://eda-controller.internal/api/v1/
# Check authentication
curl -k -H "Authorization: Bearer $EDA_TOKEN" https://eda-controller.internal/api/v1/activations/

Conclusion

Event-Driven Ansible transforms automation from "run playbook when told" to "run playbook when needed." Use Alertmanager webhooks for Prometheus-based monitoring, Kafka consumers for event streams, and CloudWatch for AWS-native events. Rulebooks define the matching logic with Jinja2 conditions and throttling to prevent alert storms. The critical pattern is escalation: attempt auto-remediation first, collect diagnostics if it fails, create an incident ticket, and notify the on-call team. EDA doesn't replace humans — it handles the first 80% of incidents automatically so humans can focus on the complex 20%.