AnsibleByExample

Ansible splunk.itsi Module Example: EDA-Triggered Remediation

Beginner
2024-01-013By Luca Berton

Lesson 1 of 15 in Foundations

Ansible Foundations to Production — full curriculum
  1. 1. Introduction to Ansible and Automation
  2. 2. Installing and Configuring Ansible
  3. 3. Running Ad-Hoc Commands
  4. 4. Writing Your First Playbook
  5. 5. Understanding and Using Modules
  6. 6. Variables and Facts
  7. 7. Conditionals and Loops
  8. 8. Handlers and Idempotency
  9. 9. Structuring Projects with Roles
  10. 10. Inventory Management
  11. 11. Using Ansible Vault for Secrets
  12. 12. Debugging and Troubleshooting
  13. 13. Real-World Automation Examples
  14. 14. Final Project: Deploy a Web Server
  15. 15. Bonus - What's Next with Ansible?
View path overview →

splunk.itsi is one of 12 new content collections announced for AAP 2.7 at Red Hat Tech Day Netherlands 2026 in Bunnik. It brings Splunk IT Service Intelligence (ITSI) into Event-Driven Ansible so a service-health notable event can trigger an automated fix without a human touching a ticket first.

Example: rulebook + remediation playbook

# rulebooks/itsi-service-health.yml
---
- name: React to ITSI service health notable events
  hosts: all
  sources:
    - splunk.itsi.notable_event_source:
        splunk_host: "{{ itsi_host }}"
        splunk_token: "{{ itsi_token }}"
        service_name: "checkout-service"
        interval: 30

  rules:
    - name: Remediate degraded checkout service
      condition: event.itsi.health_score < 40
      action:
        run_playbook:
          name: remediate_checkout_service.yml

# remediate_checkout_service.yml
---
- name: Closed-loop remediation for degraded ITSI service
  hosts: checkout_service_nodes
  gather_facts: false

  vars:
    itsi_service_id: "{{ ansible_eda.event.itsi.service_id }}"
    itsi_kpi_id: "{{ ansible_eda.event.itsi.kpi_id }}"

  tasks:
    - name: Acknowledge the ITSI notable event
      splunk.itsi.notable_event:
        splunk_host: "{{ itsi_host }}"
        splunk_token: "{{ itsi_token }}"
        event_id: "{{ ansible_eda.event.itsi.event_id }}"
        status: "in_progress"
        owner: "aap-automation"

    - name: Restart the checkout application service
      ansible.builtin.systemd:
        name: checkout-app
        state: restarted

    - name: Wait for application health endpoint to recover
      ansible.builtin.uri:
        url: "http://localhost:8080/healthz"
        status_code: 200
      register: health_check
      until: health_check.status == 200
      retries: 5
      delay: 10

    - name: Close the notable event in ITSI with resolution notes
      splunk.itsi.notable_event:
        splunk_host: "{{ itsi_host }}"
        splunk_token: "{{ itsi_token }}"
        event_id: "{{ ansible_eda.event.itsi.event_id }}"
        status: "resolved"
        comment: "Auto-remediated by AAP: checkout-app restarted, health check passed."

What this does

The rulebook uses the splunk.itsi.notable_event_source EDA source plugin to poll a named ITSI service and fires when its health score drops below a threshold. The matched event lands in ansible_eda.event, so the remediation playbook can pull the ITSI service ID, KPI ID, and event ID straight out of the payload. The playbook acknowledges the notable event first (so nobody else double-remediates it), restarts the affected service, confirms recovery through a health check, then writes the resolution back into ITSI. That last step is the "closed-loop" part: the remediation status is visible in the same ITSI glass table the on-call team is already watching.

Notes / Gotchas

  • Store itsi_host and itsi_token in an AAP credential type, not in plain vars — inject them at rulebook activation time.
  • The until/retries loop on the health check matters: without it, the playbook would close the notable event before confirming the restart actually fixed anything.
  • Scope hosts: in the remediation playbook tightly (inventory group per service) so a notable event for one service can't accidentally trigger action against unrelated hosts.
  • splunk.itsi was announced alongside splunk.enterprise (Universal Forwarder lifecycle) and splunk.es (Enterprise Security incident workflows) at Red Hat Tech Day Netherlands 2026 — the three collections are designed to be used together across detection, enrichment, and remediation.
  • Always test the remediation action manually before wiring it to auto-run from EDA; a restart loop triggered by a flapping health score is a bad first production experience.

Wire splunk.itsi notable events into an EDA rulebook and you get remediation that starts and closes itself, with the audit trail living where the service owners already look.

References

All Tutorials