AnsibleByExample

Ansible Configuration Drift Detection and Remediation

Beginneransibleconfiguration-driftcompliancecheck-mode
2024-01-014By Luca Berton

Lesson 1 of 16 in Security & Compliance

Ansible Security and Compliance Automation — full curriculum
  1. 1. The Compliance-as-Code Model (and Where Ansible Fits)
  2. 2. Ansible's Security Automation Building Blocks
  3. 3. CIS Benchmarks and DISA STIGs Explained
  4. 4. Scanning Hosts for Compliance Drift with OpenSCAP and Ansible
  5. 5. Ansible Lockdown: Using STIG and CIS Roles in Practice
  6. 6. Custom Compliance Checks with assert and Facts
  7. 7. Remediating Findings Safely
  8. 8. CVE-Driven Patch Management
  9. 9. Firewall and SELinux Hardening at Scale
  10. 10. User, Access and Sudo Policy Enforcement
  11. 11. Secrets and Credential Hygiene at Scale
  12. 12. Audit Logging, Reporting and Evidence for Auditors
  13. 13. Event-Driven Remediation with Event-Driven Ansible
  14. 14. Continuous Compliance in CI/CD Pipelines
  15. 15. Change Control and Safe Rollback for Security Playbooks
  16. 16. Capstone: Automated Compliance Pipeline — Scan, Remediate, Report
View path overview →

Ansible Configuration Drift Detection and Remediation

Configuration drift occurs when servers gradually deviate from their intended state — manual changes, ad-hoc fixes, and untracked modifications accumulate until systems become unpredictable. Ansible is one of the best tools to detect and fix this.

What is Configuration Drift?

Day 1: All servers match desired state ✅
Day 30: Someone changed a config manually 🔧
Day 60: Emergency hotfix on 2 servers ⚡
Day 90: "Why is server-3 different from server-1?" 😱

Common causes:

  • Manual SSH changes bypassing automation
  • Emergency fixes applied inconsistently
  • Partial playbook runs that failed midway
  • Different team members making ad-hoc changes

Detect Drift with Check Mode

Ansible's --check (dry run) mode shows what would change without making modifications:

ansible-playbook site.yml --check --diff

If the output shows changes, those servers have drifted from the desired state.

---
- name: Detect configuration drift
  hosts: webservers
  tasks:
    - name: Ensure nginx config matches template
      ansible.builtin.template:
        src: nginx.conf.j2
        dest: /etc/nginx/nginx.conf
        owner: root
        group: root
        mode: '0644'

    - name: Ensure required packages installed
      ansible.builtin.package:
        name:
          - nginx
          - python3
          - htop
        state: present

    - name: Ensure unwanted packages removed
      ansible.builtin.package:
        name:
          - telnet
          - rsh
        state: absent

Run with --check --diff to see drift without fixing it.

Automated Drift Detection Playbook

---
- name: Configuration drift audit
  hosts: all
  gather_facts: true
  tasks:
    - name: Check SSH config
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^PermitRootLogin'
        line: 'PermitRootLogin no'
      check_mode: true
      register: ssh_drift

    - name: Check firewall rules
      ansible.builtin.command:
        cmd: iptables -L -n
      register: firewall_output
      changed_when: false

    - name: Check NTP configuration
      ansible.builtin.lineinfile:
        path: /etc/chrony.conf
        regexp: '^server'
        line: 'server ntp.company.com iburst'
      check_mode: true
      register: ntp_drift

    - name: Report drift
      ansible.builtin.debug:
        msg: |
          Drift detected on {{ inventory_hostname }}:
          SSH config: {{ 'DRIFTED' if ssh_drift.changed else 'OK' }}
          NTP config: {{ 'DRIFTED' if ntp_drift.changed else 'OK' }}

Remediation Strategies

Write playbooks that describe the desired state, not steps. Run them regularly:

# GOOD — declarative, idempotent
- name: Ensure nginx is running and enabled
  ansible.builtin.service:
    name: nginx
    state: started
    enabled: true

# AVOID — imperative, not idempotent
- name: Start nginx
  ansible.builtin.command: systemctl start nginx

2. Scheduled Runs

Run playbooks on a schedule via cron or Ansible Automation Platform:

# Crontab: enforce state every 4 hours
0 */4 * * * cd /opt/ansible && ansible-playbook site.yml --diff 2>&1 | mail -s "Ansible drift remediation" ops@company.com

3. Compliance Reporting with Tags

---
- name: Compliance check
  hosts: all
  tasks:
    - name: CIS benchmark - SSH protocol version
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        regexp: '^Protocol'
        line: 'Protocol 2'
      tags: [compliance, cis, ssh]

    - name: CIS benchmark - password max days
      ansible.builtin.lineinfile:
        path: /etc/login.defs
        regexp: '^PASS_MAX_DAYS'
        line: 'PASS_MAX_DAYS 90'
      tags: [compliance, cis, password]

Run compliance checks only:

ansible-playbook site.yml --tags compliance --check --diff

Tools for Drift Detection

ToolDescription
ansible-playbook --check --diffBuilt-in dry run with diff output
ARA Records AnsibleWeb dashboard for playbook history
Ansible Automation PlatformEnterprise drift management
InSpec/OpenSCAPCompliance frameworks that integrate with Ansible

Best Practices

  1. Run playbooks regularly — Don't let drift accumulate
  2. Use --diff mode — See exactly what changed
  3. Lock down manual access — Minimize SSH access to production
  4. Version control everything — All playbooks in Git
  5. Alert on drift — Integrate check mode output with monitoring
  6. Document exceptions — If a server intentionally differs, document why

Conclusion

Configuration drift is inevitable without automation. Ansible's idempotent design makes it the ideal tool for both detection (check mode) and remediation (regular playbook runs). Start by running your existing playbooks with --check --diff to see how much drift exists today.

All Tutorials