How to create an API Token?

Q: How to create an API Token?

Or create one in the UI: Settings → Tokens → Add Token (read scope only).

Ansible Controller — Prometheus Grafana

Introduction

Ansible Automation Controller (formerly AWX/Tower) exposes a Prometheus-compatible /api/v2/metrics endpoint that provides real-time data about job execution, host status, system performance, and more. By connecting Prometheus to scrape these metrics and Grafana to visualize them, you get a complete real-time monitoring stack for your automation platform. This article covers the full integration setup from metrics endpoint to production dashboards.

Architecture Overview

┌──────────────────────┐     scrape      ┌──────────────┐
│ Automation Controller│────/metrics────→│  Prometheus   │
│ (ac.example.com)     │   every 5s      │ (time-series │
└──────────────────────┘                  │  database)   │
                                          └──────┬───────┘
                                                 │ query
                                          ┌──────▼───────┐
                                          │   Grafana     │
                                          │ (dashboards)  │
                                          └──────────────┘

Prerequisites

Component	Version	Purpose
Automation Controller	4.x+	Metrics source
Prometheus	2.x+	Metrics collection and storage
Grafana	9.x+	Visualization and alerting
Network access	HTTPS 443	Controller → Prometheus

Step 1: Enable Metrics on Automation Controller

The metrics endpoint is available at https://controller.example.com/api/v2/metrics. You need an API token:

Create an API Token

# Via the API
curl -k -X POST \
  https://ac.example.com/api/v2/tokens/ \
  -H "Content-Type: application/json" \
  -u admin:password \
  -d '{"scope": "read"}'

Or create one in the UI: Settings → Tokens → Add Token (read scope only).

Verify Metrics Endpoint

curl -k -H "Authorization: Bearer YOUR_TOKEN" \
  https://ac.example.com/api/v2/metrics/

Available Metrics

Metric	Type	Description
`awx_system_info`	gauge	Controller version and install info
`awx_organizations_total`	gauge	Total organizations
`awx_users_total`	gauge	Total users
`awx_teams_total`	gauge	Total teams
`awx_inventories_total`	gauge	Total inventories
`awx_projects_total`	gauge	Total projects
`awx_job_templates_total`	gauge	Total job templates
`awx_workflows_total`	gauge	Total workflow templates
`awx_hosts_total`	gauge	Total hosts (by status)
`awx_schedules_total`	gauge	Total schedules
`awx_running_jobs_total`	gauge	Currently running jobs
`awx_pending_jobs_total`	gauge	Queued jobs waiting to run
`awx_status_total`	gauge	Jobs by status (successful, failed, error)
`callback_receiver_events_*`	counter/gauge	Event processing metrics
`subsystem_metrics_*`	gauge	Internal task system metrics

Step 2: Configure Prometheus

prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s
  scrape_timeout: 10s

  external_labels:
    environment: production

rule_files:
  - /etc/prometheus/rules/*.rules

scrape_configs:
  # Prometheus self-monitoring
  - job_name: prometheus
    metrics_path: /metrics
    static_configs:
      - targets:
          - localhost:9090

  # Node exporters for system metrics
  - job_name: node
    file_sd_configs:
      - files:
          - /etc/prometheus/file_sd/node.yml

  # Automation Controller
  - job_name: automation_controller
    metrics_path: /api/v2/metrics
    scrape_interval: 5s
    scheme: https
    tls_config:
      insecure_skip_verify: true  # Use proper certs in production
    bearer_token: YOUR_API_TOKEN
    static_configs:
      - targets:
          - ac.example.com
        labels:
          instance: production-controller

Using a Token File (More Secure)

  - job_name: automation_controller
    metrics_path: /api/v2/metrics
    scrape_interval: 5s
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/certs/controller-ca.crt
    bearer_token_file: /etc/prometheus/secrets/controller-token
    static_configs:
      - targets:
          - ac.example.com

Start Prometheus

# Docker
docker run -d \
  --name prometheus \
  -p 9090:9090 \
  -v /etc/prometheus:/etc/prometheus \
  prom/prometheus \
  --config.file=/etc/prometheus/prometheus.yml

# Or systemd
sudo systemctl start prometheus

Verify Scraping

Visit http://prometheus:9090/targets — the automation_controller target should show UP.

Step 3: Configure Grafana

Add Prometheus Data Source

Navigate to Configuration → Data Sources → Add data source
Select Prometheus
Set URL: http://prometheus:9090
Click Save & Test

Or via provisioning:

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    url: http://prometheus:9090
    access: proxy
    isDefault: true

Build Dashboards

Job Success Rate Panel

# PromQL — success rate last 24h
sum(awx_status_total{status="successful"}) /
(sum(awx_status_total{status="successful"}) + sum(awx_status_total{status="failed"})) * 100

Running Jobs Gauge

# PromQL — currently running jobs
awx_running_jobs_total

Pending Jobs Queue

# PromQL — jobs waiting in queue
awx_pending_jobs_total

Host Status Breakdown

# PromQL — hosts by status
awx_hosts_total

Job Failure Rate Over Time

# PromQL — failed jobs per hour
rate(awx_status_total{status="failed"}[1h])

Dashboard Layout Recommendations

┌─────────────────────────────────────────────────┐
│ Row 1: Overview Stats                           │
│ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐ ┌──────┐  │
│ │ Jobs │ │ Pass │ │ Fail │ │Queue │ │ Run  │  │
│ │Total │ │ Rate │ │ Rate │ │ Depth│ │ ning │  │
│ └──────┘ └──────┘ └──────┘ └──────┘ └──────┘  │
├─────────────────────────────────────────────────┤
│ Row 2: Time Series                              │
│ ┌───────────────────────┐ ┌───────────────────┐ │
│ │ Jobs Over Time        │ │ Host Status       │ │
│ │ (stacked area)        │ │ (pie chart)       │ │
│ └───────────────────────┘ └───────────────────┘ │
├─────────────────────────────────────────────────┤
│ Row 3: Details                                  │
│ ┌───────────────────────┐ ┌───────────────────┐ │
│ │ Failure Rate Graph    │ │ Event Processing  │ │
│ │ (line chart)          │ │ (counter rate)    │ │
│ └───────────────────────┘ └───────────────────┘ │
└─────────────────────────────────────────────────┘

Step 4: Create Alerting Rules

Prometheus Alert Rules

# /etc/prometheus/rules/controller.rules
groups:
  - name: automation_controller
    rules:
      - alert: HighJobFailureRate
        expr: |
          sum(rate(awx_status_total{status="failed"}[5m])) /
          sum(rate(awx_status_total[5m])) > 0.1
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "High job failure rate on {{ $labels.instance }}"
          description: "More than 10% of jobs failing in the last 5 minutes"

      - alert: JobQueueBacklog
        expr: awx_pending_jobs_total > 20
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Job queue backlog on {{ $labels.instance }}"
          description: "{{ $value }} jobs pending in queue"

      - alert: ControllerDown
        expr: up{job="automation_controller"} == 0
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Automation Controller unreachable"
          description: "Cannot scrape metrics from {{ $labels.instance }}"

Grafana Alert Example

Configure in Grafana UI: Alerting → Alert Rules → New Alert Rule

Setting	Value
Query	`awx_pending_jobs_total`
Condition	When last() is above 20
Evaluate every	1m
For	5m
Notification	Slack/Email/PagerDuty

Ansible Playbook for Deployment

Automate the entire monitoring stack setup:

---
- name: Deploy monitoring stack
  hosts: monitoring
  become: true
  vars:
    controller_url: ac.example.com
    controller_token: "{{ vault_controller_token }}"
  tasks:
    - name: Install Prometheus
      ansible.builtin.package:
        name: prometheus
        state: present

    - name: Deploy Prometheus config
      ansible.builtin.template:
        src: prometheus.yml.j2
        dest: /etc/prometheus/prometheus.yml
        mode: "0644"
      notify: Restart Prometheus

    - name: Deploy alert rules
      ansible.builtin.copy:
        src: controller.rules
        dest: /etc/prometheus/rules/controller.rules
        mode: "0644"
      notify: Restart Prometheus

    - name: Install Grafana
      ansible.builtin.package:
        name: grafana
        state: present

    - name: Deploy Grafana data source
      ansible.builtin.template:
        src: grafana-datasource.yml.j2
        dest: /etc/grafana/provisioning/datasources/prometheus.yml
        mode: "0644"
      notify: Restart Grafana

    - name: Ensure services are running
      ansible.builtin.service:
        name: "{{ item }}"
        state: started
        enabled: true
      loop:
        - prometheus
        - grafana-server

  handlers:
    - name: Restart Prometheus
      ansible.builtin.service:
        name: prometheus
        state: restarted

    - name: Restart Grafana
      ansible.builtin.service:
        name: grafana-server
        state: restarted

Security Best Practices

Use read-only tokens — metrics endpoint only needs read scope
Enable TLS — use proper certificates, not insecure_skip_verify
Restrict network access — only Prometheus should reach the metrics endpoint
Rotate tokens — set token expiry and rotate regularly
Use token files — bearer_token_file over inline tokens in config
RBAC in Grafana — limit dashboard access to operations team

Troubleshooting

Prometheus Target Shows DOWN

# Check connectivity
curl -k https://ac.example.com/api/v2/metrics/ \
  -H "Authorization: Bearer YOUR_TOKEN"

# Check Prometheus logs
journalctl -u prometheus | tail -20

No Data in Grafana

Verify Prometheus data source connection (Settings → Data Sources → Test)
Check time range in dashboard (last 5 minutes vs last 24 hours)
Query up{job="automation_controller"} in Prometheus to verify scraping

Metrics Stale or Delayed

# Reduce scrape interval for fresher data
scrape_interval: 5s  # default is 15s

Conclusion

Integrating Automation Controller with Prometheus and Grafana gives you real-time visibility into your automation platform — job success rates, queue depth, host status, and system health on a single dashboard. Use the /api/v2/metrics endpoint with a bearer token, configure Prometheus to scrape every 5 seconds, and build Grafana dashboards with the PromQL queries above. Add alerting rules for job failures and queue backlogs to catch issues before they impact operations.

Ansible Controller — Prometheus Grafana

Introduction

Architecture Overview

Prerequisites

Step 1: Enable Metrics on Automation Controller

Create an API Token

Verify Metrics Endpoint

Available Metrics

Step 2: Configure Prometheus

prometheus.yml

Using a Token File (More Secure)

Start Prometheus

Verify Scraping

Step 3: Configure Grafana

Add Prometheus Data Source

Build Dashboards

Job Success Rate Panel

Running Jobs Gauge

Pending Jobs Queue

Host Status Breakdown

Job Failure Rate Over Time

Dashboard Layout Recommendations

Step 4: Create Alerting Rules

Prometheus Alert Rules

Grafana Alert Example

Ansible Playbook for Deployment

Security Best Practices

Troubleshooting

Prometheus Target Shows DOWN

No Data in Grafana

Metrics Stale or Delayed

Related Articles

Conclusion