Enterprise networks rarely fail from a single bug—they fail from configuration entropy:
- Interface descriptions referencing retired peers (
TEMP_UPLINK_DO_NOT_TOUCH) - Drifted hostnames across datacenters, cloud interconnects, and branches
- Legacy ACLs & firewall policies with shadowed rules,
anyservices, and stale objects - Undocumented BGP neighbors, VTI tunnels, and LAG members
Manual audits take months. Here is how we built an Agentic AI (ai-ops) pipeline to enforce deterministic standards across 1,000+ devices—without letting an LLM wing a live commit:
1. Multi-Source Truth Correlation (~50ms)
Before proposing changes, agents correlate 4 live sources:
- Oxidized: Config history across firewalls, switches, and routers
- NetBox (SSOT): Authoritative racks, circuits, prefixes, and roles
- LibreNMS & Prometheus: Live LLDP/CDP, ARP/MAC, and BGP state
- Graphify Graph: Pre-compiled topology queried in
<150tokens for blast radius
2. Semantic Interface, Hostname & Tunnel Normalization
Cross-verifying LLDP/CDP and BGP state against IPAM enforces structured schemas:
- Interfaces:
<peer_host>:<peer_port> | <role/circuit> | <vrf/vlan> - Hostnames:
<site>-<role>-<id>aligned across CLI, DNS, IPAM, and monitoring - VTI Tunnels: Standardized metadata with remote ASN, loopbacks, and routing boundaries (cloud workload vs. management paths)
3. Autonomous ACL & Firewall Policy Hygiene
Agents parse vendor syntax into normalized models to:
- Replace inline IPs with canonical, role-scoped address objects
- Flag shadowed rules, expired exceptions, and zero-hit policies via hit counters
- Enforce least-privilege segmentation across corporate, cloud, and branch zones
4. Orchestrator + Disposable Sub-Agent Architecture
To prevent LLM context rot at scale, we use a two-tier design:
- Orchestrator (
<40ktokens): Plans site waves and synthesizes fleet diffs - Disposable Sub-Agents: Audit one site/rack, query the subgraph, and output compact diffs—cutting tokens ~80% with 100% precision
5. Candidate Staging & Human-in-the-Loop Approval
AI never autonomously commits to live gear:
- Stage in candidate config or dry-run mode
- Run syntax, policy, and routing pre-checks
- Attach an exact Before/After diff to the change ticket
- Require Human Approval before commit + post-change BGP/port verification
The Result
Multi-quarter cleanups are now a continuous pipeline—turning show interface status into a living diagram.
How are you tackling config drift and ACL hygiene at scale?