Enforcing Network Standards Across a 1,000+ Device Fleet with Agentic AI

Enforcing Network Standards Across a 1,000+ Device Fleet with Agentic AI

Enterprise networks rarely fail from a single bug—they fail from configuration entropy:

  • Interface descriptions referencing retired peers (TEMP_UPLINK_DO_NOT_TOUCH)
  • Drifted hostnames across datacenters, cloud interconnects, and branches
  • Legacy ACLs & firewall policies with shadowed rules, any services, and stale objects
  • Undocumented BGP neighbors, VTI tunnels, and LAG members

Manual audits take months. Here is how we built an Agentic AI (ai-ops) pipeline to enforce deterministic standards across 1,000+ devices—without letting an LLM wing a live commit:

1. Multi-Source Truth Correlation (~50ms)

Before proposing changes, agents correlate 4 live sources:

  • Oxidized: Config history across firewalls, switches, and routers
  • NetBox (SSOT): Authoritative racks, circuits, prefixes, and roles
  • LibreNMS & Prometheus: Live LLDP/CDP, ARP/MAC, and BGP state
  • Graphify Graph: Pre-compiled topology queried in <150 tokens for blast radius

2. Semantic Interface, Hostname & Tunnel Normalization

Cross-verifying LLDP/CDP and BGP state against IPAM enforces structured schemas:

  • Interfaces: <peer_host>:<peer_port> | <role/circuit> | <vrf/vlan>
  • Hostnames: <site>-<role>-<id> aligned across CLI, DNS, IPAM, and monitoring
  • VTI Tunnels: Standardized metadata with remote ASN, loopbacks, and routing boundaries (cloud workload vs. management paths)

3. Autonomous ACL & Firewall Policy Hygiene

Agents parse vendor syntax into normalized models to:

  • Replace inline IPs with canonical, role-scoped address objects
  • Flag shadowed rules, expired exceptions, and zero-hit policies via hit counters
  • Enforce least-privilege segmentation across corporate, cloud, and branch zones

4. Orchestrator + Disposable Sub-Agent Architecture

To prevent LLM context rot at scale, we use a two-tier design:

  • Orchestrator (<40k tokens): Plans site waves and synthesizes fleet diffs
  • Disposable Sub-Agents: Audit one site/rack, query the subgraph, and output compact diffs—cutting tokens ~80% with 100% precision

5. Candidate Staging & Human-in-the-Loop Approval

AI never autonomously commits to live gear:

  1. Stage in candidate config or dry-run mode
  2. Run syntax, policy, and routing pre-checks
  3. Attach an exact Before/After diff to the change ticket
  4. Require Human Approval before commit + post-change BGP/port verification

The Result

Multi-quarter cleanups are now a continuous pipeline—turning show interface status into a living diagram.

How are you tackling config drift and ACL hygiene at scale?

Related Engineering Notes

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.