Letting Agents Break Things

A small robot sitting in a snow-dusted sandbox marked OVHcloud, wiring together toy routers and switches next to a containerlab logo drawn in the sand

Allowing LLMs to make unattended changes in live networks is either lunacy or a constant fight against guardrails. Getting the most out of our new plinko-brained companions does however require letting them experiment and break things on their own. So I give them isolated & ephemeral labs built with Pi, containerlab and some homebrewed tooling.

TL;DR

Level-set

Agents are LLMs with tools queried in a loop. They act, read the result and evaluate what the next action should be. Given good & sufficient data they are starting to perform pretty well on networking tasks. The downside is that when they are wrong they tend to be confidently wrong.

When prompting your favourite model by hand you catch this, correct it and it will often find its way back on the narrow path. This works well. You can have the LLM perform most of the yak-shaving, but you still need to pay attention to it and do the nudging.

To tighten the loop you want to remove yourself from it. Without you the agent needs something else to tell it when it is wrong, and that should not be your live network. Coding agents get this from compilers and test suites. In networking this is where sandboxes come in. Given the right directions your agents can plan, test and verify before suggesting any actions that affect your live environment.

Sandbox

There are a few things that shape what the implementation of this looks like for me:

I am the only user
The environments need to be ephemeral
Environments need to become available fast
Isolation must be enforced externally
My provider of choice is OVH (see Switching to Containerlab).

The resulting architecture looks like this:

Sketch of the sandbox architecture: Pi and the lab-factory CLI on my machine, a sandbox VM inside an OVH security group with internet access blocked

Code has become cheap, so I will mostly focus on design choices here.

I implemented the orchestration in the form of a CLI tool written in bash. Each sandbox consists of a VM deployed from a golden image. My image library is attached as a cloned copy-on-write volume. There is nothing that needs to be pulled, so the host is ready as fast as the server is able to boot (about 2 minutes).

Containerlab was chosen for this project as LLMs are exceedingly good at writing lab specifications. The format is text-based, simple and well documented. It also lends itself to simple lifecycle management: There is no license registration to be handled, installation is done by fetching a binary, and the images are portable.

Egress traffic is denied by an OVH security group, so a misbehaving or prompt-injected agent can’t attack anything from the lab or send data out of it. Any external information must therefore be fetched through other tools instead of from inside the sandbox.

Every sandbox has a fixed lifetime that is set when it is spawned, enforced by a background job on my workstation that runs the destroy workflow. This ensures that I won’t be paying in perpetuity for the instance even if (when) the agent doesn’t clean it up itself. Due to the nature of the local CLI-driven orchestration this is however dependent on my workstation staying up.

Harness

My harness of choice is Pi. It lends itself well to workflows like this due to being minimalist at its core and very extensible. Pi runs on my workstation as opposed to on the ephemeral server, which ensures that it can’t alter its own configuration.

Pi is a provider independent harness. I have been running it with Mistral Large 4 through a Mistral Vibe subscription. The token budget is generous and the subscription permits the use of third party harnesses without breaking TOS.

The harness can utilize the CLI tool through a set of tools. Two to handle lifecycle (lab_spawn, lab_destroy) and four to interact with the lab (lab_bash, lab_read, lab_write, lab_edit). The interactive ones are Pi’s builtin tools, but run over SSH. By keeping the tool names separated instead of redirecting the builtin tools I am able to give a session local tools, lab tools or both.

I created a sub-agent to enable delegating autonomous work within my Pi sessions. A Pi sub-agent consists of a single file containing a header defining available tools and a system-prompt. It only has the interactive lab tools and a select set of safe tools for fetching external data. None of these execute anything on my workstation, so whatever the sub-agent decides to do it can only do inside the sandbox. The sub-agent is able to raise questions to the supervisor/main session when required, but otherwise only returns its last response.

Workflow

Whenever I have a task that I think would benefit from this I start by gathering the required information, whether that be log dumps, output copy-pasted from SSH sessions or data pulled from the network through MCP. Then I ask my main agent to formulate a spec for the task. Spending some time validating this is important as this is the main place I can course-correct before launching the sub-agent.

One thing I discovered after trying and failing a few times is that the agent that has interactive access to the sandbox and containerlab needs to create the topology file itself. It needs a few attempts to get the lab right as well, and that is best done when the feedback is immediate. Make sure that you include enough information in the task spec for it to do so on its own.

Depending on the task and topology the sub-agent can run for a while. The example below took a few hours.

If I want to validate the agent’s work afterwards I can access the sandbox environment directly from my main agent before destroying it. This also lets me work in the lab myself, with the agent alongside.

In practice

A variation on the EEM applet from my PnP licence level post worked flawlessly on IOS-XE 17.9, but on 17.12 routers started losing their licence level on the first reboot. Each attempt at reproducing this takes 20 minutes as you need to modify, boot, provision, reboot, then verify. This is also an edge case behavior that is not documented anywhere, and there is no bug for it in the BST. Finding the root cause would require a lot of trial and error.

From working similar issues previously I had some hunches about which parameters mattered for this. So I gave my Pi session sanitized versions of the applets and asked the agent to work through variations of it with differing parameters. My main agent deployed a sandbox, and the sub-agent created multiple containerlab environments with single c8000v routers to test with. After a few hours and many iterations it found the cause: writing the config twice in quick succession right after setting the boot variable caused the error to occur.

Once the agent had finished its work I just needed to verify it and remediate it by making a slight logic change that saved the config only once at the end of the provisioning process. Estimating how much time this saved me is difficult, but this exact combination of parameters was not high on my list of things to test.

Here are some other use cases I have had success with:

Validating planned migrations: A plan is good, a plan that is rehearsed is better. If an agent can flush out both known and unknown unknowns before even starting a change it can save a lot of headache at 1am.
Creating mock networks: When I face complex tasks with protocols that I don’t work with day to day it is useful to work it through for my own comprehension. Getting an agent to do the yak-shaving lets me spend more time on the network itself.
Writing IaC code: LLMs write great looking code that often doesn’t account for platform specific behaviors and NOS quirks. Giving it free access to the sandbox saves a lot of correction/re-prompting.

One use case I have yet to try, but have big hopes for is validating LLDs against requirements. Thorough testing is tedious, so it is easy to make mistakes based on assumptions (and assumption is the mother of all fuckups!). Using agents to achieve a higher test coverage would provide a lot of value for an LLD.

Keep in mind that virtual devices only get you so far. They don’t have ASICs or hardware queues, and will behave differently than hardware for technologies like QoS.

Summary

Agents perform best when they can get feedback as to whether something works or not. In coding this comes from compilers and tests, in networking we can use virtual network sandboxes. I implemented this as short-lived OVH VMs running containerlab, with egress denied by a security group and a fixed lifetime. The Pi harness runs on my workstation and a sub-agent with only lab tools does the unattended work in the sandbox.

For a given task I feed my main agent relevant data and let it write a task spec, which I thoroughly validate before handing it to a sub-agent. The sub-agent autonomously builds the lab, works through the task, and leaves the sandbox up for me to review the result.

My homemade orchestration tooling is available in the lab-factory repo, mostly for inspiration as I will not be maintaining it.

Table of contents