Automating Intellectual Toil: How AI Researchers Leverage Copilot for Agent-Driven Development
In the world of software engineering, it's a common tale: you build a tool to eliminate repetitive tasks, only to end up maintaining that tool for your entire team. As an AI researcher on the Copilot Applied Science team, I recently pushed this concept further by automating my own intellectual toil—analyzing the decision-making logs (trajectories) of coding agents. This led me to create 'eval-agents', a system that lets my peers do the same. Below, I answer key questions about this journey, from the initial spark to the powerful collaboration it enables.
What sparked the need for agent-driven development in your work?
My daily work involves evaluating coding agents against standard benchmarks like TerminalBench2 or SWEBench-Pro. For each task in a benchmark, an agent generates a trajectory—a JSON file detailing its thought process and actions. With dozens of tasks per benchmark and multiple runs per day, I faced hundreds of thousands of lines of code to analyze. It was impossible to do manually, so I turned to GitHub Copilot to surface patterns, reducing my reading load from hundreds of thousands to a few hundred lines. But this still meant repeating the same cycle of querying Copilot, investigating, and moving on. The engineer in me saw that repetition as an opportunity: why not automate the entire analysis itself? That epiphany sparked the creation of eval-agents.
What is eval-agents and how does it work?
Eval-agents is a project that automates the intellectual toil of analyzing agent performance trajectories. Instead of me manually scanning JSON files, I built a set of agents that can ingest the trajectories, identify patterns, highlight anomalies, and summarize findings. The system is designed to be shared and extended by my entire team. At its core, it uses GitHub Copilot as a reasoning engine to interpret the complex data. When a new benchmark run comes in, the agents process the trajectories and present a distilled report. This freed me from the repetitive loop of 'ask Copilot, investigate, repeat' and turned it into a one-click process. The agents themselves are coded as simple scripts that others can modify to fit their specific analysis needs.
What were the main design goals for this project?
I built eval-agents with three guiding principles: easy sharing and use, easy authoring of new agents, and making coding agents the primary contribution vehicle. First, I wanted anyone on the team to be able to run an analysis without friction—no complex setup, just clone and go. Second, I focused on a simple, modular architecture so that colleagues could write their own agents for specialized tasks, using familiar Python patterns. Third, I encouraged team members to contribute improvements or new agents directly as code, fostering a collaborative environment. These goals align with GitHub's core values and my own experience as an open-source maintainer on the GitHub CLI. The result is a system that grows organically as new needs emerge, without me becoming a bottleneck.
How did your background in open source influence this approach?
My stint as an open-source maintainer for the GitHub CLI taught me that good tools are ones people want to use and extend. For eval-agents, I applied lessons like documentation-first design and low barriers to entry. In open source, you thrive when contributors can quickly understand your code and make changes. Similarly, I designed eval-agents with clear READMEs, examples, and a plug-in style for agents. I also emphasized testing and continuous integration, so new contributions don't break existing functionality. This open-source mindset turned my personal automation project into a team asset. My peers could fork, modify, and share their own agents, just like they would with any open-source repository. It transformed analysis from a solo chore into a shared resource.
What benefits have you and your team seen from this new workflow?
The most immediate benefit is speed: what used to take hours of sifting through trajectories now takes minutes with eval-agents. I've reclaimed time for deeper research and more creative problem-solving. My teammates can now independently run their own analyses without waiting for me to write custom scripts. This has led to faster iteration cycles—we can test a hypothesis in the morning and have results by lunch. Another unexpected outcome is that team members have started authoring their own agents for needs I hadn't anticipated, like visualizing agent decision trees or comparing performance across model versions. This collaborative growth has made our evaluation process more robust and flexible. We're no longer just analyzing agents; we're using agents to analyze agents, creating a powerful feedback loop.
What key lessons did you learn about collaborating with GitHub Copilot during this process?
I discovered that Copilot works best when you don't treat it as a magic solution but as a pair programmer that understands context. By breaking down the problem into small, well-scoped functions and providing clear comments, I found Copilot could generate accurate analysis code quickly. I also learned to iteratively refine prompts—starting broad, then narrowing with specific constraints. Another lesson: use Copilot to generate tests first, which clarified my intent and helped avoid bugs. Finally, I realized that sharing prompts and patterns with my team amplified everyone's productivity. We now maintain a shared library of effective prompts for different analysis tasks. Copilot didn't replace my thinking; it amplified it, letting me focus on the 'why' while it handled the 'how'.