Revolutionizing AI Research: How Agent-Driven Development with GitHub Copilot Transforms Workflows

Software engineers have a knack for automating the tedious parts of their work—whether out of inspiration, frustration, or sheer laziness. They build systems to eliminate drudgery, then end up maintaining those very systems for the benefit of their peers. As an AI researcher, I recently took this concept to a new level: I automated my own intellectual toil. Now I maintain a tool that lets my entire team at Copilot Applied Science do the same.

This journey taught me a great deal about effectively creating and collaborating with GitHub Copilot. The insights unlocked an incredibly fast development loop for me and empowered my teammates to craft solutions tailored to their needs. Let me walk you through how this came to be.

The Challenge: Analyzing Thousands of Agent Trajectories

A significant part of my job involves evaluating coding agent performance against standardized benchmarks like TerminalBench2 or SWEBench-Pro. This means sifting through countless trajectories—records of the thought processes and actions agents take while solving tasks. Each task in a benchmark set generates its own trajectory, typically a .json file hundreds of lines long. Multiply that by dozens of tasks per benchmark, and then by the many runs analyzed daily, and you’re looking at hundreds of thousands of lines of code to review.

Doing this manually is impossible. So I turned to GitHub Copilot for help. It surfaced patterns in the trajectories, reducing my reading load from hundreds of thousands to just a few hundred lines. But I found myself repeating the same loop: ask Copilot to find patterns, then investigate them myself. The engineer in me rebelled: “I want to automate this.”

The Repetitive Loop

The cycle was efficient but still manual. I’d use Copilot to highlight anomalies or common errors across trajectories, then dive into the details. Each new benchmark run meant another iteration. That’s when I realized that agents themselves could automate this intellectual work. Thus, eval-agents was born.

Building the Solution: Introducing Eval-Agents

Engineering and science teams work better together. That principle guided my design of eval-agents. I set three clear goals:

• Make these agents easy to share and use – so everyone on the team can benefit.
• Make it easy to author new agents – lowering the barrier to contribution.
• Make coding agents the primary vehicle for contributions – turning analysis into a collaborative, agent-driven process.

These goals align with GitHub’s core values. My background as an open‑source maintainer for the GitHub CLI taught me the importance of shareable, modular tools.

Implementing the System

I built eval-agents on top of GitHub Copilot, leveraging its abilities to generate code and reason about complex data. The system allows team members to define their own analysis agents—simple scripts that ingest trajectory files and output insights. These agents are stored in a shared repository, version-controlled, and easily discoverable. To author a new agent, a team member writes a natural‑language description of the analysis they want, then Copilot suggests code templates. The result is a growing library of specialized agents that anyone can run.

For example, one agent might flag trajectories where the agent repeatedly retries the same command; another might summarize success rates across benchmark categories. Each agent is self‑contained and can be combined with others for deeper analysis.

The Impact: Faster Development and Team Empowerment

The results have been transformative. My personal development loop shrank from hours to minutes. I no longer manually inspect each trajectory; instead, I invoke an agent that instantly surfaces the patterns I care about. More importantly, my teammates have embraced the tool. They’ve created agents tailored to their research questions, from identifying specific failure modes to comparing performance across agent versions.

This approach also fosters collaboration. When someone develops a new agent, they share it in our central repository. Others can review, tweak, and reuse it. The team’s collective intelligence grows with each contribution, and we spend less time on repetitive analysis and more on creative problem‑solving.

Looking Ahead: A Future of Agent‑Driven Research

Automating intellectual toil might sound paradoxical, but it’s the natural next step for AI researchers. By building agents that analyze other agents, we’ve created a self‑reinforcing cycle of improvement. The lessons I learned about agent‑driven development with GitHub Copilot are now enabling my entire team to work smarter, not harder. And who knows? Maybe I’ll soon automate myself into yet another new role—but that’s a story for another day.