Curating calls for papers for legal academics is real work requiring staff to read sources in different formats, extract deadlines, classify topics, catch duplicates, and ultimately to exercise judgment about what is and what is not an opportunity to publish. The Docket, our Python and JavaScript CFP aggregator, staffs this work with AI agents driven by Claude models from Anthropic. A fast, inexpensive model handles routine extraction across hundreds of sources; a more capable model solves problems; we adjust the system when they get it wrong. Three months of building taught us something we didn’t expect: every time an agent failed, the fix looked like better management — clearer instructions, narrower responsibilities, and specialists for the hard cases. The main role of the human-in-the-loop was not to countermand specific decisions but to create policy to solve problems permanently. We started with one agent to do it all. We quickly hit problems any human manager might recognize: too many responsibilities degraded quality, some tasks required special expertise, and a second agent offered strategic advice when its job was data entry. Each restructuring taught us something about how to divide responsibilities and what understanding each job actually required. The system still escalates problems to a human who can review any agent’s reasoning and override any decision, but this is a backstop. The human’s primary role is to supervise processes: to observe what agents report and where failures cluster. The human work is reading these patterns, not reading CFPs. The Docket has been live since late 2025, processing nearly 19,000 agent decisions across 500+ sources, and is in pilot with faculty at Richardson. This session is a case study in overcoming apparent limitations on AI’s ability to do real wo rk by reframing them as a staffing problem we already know how to solve.

Speaker