# How Crosby is building an AI-native law firm with Devin ![Vimeo](https://devin.ai/_next/image?url=%2F_next%2Fstatic%2Fimmutable%2Fmedia%2Fcover.0le34rtffnk3g.webp\&w=3840\&q=75) 20–40/day Bugs and alerts getting a first pass from Devin 5 mins Time for most bugs and alerts to get a first pass 1 engineer On-call size maintained as legal workforce 5x'ed 50% Reduced noise in Sentry alerts ## About the company Founded in New York in 2024, [Crosby](https://crosby.ai/) is an AI-native law firm working with companies including Cursor, Ramp, Clay, and Granola. The company has raised $86 million from investors including Lux, Index, Sequoia, and Bain Capital Ventures. Industry: Legal Services**[Visit site](https://crosby.ai/)** ## Paying for delivered work, not hours Crosby is rebuilding the law firm around a simple idea: customers should pay for legal outcomes rather than lawyer hours. AI agents research, analyze, draft, and learn each customer's negotiating preferences. Crosby attorneys review, make adjustments where necessary, and deliver the legal work to clients. Crosby charges fixed rates by the document, so better software means faster turnaround, more consistent work, lower costs, and more lawyer time for judgment, negotiation, and advising clients. Commercial contracts are the starting point. Crosby's broader ambition is a law firm where AI takes on more of the work of delivering legal services, from anticipating counterparty responses to negotiating on clients' behalf. Crosby's own software is therefore part of the legal service, and Raymond Lin, a founding engineer, describes its scope as "everything from the client relations to the actual tools that our lawyers are using." Crosby builds all of it with about 12 engineers for 50-plus attorneys and its operations and go-to-market teams. ## Software development guided by legal experts At Crosby, the lawyers doing the legal work also help improve the software used to deliver it, and they work directly with Devin to do it. Leah Yared, a legal engineer with a Harvard JD/MBA who practiced at Cravath, reviews contracts, works out customers' negotiating preferences, encodes those preferences into rules Crosby's AI applies on every review, and runs evals on the output. > "The goal is to really get the lawyers involved in the actual development of our tools." > > — Leah Yared, Legal Engineer The most direct path for lawyers to influence the product is in reviews. Each time an attorney reviews a contract with Crosby's AI, they are asked what went well and what was unusual, and the answers post to an internal Slack channel where Devin is a member. Devin labels each message as a feature request or a bug, notes when a bug has already been reported and has a ticket, and groups reports of the same problem for engineers to see. Engineers review the grouped feedback in their engineering meetings, and legal engineers use it to decide whether a problem needs a fix to the system or training for the team that reported it. Evals are the second path. An eval tests Crosby's contract-review AI against contracts where lawyers have already marked what the correct redlines and comments should be, so it shows whether a change to a prompt or a rule made reviews better or worse. Legal engineers choose which contracts to test on and define the right answer, and use the results to improve the rules they have encoded. Legal engineers also prototype ideas themselves with Devin and bring working versions to engineering. The evals platform is a major area where the legal team has contributed using Devin. Product managers prototype with Devin too. Sreya Guha, a product manager, can get to a working version of a new idea in about an hour, show it to a lawyer, get feedback, and revise it. When a bug or feature request comes in, she investigates it herself with Devin. Crosby is also building benchmarks to evaluate frontier models on the workflow of senior commercial lawyers in live contract negotiations, and it is funding research into problems including expert disagreement, realistic negotiation, precise contract editing, and knowing when an AI should hand work to a human. ## Building a self-improving system Crosby's engineers use Devin for end-to-end engineering work. They have configured the environment and instructions Devin needs to run Crosby's applications, including how to simulate the Slack and contract-management integrations that feed them, and how to determine whether a task succeeded. Devin runs the software in its own virtual machine, with its own shell, editor, and browser, tests the result, and keeps working until CI passes. The team likes that Devin records video of its work. When Guha was ramping into her product role, she asked Devin to walk through every path a user could take through one of Crosby's systems and show her how each one behaved, and Devin returned videos of the flows. She uses the videos to prototype and test new flows. > "The ability to say, this is what I think I want, and then see it right away through the video is amazing." > > — Sreya Guha, Product Manager Lin spends much of his time running experiments on Crosby's AI systems. When he wants to try a new eval, he tells Devin to set up the cloud environment, run the experiment against Crosby's infrastructure, and return the result. The biggest unlock, Lin says, is convenience rather than simply the number of experiments he can run. He can launch experiments without maintaining the environment on his own machine, check results remotely, and easily share Devin sessions with teammates. When Lin iterates on an eval, Devin groups similar failures, tries changes to the prompts or context, reruns the evals, and compares results. Engineers spend more time on the systems that let agents implement, test, and refine the software reliably, and more time on what Crosby should build. Vinay Khemlani, a member of technical staff, describes the shift as thinking "more about the business and less about how things get done." > "Where a lot of engineering is going is defining the interfaces to systems and the verification loops agents need to run: verifying that given inputs create the right outputs. We've invested in the groundwork and given Devin the instructions to set up this environment. It understands what the verification loop should look like for a given ask, and does a good job iterating until it's satisfied, without requiring someone to provide a ton of technical detail. That's core to how we want to operate as we scale." > > — Vinay Khemlani, Member of Technical Staff ## Automating bug reports and alerts Crosby also uses Devin to automate the first response to problems in its software. Lin built Crosby's original on-call process. With 12 engineers each specialized in different parts of the platform, Crosby runs a single weekly rotation, and on-call is treated as a full engineer's capacity for the week. Crosby has kept that rotation to a single engineer even as its legal workforce has grown roughly 5x. Today Devin takes a first pass at roughly 20 to 40 bug reports and automated alerts every day. Crosby's lawyers report bugs in a Slack channel, and an automation starts a Devin session on each new report. Devin reads the report, pulls the relevant traces and logs, runs the applicable dev scripts and queries the data, and posts what it found in the thread, so the investigation is waiting for the on-call engineer when they open it. For most bugs and alerts, Devin returns a reasonable first pass in about five minutes. Lin says that is roughly as fast as an engineer already very familiar with the system could investigate the issue themselves — increasingly useful as Crosby's systems grow more complex and it becomes harder for any one engineer to maintain context across the platform. When Devin can identify the fix, it opens a pull request. The same automations watch Crosby's alerting. Devin's automated first pass on Sentry errors has reduced alert noise by at least 50%. Errors that had previously been too noisy to act on now arrive with a PR or a recommendation to silence the error, which the engineer can confirm. When a deploy recently collided with an existing database revision, Devin caught the conflict and opened a PR before the deployment pipeline froze. For common issues, Devin answers the reporter directly. Crosby's lawyers work heavily in Microsoft Word, and one known issue occurs when its Word add-in fails to refresh. Lin wrote a knowledge note with the symptom and the refresh step, so when the issue appears again Devin gives the lawyer the instruction Lin would otherwise have sent himself. > "I would rather wait 5 to 10 minutes for Devin to take a first stab at it. Even if it doesn't end up being correct, it's explored one path that I would have explored." > > — Raymond Lin, Founding Engineer Lin estimates that Devin saves the on-call engineer one to two hours per day. The engineer on rotation can now get some product work done during the week. ## Delivering better results to customers Crosby is building a self-improving law firm, and Devin is a key part of it. A customer preference, a correction from a lawyer, a bug, or a failed eval each reveals something the software should do differently. Crosby's system captures those signals, tests improvements, and automates more of the cycle from identifying a problem to verifying a better result. Lawyers bring their legal judgment to the system and partner with engineers to improve it. For customers, the result is faster contracts, more consistent application of their negotiating preferences, and better legal judgment.