What Is It Like to Train Large Models at Harvey?
Contents
“When an Agent can see 30 files and 3,000 files, no matter which model you use, the Agent that sees 3,000 files will get more out of it.” Joey Wang, on why Harvey treats engineering systems as part of model capability
Guest: Joey Wang | Engineering Lead at Harvey; previously at Figma and Uber
Harvey was valued at $15.5 billion in its latest funding round. The episode page lists more than 3,500 customers across 85 countries. The hard part of legal AI shows up in everyday work. Lawyers need to handle files in familiar software, review changes one by one, and trace where the material came from. Harvey has to connect model capability to those workflows, while also getting search, evaluation, and permissions right.
Joey Wang works on Agent engineering at Harvey and helped build Spectre, the company’s internal cloud Agent platform. Before Harvey, he worked in engineering at Figma and Uber. He joined Harvey in its early days and has seen its products grow as post-training moved closer to commercialization. We start with how lawyers review a contract, then talk about model evaluation, the people needed to train models, and the permissions required for an internal Agent.
What follows is the full conversation, edited and condensed.
1. From Figma to Harvey, Still Building Products People Use
Raymond: You joined Harvey around 2023, when the company was still very early. You had worked in engineering at Uber and Figma. Why did you want to join a legal-AI startup then?
Joey: GPT had just arrived in 2022 and 2023. At Figma, I got fairly early exposure to large language models and tried using AI for design-related work. I was working on Figma Slides and wanted to see whether AI could help people make slide decks. We could already see how capable the models were, but they did not yet understand design very well. Maybe there was not enough data; maybe the models did not yet have enough taste.
I met Harvey by chance. A friend who was doing research ran a Twitter account, and I had followed him for a while. He later introduced me to Harvey and went on to become the company’s first recruiter. Seeing someone from research take on recruiting made me think he had real conviction about the company. I spoke with the founders and wanted to see what the next generation of software companies might look like, so I was excited to join an early team.
Raymond: Uber still had a startup pace when you were there, and Figma did too when you joined. What changed at Harvey?
Joey: When I joined Uber, it was already fairly large, but it was still a startup and moved quickly. Early at Figma, decisions were also quick. Some ideas I proposed could make it into product experiments. As a company grows, more work needs approval and departments start pulling in different directions. I like building things. At Harvey, I found that early-startup pace again: we could take an idea customers actually needed and turn it into a product quickly.
Raymond: In Silicon Valley lately, I’ve heard a lot of people at big companies complain that organizational complexity keeps ideas from shipping. They want to build and ship something themselves and see customers use it. That experience can become less common as a company grows, so some people start looking for smaller companies again.
2. Lawyers Should Not Have to Learn a New Workflow for AI
Raymond: A lot of listeners have never seen Harvey. Could you start with what a lawyer does in a typical day? What work does Harvey take on?
Joey: To build a vertical product, you first have to understand how customers work. Lawyers still spend their days in Word, Excel, Outlook, and document systems. They review contracts, compare materials, research questions, draft documents, reply to email, and coordinate with colleagues and clients. The product has to fit into those existing tools and workflows.
Harvey can connect to a customer’s existing document and email systems and put Agents into that work. The interface has a chat box, of course, but the final deliverable is often still a Word document the lawyer knows how to use. The question is whether the product can carry out the work in the lawyer’s workflow and make it possible to see how each step happened.
Raymond: I used to work in investment banking and have seen a lot of NDAs. A general chatbot might revise an NDA by applying a general standard, but a lawyer’s judgment is more complicated. Some clauses must be held firm; others can be negotiated. It also depends on what each side cares about. How does Harvey handle that judgment?
Joey: Lawyers consider both sides’ positions and decide which clauses to hold firm on and which to negotiate. Take an NDA: both parties have confidentiality obligations, but they may care about different things. A general model can edit the text when prompted, but it may not know when a lawyer should give ground and when they should not.
We need to learn how lawyers have handled this work and what experience they are willing to let the system use. The data stays in the customer’s own cloud. Harvey accesses it only with permission and then uses it in the customer’s workflow. The final result remains in a format lawyers already know.
Raymond: So the deliverable is a redline, with suggested edits marked, right?
Joey: Right. The system marks the proposed changes and explains why it made them. The lawyer can review and accept them one by one.
Raymond: That’s a big difference. If you overwrite the text, the lawyer has to play “spot the difference” and figure out what changed. A redline shows which words were removed and which ones are proposed additions.
Legal documents need a record of who made each change and why. Some edits come from Harvey; others may have come from the client or opposing counsel. When the document is exported, Harvey’s changes can appear under the lawyer’s name. The lawyer still decides whether to accept and how to explain each change.
I think of it as a junior lawyer doing a first pass for a partner to review and approve. It saves repetitive work, and it also puts the result into a form a professional can take over.
Joey: Right. For lawyers, the source and reason for a change matter. Engineers look at Pull Requests to review code changes, commits, and authors. Legal documents need a similar record. People need to know what changed, who changed it, and why.
Raymond: You just said “Harvey.” Here’s a pointed question: why not call it Mike? In Suits, Harvey is the senior lawyer and Mike is the younger one below him. You do a lot of tedious work, so Mike sounds closer to what you do.
Joey: As an employee of a legal-AI company, I can neither confirm nor deny where the name came from. In the world of Suits, Harvey wants to take part in legal decisions and provide intelligence and taste.
One client told us that after Harvey took care of a lot of tedious work, he started discussing questions with Harvey that he had never considered before. Harvey raised new ideas. That showed us that it could take part in work requiring more judgment. It is still a vision, and we want the system to contribute to work with more judgment in it.
3. Vertical Products Have to Keep Up With How Customers Work
Raymond: I’ve seen M&A situations where two companies merge and the data room contains hundreds or thousands of contracts. Lawyers have to find change-of-control clauses, figure out which contracts require consent, and decide who will manage each contract after closing. Reading everything by hand takes a long time. An Agent could identify the key clauses and organize them in bulk. The system first has to find the relevant clauses across a large set of materials. Then lawyers need to know which document each result came from and which judgments they still have to review. Is Harvey bringing work lawyers already do into their daily workflow?
Joey: Right. The product has to fit into the way customers already work. Even if the underlying system is complex, customers should be able to work in tools they know. We have to understand where lawyers spend time, how they review work, and how they deliver it before we can decide where the product belongs.
Raymond: Answering a legal question is only part of the work. You have to know where the contracts are, how the materials connect, how to record the answer, and how a lawyer will review it.
Joey: Exactly. An Agent has to understand the task and work with the customer’s data and tools. After it reaches a conclusion, users need to see the sources, edits, and reasoning. A professional has to be able to take over the process.
Raymond: As models get stronger, could a general-purpose model replace Harvey? If the underlying model understands law and can read documents, what room is left for a vertical product?
Joey: Better models will change the product. We keep testing different models, and we adjust the product as their capabilities change. At the same time, Harvey has to understand customers’ workflows, connect to their data, orchestrate complex tasks, and present the result. We still have to evaluate how each model performs on actual legal work.
When Harvey was getting started, the founders used GPT-3.5 to run legal workflows and compared its results with work done by professionals. They found it could already do well on many tasks. They showed the demo to OpenAI’s General Counsel and then got early access to GPT-4. They used the newer model to build a product and sell it to law firms. Better model capability improved the product workflow, and real customers helped the team understand legal work better.
For a while, the models were not reliable enough to handle the whole process. Lawyers had to define the workflow themselves: ask users for feedback at one step, request a file at another, enter a prompt, then route the result through branch A or B before generating a document. At the time, an Agent given a single prompt might forget what it was supposed to do halfway through, stop before finishing, or generate an answer without first asking for the information it needed. Workflow Builder let people who understood the process codify it as a structure that a team or an entire firm could reuse. This year Harvey turned it into Agent Builder. The user could provide a prompt and its source materials, for example asking for a comparison under Japanese and Singapore law and a PowerPoint as the deliverable. As models improved, users could define an outcome more freely and have the Agent deliver it directly.
Raymond: A vertical product’s strength comes from the workflow customers use every day, the data they are willing to connect, and results a professional can inspect and take over. Changes in model capability reshape the product. Someone still has to work out how customer files enter the process, how lawyers review it, and how results get delivered.
Joey: Right. We have to keep understanding customers and decide where models, engineering, and product design each need to do the work. Harvey built Workflow Builder last summer. At the time, an Agent might forget the goal halfway through or generate a result before the user had provided the necessary material. Lawyers could break a task into steps, ask users for information, and set branches for different results. As models improved, Harvey moved toward Agent Builder. Users provide the goal, prompt, and data sources, and the Agent carries out the task. The customer’s document systems, review process, and delivery requirements still need product work.
4. Taste in Legal Models Is Not About Short Answers
Raymond: OpenAI has been an important partner and investor in Harvey. When did you start seriously building your own models, and why did the work begin to look commercializable?
Joey: Legal-model research has been going on since Harvey’s early days. The team worked closely with OpenAI and did some pre-training together. Back then we had fewer customers and limited post-training data; pre-training also requires huge amounts of data. Harvey once worked with PwC on a tax pre-training model, with the intention of offering it to more customers. General models improved quickly afterward, so the team began reassessing the results of building its own pre-trained models versus using general models.
As the customer base grew, product feedback that did not involve confidential customer data gave the team a better understanding of lawyers’ tasks. We also learned to simulate legal scenarios, generate synthetic data, and turn those scenarios into benchmarks. Once the evaluation set reflected real tasks, we could compare models, post-train for specific work, and weigh quality against cost and latency. The research had been under way for a long time; it was only now beginning to look commercializable.
We look at how a model performs on a particular task. Legal work has its own requirements. Taste also includes whether an answer is complete and whether the reasons and citations are clear. A model may write a long answer that others consider verbose, but lawyers may need the full reasoning and sources. Every statement should have something the lawyer can trace it back to. Shorter is not always better.
Raymond: That differs from general-purpose products that often aim for answers to be faster and shorter. A lawyer may need to know which document supports each judgment. A concise answer without citations may not work for them. This makes “taste” concrete: a lawyer may prefer to read a few more paragraphs so they can check the source of each judgment.
Joey: Right. Users care about whether the answer is accurate, whether it misses anything, and whether it fits the work. A high score on one evaluation only tells you how a model performed on that task. Legal research and extracting clauses from a batch of contracts require different capabilities. A model may be strong at reasoning but weaker in speed or reliability. We use benchmarks and customer needs to decide how to combine models.
Harvey tests models such as Kimi, GLM, Qwen, and DeepSeek. They perform differently on different tasks. We can post-train for particular work or use a stronger model as an orchestrator to decide which model or tool should handle a task.
Raymond: It’s like a library front desk that understands the question and sends it to the right book. Or a manager who assigns work to people with different specialties and then organizes the results. A stronger Agent can select the right model for each task, right?
Joey: Right. Model selection depends on performance in the product and benchmark results. We compare models on different tasks and then decide how to combine them. Some tasks need strong reasoning; others are mostly bulk processing, where speed and cost matter too. We test Kimi, GLM, Qwen, and other open-source models for particular scenarios.
We also try to leave the choice to the customer. Local rules or internal requirements may affect which models a customer can use, and some customers will specify their preferred model. A model’s data-handling policy matters too. For example, Joey said that under one provider’s policy, data would be retained by that provider for 30 days if the customer chose that model. Harvey has to make that clear so the customer can decide whether to accept it, while maintaining access and reliability.
Raymond: What is Tenet? Is it a standalone base model or a transferable training method?
Joey: Tenet is a training direction and one of the more promising projects among the models we have trained. We have trained a version on Kimi K3. We will test other base models and deploy models for different scenarios. Some do not even have separate names.
Raymond: So you are trying to build reusable training and evaluation capabilities, then choose different base models for different tasks. Tenet is one of those training directions?
Joey: Right. Tenet is one training direction. We will not use only this one model. We may not deploy every model we train to customers. We are still learning how to improve models on legal tasks. If the base model changes, our training and evaluation methods can still transfer. The point is to understand how to train and evaluate models, then determine which work each one can handle.
Raymond: What share of customer model calls currently go to Tenet or other models Harvey has trained?
Joey: I do not know the figure, and I cannot disclose it. Customers can choose models based on their preferences, and we try to give them the options they need. Anthropic’s models sometimes provide longer answers and explain why they did something, which can be useful for lawyers. Customer choice also affects usage shares.
5. Legal Benchmarks Rarely Have One Right Answer
Raymond: Training needs data, but how do you evaluate legal work? The answer can vary with the client’s situation and negotiation strategy. Who decides whether it is good enough? Beyond data volume, you need to know which questions are worth measuring and what kind of answer helps the customer.
Joey: Legal judgments often have no single right answer. Two lawyers may review the same contract and reach different conclusions because their clients and negotiation strategies differ. We break tasks down into more stable, measurable pieces and are building a Legal Agent Benchmark. Early on, we can test clearer tasks, such as whether an answer is accurate or meets a condition. Complex tasks need playbooks, evaluation criteria, and input from domain experts. We can start with a relatively stable question and specify the checks an answer must pass, then gradually increase the task’s complexity. Ultimately, evaluation has to tell us whether the Agent can complete real work.
Automated scoring is only one part of it. Harvey’s internal legal team can review results, and we also bring in outside professionals. We have to find out what users actually value and turn that into criteria we can reuse.
Raymond: How do you turn lawyers’ judgments into criteria people can discuss and compare? Ten senior lawyers may disagree when asked to evaluate the same answer. How does the team handle that?
Joey: Evaluation needs constant improvement. The team starts with typical questions and checks whether the evaluation method makes sense. Once the direction is clear, it expands to more tasks and data. Benchmarks should get closer to customers’ real work and sometimes anticipate how they will use Agents in the future. The team and legal experts need to define what to evaluate and which criteria to apply before scaling up the data.
Raymond: The work customers do will keep changing. You might test historical questions and get a high score, then customers begin using Agents for something new next month.
Joey: Evaluations have to change with the product. As the data set grows, outside vendors can help bring in more professionals to make yes-or-no judgments and produce a wider range of results. The team still has to own the scoring criteria and understand whether a problem comes from missing data, the model, or the product workflow. That takes domain experience.
Raymond: When an evaluation set grows from 50 items to 500 or 5,000, the team has to define what each question measures and keep its scoring consistent. Would you work with a vendor such as Mercor to organize more legal experts to score the results?
Joey: We will need outside talent to scale. We can use different levels of labeling and scoring, and we should not rely entirely on LLM-as-judge. If one model judges another, the evaluator’s own ability becomes a ceiling on the model we are training. Early labels, later evaluations, and scoring can use different methods to cross-check one another.
Raymond: Could a benchmark look strong, then the product launch and customers still refuse to use it?
Joey: That can happen. This work keeps iterating. When users do not buy in, we need to understand why and then change the data set and evaluation. A product is never perfect. If we can hear what users say and turn that feedback into changes to evaluation and data, the team can accumulate progress customers actually notice.
6. The Best Open-Source Model May Be Only the Starting Line
Raymond: What is the most common pitfall when a vertical company starts training its own models?
Joey: The fact that an open-source model runs does not prove that training your own model will add value. If the best open-source model is already strong in your field, post-training has to improve on it for the project to be worthwhile. You have to consider cost, quality, and latency, not just the training process. Building a benchmark does not mean you will deploy every model you evaluate. Sometimes an evaluation shows that the open-source baseline is already high. Post-training has to beat it and bring additional value in price, privacy, reliability, or user satisfaction.
Many experiments fail, and teams usually do not write about the early failures in their blogs. A benchmark may reveal that an open-source model already performs better on legal work than an OpenAI or Anthropic model. Post-training on that base model has to improve the results. Training does not always raise the score; one run might move from 78 to 75. The learning from that attempt could help the next run reach 76, then 80, 82, or 85. More importantly, a trained model does not automatically go into production. The deciding question is still whether users will use it. Training is a long learning process, and what the team learns from early failures may bring later experiments closer to the goal.
Raymond: Do early teams need a lot of researchers? Where did you find the bottleneck?
Joey: Harvey had many researchers early on. Later, we found that product talent could be scarcer. Someone has to turn user problems into clear product requirements, decide which tasks are worth solving, and decide how to evaluate them. People who understand law and can work with experts to clarify those requirements may matter more than adding more researchers. The team has to break down an expert’s “this answer is wrong” into specific causes, such as weak citations, a missed clause, the wrong comparison, or a failure to understand the parties’ negotiating positions. Those distinctions become product requirements and evaluation criteria.
Many engineers use coding agents, and lawyers can write their own benchmark or evaluation engine. People in different roles can build evaluation tools; the key is knowing what to measure.
Raymond: A lot of AI companies run into a similar experience. They build the model first, then discover the harder work is finding the customer’s real problem and spelling out what the product needs to do.
Joey: Right. Models will keep changing, but users also feel the product and engineering. We need to find improvements outside the model that make the experience better right away.
7. Search Became Ten Times Faster, Letting the Agent Read One Hundred Times More Files
Raymond: You said engineering improvements can be easier for users to feel than model changes. If a team has one budget left, how should it choose between hiring an engineer to improve the system and a researcher to try another post-trained model?
Joey: Search is one example. We brought document search down from about two seconds to 200 milliseconds. Users waited less. In a fixed amount of time, an Agent could run more searches, growing from roughly 30 files to 300. Moving from searching one file at a time to searching ten in parallel then let us expand from 300 to 3,000 files. The model did not change, but it could read a much larger context.
Raymond: A two-second search became 200 milliseconds. That is ten times faster. The Agent can now find far more material, while the model stays the same.
Joey: Right. The results are very different when an Agent can see 30 files versus 3,000. More files give the model more context to work with. We also made search parallel so the Agent could check more files within the same time. Engineering changes like that often show up in the user experience faster than waiting for the next model generation.
Raymond: A vertical product also has to get search, permissions, review, and delivery right. Slow search means the Agent sees less. Unclear permissions can prevent it from reading files safely. If results are hard to inspect, lawyers have trouble using them. Model capability has to pass through these systems before it becomes usable work. Engineering improvements can improve user experience sooner; model training takes longer and may help control costs and improve margins over time. Joey says Harvey is studying the long-term margin effects, while short-term training remains experimental.
Joey: We look first at what users can feel right away. Engineering optimization often has a faster effect; model training takes longer. We can improve the product experience first. A proprietary model may help control costs and raise margins over the long term. Early-stage startups still have to compare the return on investment before increasing their spending on model training.
8. Spectre Lets Teams Share an Agent and Requires Clear Permissions
Raymond: You also built an internal tool called Spectre. What problem was it meant to solve?
Joey: Around February, I started thinking about how the company could use cloud-based Background Agents running in sandboxes. They can run in the cloud, so they are not limited by the hardware on an engineer’s own computer, and they can handle more work in parallel.
I came from Figma and understood the importance of collaboration tools. Imagine a manager asks a question and five engineers each run it through their own Claude Code or Codex. They paste the results into a group chat, but no one knows exactly how each result was produced or can easily compare the tool calls and context. Spectre is meant to provide a shared Agent runtime and conversation context so team members can see the process, add their own judgment, and agree on a result.
I wanted to put the Agent in a runtime and context that everyone could participate in. Team members could tag it in Slack and let it join a project as a teammate. They could see what the Agent did, which tools it called, and how it arrived at a result, then work together to move the output into a product that actually runs.
Raymond: So several people can share one Agent in a group. It has the project context and execution history. Claude later introduced a similar collaboration model. You started in February because you wanted the Agent to act as a teammate. But could the Agent know too much? If someone asks it a question in a public channel, could it reveal material another person does not have permission to see?
Joey: The difficult part of shipping the product is permissions. The team has to decide who can tag the Agent, what information it can access, and how the context differs between public and private channels. We work with the security and IT teams to understand which data can be opened up. We also use design documents and code review to manage risk.
An Agent writing code also has to follow the team’s existing rules. If two people have to review code before it goes to production, can the Agent bypass that rule? Could one person use an Agent to change someone else’s code and submit it? We have to think through those cases in advance.
Raymond: The Agent may take part in projects and encounter information that different people have permission to see. The company has to decide whom it can talk to and what it can access before fitting it into the workflow.
Joey: Right. If employees worry that what they tell the Agent will be shown to their boss the next day, they will not tell it their real problems. Users have to trust that their information will stay within permission boundaries. They also have to trust the Agent’s harness and understand how it runs and handles data. Programmers already know Claude Code, Codex, and Cursor and understand how they run and communicate with an Agent. Using a familiar harness helps teams understand the behavior of a cloud Agent. Users need to understand what the product does, what it can access, and where data goes. That helps build trust and gradually gives them confidence to hand it more work.
Raymond: I once heard people say an organization should unify its context as much as possible. Even with a shared system, each person still reads materials according to their permissions. Managing project data, employee privacy, and collaboration is part of the product’s job.
Joey: Right. Spectre aims to let teams work with an Agent together and see its execution while ensuring it follows security requirements. People need to build trust step by step before they will really use it.
9. Legal Mistakes Carry More Serious Consequences
Raymond: Harvey now mainly serves large law firms and corporate legal teams, through an enterprise-sales model, or SLG. Figma’s product-led growth, or PLG, starts with individuals and then spreads through a team. There are many U.S. law firms with only two or three people. Can they use Harvey directly? I used to imagine a lighter “Small Harvey” for small teams, maybe even called Mike.
Joey: We are considering the small-firm market too. Their needs differ from those of large customers, and Harvey still lacks some features small firms may need. Large firms often have dedicated teams, procurement, and security processes for deploying enterprise software. Smaller teams have different staffing and management arrangements, so Harvey needs the right product and services to help them understand the value and set up the workflow.
Responsibility for legal outcomes is especially sensitive. Users of coding agents usually know they have to check the code and are responsible for the result. If a legal Agent becomes self-serve, users may assume Harvey is responsible for the legal outcome. Mistakes in legal work can have more serious consequences than mistakes in code, so customer education and the partnership model need careful design.
Raymond: You would first work with partners that have systems in place and can manage permissions and review results, and that know how to use an Agent.
Joey: Right. We need the right design partners. They have policies, know how to use Harvey, and can manage the results. The team also has to understand the small firms’ needs and willingness to pay, confirm they understand the product’s boundaries and can review the results, then arrange the right sales and support.
Raymond: We have come back to the question we started with. A vertical AI company’s long-term value depends on more than the model’s answers. It has to understand how customers work, which outputs need review, and how to connect files, workflows, permissions, and evaluation. Better models give these products more capability; customers will entrust important work to the whole system only if it proves reliable.
Joey: Right. Sometimes users do not see a perfect Harvey. They see a Harvey that is still improving and whose way of working they can understand. That process matters.
If you're working on this too — or you think we've got it wrong — write to us at [email protected]; if you'd rather not write, just leave your email below.