Why Google Didn’t Build Muse First

Why Google Didn’t Build Muse First

“Google is too slow.”

— Lewis, on why a new personal assistant can create value from Google-owned data before Google turns it into a product people use every day

Guest: Lewis, Product Lead at Microsoft and a founding product manager on Google’s Gemini team

Raymond gave Muse permission to see his email, and its first suggestion was to follow up on a hospital thread. The thread had already accumulated messages from more than ten people. Miss a day and the conversation moved on; the assistant surfaced the next action anyway.

The data was in Gmail. The calendar was in Google Calendar. Google already had frontier models, consumer distribution, and years of work on assistants. Yet the experience that made Raymond use an assistant differently came from elsewhere. The question is not simply why Google has failed to build an agent. It is why an agent that can read the same inbox suddenly feels useful only when it turns context into a concrete invitation to act.

Lewis has watched this question from inside the major platforms. In 2022, before ChatGPT’s release, he joined the small team turning Google’s PaLM 2 research model into a product. After Gemini 2.5 Pro shipped, he left Google for Microsoft, where he now works on memory and enterprise knowledge for agents. In this conversation, he and Raymond move from the early days of model productization to enterprise permissions, consumer assistants, China’s closed data ecosystems, local models, and the increasingly secretive race at the frontier.

Key takeaways

• A personal assistant becomes useful not when it can technically connect to email, but when it uses context to propose a specific piece of work the user recognizes immediately.
• Memory is not one feature. Personal preferences, shared team knowledge, and procedural lessons from an agent’s past attempts require different storage, access, and consent rules.
• Google’s advantage in data and models does not automatically produce a better assistant; speed of iteration, product focus, organizational structure, and risk tolerance matter as much as raw capability.
• Data portability will shape who can build assistants. Open APIs make a user’s context composable; closed ecosystems create both moats and fragmentation.
• Consumer agents still need a durable business model. A successful refund or a cheap flight can demonstrate value, but recurring value—and eventually cheaper local inference—will determine whether people keep paying.

I. Before ChatGPT, what was a “large-model product” supposed to be?

Raymond: You joined the PaLM 2 effort before ChatGPT had made the category legible to consumers. What did you and the team think you were building at the time?

Lewis: I started as an engineer, then spent years on public cloud at Oracle. Around 2011 and 2012, cloud computing was still early enough that the basic message was simply that computation and storage would move online. I eventually went to Google and moved into product management. I found that I cared more about deciding what to build than only about how to build it, and I worked on several AI products from zero to one before large models arrived.

The PaLM 2 project sat between Labs and the Cloud organization. We were trying to turn a research model into something people could actually use. At the beginning, the model had been pretrained, but it had not been sufficiently post-trained for the varied needs of customers. Our work was to find use cases, tune the model, expose it as an API, establish pricing, and solve concrete customer problems.

We began roughly three months before ChatGPT launched. There were already signals around the Bay Area that this would become important. The team put other work aside and focused on it. In August or September 2022, there were only eight or nine of us. By the end of the year, the immediate goal was modest: get the model running and let the rest of Google try it internally.

There was already a reason to want generalized intelligence in enterprise software. We had worked on Document AI, where the problem was that every person’s and every company’s documents looked different. If you wanted 99.9 percent accuracy, you often had to build and train a specific model for every document type. That was expensive and slow. We were asking whether one model could generalize across documents rather than treating each format as a separate problem.

ChatGPT changed the public understanding of the category. It put the model directly in front of consumers, who could ask strange questions, get an immediate response, enjoy the interaction, and spread it. Google’s path had been more industrial and enterprise-oriented. ChatGPT made the consumer experience itself the proof that the technology mattered.

Raymond: Internally, did that feel like a surprise? Google had its own conversational systems, and Bard even gave this show its name in 2023. I tested several tools, and the name it suggested was the one I liked—so I do owe Google that. Did people think the company had simply failed to release what it already had, or that OpenAI had genuinely pulled ahead?

Lewis: Both views existed. Some people pointed to Meena, Google’s earlier chatbot, and argued that responsible-AI and legal requirements had slowed releases. Others thought OpenAI’s models were genuinely better at conversation in many settings than what we had exposed internally.

The more important point was that technical capability and productization are different jobs. A research organization can have an impressive model and still struggle to turn it into a reliable API, price it, make it accessible, and bring it to market. At that time, DeepMind, Google Research, and Google Brain were still spread across different organizations; even the first generation of product work was not simply a model trained by DeepMind. The urgency of competition pushed the company to concentrate resources.

The mood was not defeatist. Google had deep research capacity and plenty of technical assets. But the pressure was real. Sergey Brin did not appear one morning and suddenly take over everything; his involvement visibly increased. He would show up in group chats and ask specific questions about technical results and evaluations. That kind of attention changes the temperature of a project.

Raymond: If Sergey Brin is asking you questions directly in a chat, that is probably as much motivation as anyone needs.

II. If intelligence is water, what happens when the faucet is finished?

Raymond: You left after Gemini 2.5 Pro, when Google was having a strong moment. You once framed the change this way: if intelligence is water, product managers have spent years designing the faucet—the shape, the flow, the interface. Once the faucet is largely understood, the competition shifts back to the water itself: model quality, cost, and benchmarks. Was that why you left?

Lewis: That was a big part of it. For a product manager, many of the routes around model productization had become flatter: how to package a model, how to price it, which use cases to target, how to build a developer community. The frontier work was moving toward model iteration, new architectures, and benchmark performance. I wanted to work on problems whose product form was not yet settled, which is why I moved to Microsoft to work on agents and knowledge.

Organization mattered too. Once a project becomes a company-wide priority, many more people join. Layers appear, and information can begin to travel through intermediaries. In my experience, some middle layers can keep information because holding it makes their role feel necessary. The work becomes less direct.

Microsoft was attractive because enterprise AI still has a very large unsolved surface area. Many companies have not truly started using models or agents, even though they sit on enormous amounts of data. SharePoint contains presentation decks, documents, and internal materials at extraordinary scale. But having the data is not the same as making it usable. Many enterprises do not have a search experience remotely as effective as Google’s. The data is there; the people who need it often cannot find it. Making models and agents useful against that corpus is difficult and valuable work.

Raymond: You now work on agent memory and knowledge. A consumer may hear “memory” and think: the assistant remembers my preferences or knows I am still waiting for a refund. But you also describe something called procedural memory—the agent remembering how to do work. Are those the same thing? And how is procedural memory different from a skill?

Lewis: I work mainly on developer-facing capabilities—the infrastructure other people can turn into applications. Personalization is the easiest example. When you return to customer support, the agent knows who you are, what you discussed last time, and that a refund issue remains open. It can ask whether you want to continue.

The other part is how an agent improves from experience. Today an agent may try several approaches to a task, fail, and try again. That can consume a lot of tokens. Procedural memory captures the useful part of those runs: what failed, what worked, and which method solved the problem. The next time a similar task appears, the agent does not need to begin from zero.

In practice, the distinction from a skill is not large. A skill is one way to package reusable experience. Procedural memory is the broader question: what experience should be condensed, how should it be retrieved, and under what circumstances should it be applied? You can learn a procedural lesson from a failed run and store it as a skill.

This is not fundamentally tied to whether the eventual application runs on Mac, Windows, or the web. The platform can provide models, an agent runtime, and a way to store and retrieve memory. Developers can build on top of that for any operating system. The harder question is how agents organize and use the context behind the application.

Raymond: But an enterprise cannot treat all memory as one shared folder. A conversation about a board appointment and a conversation about a routine operational problem cannot have the same access rules. And an employee who leaves may leave behind useful documents but also millions of words of chat context that no manager can realistically read. I had seen a Shopify example—at least as I understood it—in which employees spoke with the same Claude agent and the resulting knowledge could be seen by others. That is an aggressive approach. If workers have to mark every sentence shareable or private, it is too slow; if they know every message is visible, they may simply move sensitive conversation to WeChat or WhatsApp. How should that work?

Lewis: This is central. Many memory startups begin with individual personalization: one user’s data, another user’s data, broadly similar structures. An enterprise has many levels. A CEO has a private memory. A developer investigating a system bug cannot simply open that memory because it might be useful. Permissions have to be enforced.

There can be a memory for one employee, one for two colleagues working together, one for a team, and one for the company. The question is not only who can read each layer, but which knowledge is allowed to move upward. Two people might learn something in a work conversation that ought to become a team asset. But not every part of their discussion belongs there.

The product cannot solve this by silently watching every private exchange. If people think a conversation is private but discover it has always been visible to others, they change how they talk. They stop discussing personal matters; they stop making casual exploratory remarks. It is like working while someone stands behind you all day.

A better design may have the model extract candidate information and then ask the people involved whether it should be shared with a team or the broader company. That does not require someone to reread every line of chat, but it preserves human control over promotion. There are hard gray areas: if two colleagues complain about their manager, should that be part of team memory? The people who own the conversation should have a say.

Raymond: There is also a second boundary. A company can tell employees that certain internal interactions are recorded, and employees can decide whether they accept that policy. But when a company calls an external model, users may worry that a model provider sees the data. A contractual statement is hard for an outsider to audit.

Lewis: Those are distinct layers. Internal employee policy is one question. Model deployment is another. You have to ask where a model actually runs and who processes the data. In enterprise settings, models associated with OpenAI or Anthropic may run through Azure; Google has corresponding cloud deployment choices. The model’s brand name alone does not tell you enough about the data path. That distinction matters more than treating every use of a named model as identical.

III. A cancellation button can be more useful than an empty chat box

Raymond: People used to say that only enterprises would pay for agents. Then personal assistants reappeared: booking flights, buying groceries, following up with insurers, handling refunds. You use Muse alongside Hermes. Why did those survive while other tools did not?

Lewis: I was a heavy OpenClaw user early in the year. When version 2.0 arrived, I ran into migration problems and things that had worked stopped working. Hermes performed better in several situations, so I stopped trying to repair OpenClaw and moved over. Both could run on the same virtual machine; I could tell Hermes to retrieve the data from OpenClaw and it handled the migration.

I also tried Grok’s agent product. In my experience, it was aimed at professional users but withheld something those users need: direct access to the work product. With Hermes or OpenClaw, I can access the code, website, and files on my virtual machine. With Grok, the machine behind the task was not directly accessible. It could show me a site through its own browser or let me download something, but I could not work naturally with the environment. For someone building things, that was not enough.

Muse is different in a more product-oriented way. Hermes requires maintenance. Muse is an application with a clear presentation of what the agent can do. Its idea tab offers task suggestions, many derived from the user’s own context.

An empty chat box puts the burden on the user to imagine a use case. A list of concrete, timely suggestions does not. The underlying capability may be similar—connecting Gmail or a calendar—but the experience is not. A user often has to learn that a capability exists, find an example online, copy a prompt, and then return to the product to try it. Muse can put “review subscriptions you are paying for but not using” in front of the user as a direct action. That is a much lower-friction path.

Raymond: That describes my experience. After I gave Muse permission, it surfaced a hospital email thread. In the United States, many life tasks begin in email: making a doctor’s appointment, enrolling a child in a class, dealing with a bill. A long thread can move while you look away for a day. I had already given other tools access to Gmail, including Claude Code, but I had never used them this way.

Lewis: Email and calendars are rich sources of personal context. They reveal what you are doing and what may need attention. But integration alone does not create value. The product has to teach the user, often without making the user study a manual, what work is now possible. That is why there is still room for application companies even if the underlying models and connectors become broadly available.

IV. If Google owns the inbox, why didn’t Google make the assistant—and who else can?

Raymond: Privacy makes this even stranger. People may distrust Mark Zuckerberg with their email, then hand their inbox to a young assistant company such as Instinct. How do they decide whom to trust?

Lewis: I did not use Instinct because I was not comfortable with the terms I saw. My reading was that the rights around data collected through the product were broad enough that email and other personal data might be used to improve the product. I did not know the people behind it well enough to bridge that trust gap. That was enough for me not to proceed.

Raymond: So return to the original puzzle. Gmail and Google Calendar are Google’s data. Google has models and consumer products. Why not Google?

Lewis: Google is too slow.

Part of the reason is that large companies can become reactive. When we worked on Gemini, OpenAI was the primary competitor. OpenAI was succeeding with chat and consumer products, so much of the organization’s energy went into making polished demonstrations and consumer-facing experiences that could compete directly.

Anthropic made a different product bet. Many people believed core engineering work would be difficult to replace, and Anthropic leaned into engineering and enterprise. I think that choice was right. OpenAI later redirected significant energy from pure consumer experiences toward coding, and Google has moved in that direction too.

Google had talked about an ambient assistant long before the current wave of personal agents. Bard and Gemini were framed around that ambition, and Google I/O had discussed it years ago. But an ambition and a finished product are not the same thing. In Lewis’s view, the pace of visible iteration has been slower than competitors’. xAI moved early, but its product did not solve his own needs even after he paid for it. Meta, by contrast, appeared to spend a long time refining its offering before producing something he found effective. Being first and being good are different achievements. Lewis had heard rumors of a comparable internal Google product, but treated them as rumors rather than evidence that a ready product was imminent.

Raymond: There is also an organizational explanation. A large company’s legal and compliance teams have to ask what happens if an assistant reads sensitive information or gives bad advice. A startup can be more aggressive. Combine that with layers of management, and everyone may understand the opportunity while the product still takes too long to arrive.

Raymond: Google could make life difficult for assistants by closing Gmail and Calendar access. Why would it leave the door open to competitors?

Lewis: I do not think that is its natural instinct. Google has a relatively open culture around APIs; products such as Superhuman have been built on top of email access. More fundamentally, the product philosophy is that users should be able to access their own data. Based on what I have seen, Google has not tried to make “you cannot call your own data” its primary moat.

The contrast becomes clear in China. When I use Muse, Hermes, or OpenClaw, I would like them to work with my WeChat context. WeChat—Tencent’s messaging, payments, and mini-program super-app—contains an enormous part of many users’ lives, but that context is difficult for an outside agent to access. The data sits inside the platform’s ecosystem. That gives the platform a powerful moat, and it means an assistant that can use that data may be easier for the platform itself to build than for an outsider.

Raymond: China’s mobile internet may have created a different kind of problem for the AI era. It built high walls around very useful gardens. I can search RedNote—Xiaohongshu, the Chinese lifestyle and discovery platform—inside its own tool, and it can surface and categorize posts. But an external agent may not be able to search it. If the ten or fifteen largest platforms each have a knowledge agent that cannot talk to the others, the user becomes the integration layer: I have to ask them one by one.

Lewis: My own experience is somewhat mixed. On a virtual machine, I have been able to log into the RedNote web version and have an agent inspect some account-level information. You may encounter forced logouts where I did not, so detection and actual access can differ. But WeChat is genuinely harder to treat like an ordinary website.

The two internet environments also differ. In the United States, information may arrive in email, but the next step happens on a hospital site or a utility’s website. Services are fragmented. WeChat combines a great deal of activity inside one environment, often through mini-programs. That makes the user journey convenient, but it makes it hard for an external agent to assemble all the relevant context in one place.

Raymond: That is why Tencent may be able to wait. Its model does not have to lead immediately if the WeChat relationship graph is hard to replace. What about other Chinese platforms?

Lewis: RedNote has an opening because a personal assistant needs context and preferences. What users say, click, search for, and like on RedNote contains part of that signal. It is also an unusually useful network for real-world discovery: finding a restaurant, a hiking partner, or even a career coach in the Bay Area.

But no one platform has the whole life. RedNote skews toward leisure. WeChat contains work and family. WeCom, Tencent’s enterprise collaboration product, and personal WeChat may be separate again. Meituan, China’s local-services marketplace, and Alibaba, China’s commerce and cloud group, may each have opportunities tied to their own capabilities. A startup may be better served by building a strong assistant for one domain than by trying to own every part of a person’s life from the first day.

V. One successful refund is not a business model

Raymond: How should personal agents charge? There have been suggestions that assistants such as Muse could take a cut of shopping, alongside reports that Amazon was restricting agent access. There are tensions with platforms that want humans, not agents, to arrive and transact. Is the model subscription, payment by outcome, or something else?

Lewis: Start by separating the fee for entry from the value of the problem solved. A high monthly price is hard to sustain when a user does not yet understand what the product can do. When Grok’s agent product was asking for roughly RMB 200—about US$28 at the time—as an entry price, I found that hard to sustain. Muse’s approach made more sense to me: let people use it for free within a weekly allowance, then charge when use expands. First the user has to discover the value.

Many of these products borrow from the OpenClaw style of agent. The technical barrier is not infinitely high. If one company can build a narrow task, another can often reproduce it. So a price cannot simply be a prepayment for tokens. It has to reflect a problem worth solving.

For consumers, I lean toward freemium: let them try the service, then charge for heavier use. An agent may help recover a refund, obtain a price adjustment, or find a cheaper product. But each of those can be a one-off event. A person may be delighted when the refund works, but where does the next refund come from? A cheap box of eggs is useful once. It does not automatically establish the lifetime value needed for a durable subscription.

Enterprise economics are different because work repeats. You can estimate return on investment using something as simple as an employee’s hourly cost: how many hours did the agent save, and how often will the process recur? That does not capture every benefit, but it produces a legible equation. Consumer life tasks have weaker repetition and are therefore harder to price with the same logic.

Over the long run, Lewis expects the price of a unit of intelligence to fall. Inference may become more like a utility. If advertising eventually supports some of the cost, or if inference costs continue to fall, the consumer price can fall with it. Some work will shift to local hardware: an iPhone or Mac can handle ordinary personal tasks, while the cloud handles the unusually difficult request. Most people do not need frontier reasoning every day to manage invoices, subscriptions, or appointments.

That split could create an opening for Apple. Pricing a personal assistant too high may prevent the user from crossing the threshold at all. If local capability becomes private, cheap, and integrated, the practical product question is less about whether a model exists and more about whether the user receives an assistant that is simply there when needed.

Raymond: Local hardware seems ready before the product is. People can buy a Mac Studio or an NVIDIA DGX Spark and run models locally, but most ordinary users cannot turn that capability into an experience.

Lewis: That is the central gap. Hardware can do more than people expect. A high-end Mac Studio can run very large models, although the configuration Lewis had in mind was around US$12,000 and therefore out of reach for many users; smaller local models can already handle ordinary conversation and common tasks. But a dedicated hardware product faces existing computers. If the next Mac or PC chip already supports the relevant models, a separate device has to explain why it exists.

The harder problem is software. Does a user buy a machine and find a model already running, or do they have to install tools, configure APIs, and piece together a stack? During the early OpenClaw boom, people bought Mac minis and then paid roughly RMB 500—about US$70—for someone to install OpenClaw at home. The machine could run the software; that did not mean the product was ready for a nontechnical user. The installation gap is a product opportunity, not a minor detail.

Lewis also sees the hardware roadmap as easier to reason about than the model roadmap. A chip engineer may have a clearer view of what will be possible in six months or several years than a model researcher has of the next architecture. His view is that models now confined to the cloud may run on phones within six or seven years. That is a prediction, not a promise, but the direction resembles the old transition from phones struggling with a GIF to phones playing high-resolution video. Model compression and hardware improvement can unlock uses that are currently blocked by cost.

VI. The frontier is moving faster than the release calendar

Raymond: You still have confidence in Google, even though the field now moves so quickly that a release may be expected to lead for a week, not a quarter. Can Google catch up?

Lewis: I think it can. I still know people there, and the technical foundation remains strong. But a foundation is not the same as a product users can access today. At the time of our conversation, the next Pro release had taken longer than people expected. A model cannot merely reach the target that existed months earlier; it has to meet the best available models on the day it ships. Competitors continue to move the threshold while a team is training.

Raymond: That pressure changes how companies share. When DeepSeek, the Chinese AI lab, released a strong open model, what was the reaction inside Google? Was there surprise, and would a company such as Google take a Chinese lab’s research seriously?

Lewis: The first response from people working on models is to read the paper. Does it contain something distinctive? In the case of DeepSeek, people thought there was genuine substance and real innovation. A model of that quality being open was surprising, and some techniques had not appeared in the same form in the United States. The natural response was: this is a good idea; we should consider whether it can work for us too.

That is different from using another company’s outputs to distill a model. I can only speak to my own experience at Google, not to every company. In the work I participated in, the expectation was that user data would not be recorded or used for training, and people followed that expectation. I cannot make claims about how every competitor behaves.

The incentive to publish has also changed. Leading companies rarely release technical papers with the same frequency or specificity as before. A technique may protect an advantage for three or six months; publish the paper and it may protect the advantage for a week. People also become cautious in ordinary conversation, because a small disclosure can matter in a race this tight. Raymond has felt that shift while inviting people from model companies onto the podcast: people are increasingly reluctant to talk in public when a careless detail may alter a competitive position.

Raymond: That creates a cruel window for researchers. The bottleneck may be pretraining, post-training, or data at different moments. A particular kind of researcher may have an eighteen-month period in which they are unusually valuable. They have to protect the advantage, and then the field catches up.

Lewis: Yes. But exchange still matters. People see the problem from different angles. The difficult part is finding a way to keep that exchange alive while every organization feels how quickly a small technical lead can disappear.

This interview has been translated from Chinese, edited and condensed for clarity.

If you're working on this too — or you think we've got it wrong — write to us at [email protected]; if you'd rather not write, just leave your email below.