Silicon Valley Races to Build Virtual Workspaces for Smarter AI Agents
For years, leaders in Big Tech have envisioned AI agents capable of autonomously navigating software applications and completing tasks on behalf of humans. Yet, testing today’s consumer AI agents—such as OpenAI’s ChatGPT Agent or Perplexity’s Comet—quickly reveals the technology’s limitations. Experts believe making these agents more capable may require entirely new training methods, which the industry is only beginning to explore.
The Rise of Reinforcement Learning Environments
One emerging technique gaining traction is the use of simulated workspaces, called reinforcement learning (RL) environments. These environments allow AI agents to practice multi-step tasks in a controlled, evaluative setting. Much like labeled datasets drove the previous AI boom, RL environments are now being recognized as a critical component for advancing AI agents.
“All the big AI labs are building RL environments in-house,” said Jennifer Li, general partner at Andreessen Horowitz, in an interview with TechCrunch. “But as you can imagine, creating these datasets is very complex, so AI labs are also looking at third party vendors that can create high quality environments and evaluations. Everyone is looking at this space.”
This demand has created a surge of well-funded startups, including Mechanize and Prime Intellect, aiming to dominate the RL environment market. Meanwhile, established data-labeling firms like Mercor and Surge are investing heavily in RL environments to keep pace with the shift from static datasets to interactive simulations. According to The Information, Anthropic’s leadership has considered allocating more than $1 billion over the next year to RL environment development.
Investors and founders hope that one of these companies might emerge as the “Scale AI for environments,” referencing the $29 billion data-labeling company that helped drive the chatbot revolution.
How RL Environments Work
Reinforcement learning environments simulate real-world software tasks for AI agents. One founder described building them as “like creating a very boring video game.” For example, an RL environment might simulate a Chrome browser and challenge an agent to purchase a pair of socks on Amazon. Success earns the AI a reward signal, reinforcing correct behavior.
Though this may sound simple, AI agents can encounter numerous pitfalls, from misusing dropdown menus to over-purchasing items. Environments must be robust enough to capture unexpected behavior and provide useful feedback, making them far more complex than static datasets.
Some environments are sophisticated enough to let AI agents use tools, access the internet, or operate multiple software applications. Others are more narrowly designed for specific enterprise tasks. The concept is not entirely new: OpenAI developed “RL Gyms” in 2016, and Google DeepMind’s AlphaGo used RL within a simulated environment to beat a Go world champion that same year.
What distinguishes today’s RL environments is their application to AI agents powered by large transformer models, aiming for general capabilities across diverse tasks rather than specialized applications like AlphaGo.
A Crowded and Competitive Market
Scale AI, Surge, and Mercor are racing to meet the growing demand for RL environments. Surge CEO Edwin Chen noted a “significant increase” in interest from AI labs. Surge, which reportedly earned $1.2 billion last year from clients such as OpenAI, Google, Anthropic, and Meta, has launched an internal division dedicated to RL environments.
Mercor, valued at $10 billion, is targeting domain-specific environments for coding, healthcare, and law. “Few understand how large the opportunity around RL environments truly is,” said Mercor CEO Brendan Foody.
Even Scale AI, which has lost some dominance after Meta invested $14 billion and recruited its CEO, is pivoting toward RL environments. “Scale has proven its ability to adapt quickly. We did this in the early days of autonomous vehicles, our first business unit. When ChatGPT came out, Scale AI adapted to that. And now, once again, we’re adapting to new frontier spaces like agents and environments,” said Chetan Rane, Scale AI’s head of product for agents and RL environments.
Startups like Mechanize and Prime Intellect are also carving their niche. Mechanize, founded six months ago, aims to supply AI labs with a few high-quality RL environments rather than a wide range of simpler ones. The company is reportedly offering software engineers $500,000 salaries to build these environments. Mechanize has reportedly partnered with Anthropic on RL projects, though both parties declined to comment.
Prime Intellect, backed by Andrej Karpathy, Founders Fund, and Menlo Ventures, has launched an RL environment hub designed to give open-source developers access to tools previously available only to large labs. “RL environments are going to be too large for any one company to dominate,” said Prime Intellect researcher Will Brown. “Part of what we’re doing is just trying to build good open-source infrastructure around it. The service we sell is compute, so it is a convenient onramp to using GPUs, but we’re thinking of this more in the long term.”
The Challenge of Scaling RL Environments
The big question is whether RL environments can scale as effectively as past AI training methods. Reinforcement learning has driven major breakthroughs, including OpenAI’s o1 and Anthropic’s Claude Opus 4, at a time when traditional methods are yielding diminishing returns.
RL environments allow agents to interact with tools and software in a way that previous text-based reward systems could not. However, the approach is resource-intensive and far from foolproof. Ross Taylor, former AI research lead at Meta and co-founder of General Reasoning, warned that RL environments are prone to “reward hacking,” where AI agents cheat to gain a reward without completing the intended task.
Even within the industry, opinions remain mixed. Sherwin Wu, OpenAI’s Head of Engineering for its API business, said he was “short” on startups capable of delivering quality RL environments. Karpathy, an investor in Prime Intellect, expressed caution on X: “I am bullish on environments and agentic interactions but I am bearish on reinforcement learning specifically.”
Looking Ahead
Despite uncertainties, RL environments represent a promising frontier for AI development. By offering interactive, multi-step simulations, they provide AI agents with a richer, more complex training ground than ever before. The race is on in Silicon Valley to see which companies will lead the charge—and whether RL environments will become as foundational to AI as labeled datasets once were.
0 Comments