The Scoop
Anyscale, a billion-dollar startup that builds crucial software for OpenAI and other major players in the industry, has hired a new CEO to supercharge sales, the company exclusively shared with Semafor.
Keerti Melkote, founder of Aruba Networks, which he sold to Hewlett Packard Enterprise for $3 billion in 2015, will take the helm. Anyscale CEO and co-founder Robert Nishihara will focus on product and growing the customer base.
The move is an example of the bifurcation of the current AI boom, with one group of startups spending large sums of investor money to grow user bases and advance capabilities, while other companies reap immediate rewards by building the infrastructure.
No company exemplifies the “picks and shovels” value proposition better than Nvidia, which makes the most powerful hardware used to train and run AI models.
But AI companies also need complex software in that effort, and that’s where Anyscale comes in. Founded in 2019, it developed an open-source project, dubbed Ray, that makes it easier for firms to quickly train AI models using vast amounts of data and on multiple GPUs at once, which can be an immense logistical challenge in itself.
Training runs on the most powerful AI models are now so costly that if all of the GPUs are not synchronized in the most efficient way possible, it can amount to lighting money on fire.
Ray software is now also used by companies like Apple, Netflix, Uber and ByteDance, Nishihara told Semafor. And Anyscale has found a market for its services in small to medium-sized businesses that want to train AI models but don’t have the technical staff to handle the infrastructure.
Adoption of Ray is growing six-fold year over year, he said, measured by the number of downloads of its software and the number of unique “clusters” of GPUs that utilize Ray, which has reached about 120,000 per week.
But Nishihara, a 33-year-old computer scientist, says his heart wasn’t in making sales pitches. At a board meeting roughly six months ago, he suggested focusing on the product and bringing in an experienced CEO to ramp up revenue.
Venture capitalist Ben Horowitz, who sits on Anyscale’s board, was a bit surprised, Nishihara said. Horowitz and his partner, Marc Andreessen, are known for their almost religious belief in founder-led startups. Nishihara said he had to reassure the board that he was still fully committed to the company.
Still, Horowitz said finding the right person would be a challenge.
After selling Aruba Networks to HPE, Melkote stayed on, building it into a formidable competitor to Cisco Systems. In 2021, he resigned so he could spend time with his son before he went to college. He was also thinking about starting an AI company and talking to investors. Melkote was in Japan with his son when he got a call from a16z about Anyscale, whose co-founder includes Ion Stoica, who also co-founded analytics firm Databricks and is its executive chairman.
Aaron Schildkrout, an early Anyscale investor who has been helping to run the company, said his biggest question when interviewing Melkote was “does this guy have the hunger?” “I came out with a strong ‘yes,’” Schildkrout said.
Melkote said he realized it represented a unique opportunity to build a huge business. He also felt chemistry with the team, he said, “especially because I have walked in Robert’s shoes.”
In this article:
Step Back
Nishihara and his co-founders conceived of the idea for Ray while researching AI at the University of California, Berkeley, where they tried to solve that distributed computing puzzle themselves. They open-sourced their creation and it became an immediate success.
Silicon Valley investors like Horowitz saw that Ray was being adopted by large companies like Amazon Web Services and Intel. That, combined with their belief that AI was at an inflection point that would create even more demand for the software, spurred them to lead a $20.6 million Series A investment in 2019, according to a blog post at the time.
Anyscale’s early investors believed the company could leverage its expertise in Ray to build a business supporting companies in need of the software — a classic enterprise software business model.
But Nishihara said there were plenty of doubters who believed only a handful of companies like Google could really take advantage of Ray’s software. “It wasn’t really until generative AI that the need for scaling became just painfully clear,” he said.
Anyscale and its investors, though, turned out to be prescient. Large language models began to show huge improvements, as data and processing power used to train them grew, along with the pool of potential Ray users and Anyscale customers.
ChatGPT maker OpenAI, for instance, initially built its own, customized software to train its AI models, according to OpenAI co-founder Greg Brockman, who was interviewed at an Anyscale event.
Brockman said the software was so complex that it drained resources, which could have otherwise been used on the company’s core competency: AI research. That’s why OpenAI began searching for alternatives.
“We probably looked at half a dozen other projects and Ray was by far the winner,” said Brockman. “We’re using it to train our largest models. It’s been very helpful for us to be able to scale up to a pretty unprecedented scale and not go crazy.”
Reed’s view
It takes a lot of maturity for founders to step down from the CEO role and bring in someone more suited to the task.
It makes sense in this case because there’s a clear product-market fit. In fact, major cloud providers like Amazon Web Services already offer services to customers using Ray implementations.
With an obvious roadmap on the business side, hiring a professional CEO should, in theory, allow the technical staff to focus on improving the product.
Anyscale is also a perfect example of how this early wave of AI is probably going to go. The early winners are likely these picks-and-shovels businesses that make it easier and more efficient to create products for end users.
The killer apps and products will likely come in the next wave, when the barriers to entry are even lower and the technology is even more useful.
Inevitably, there will be an AI letdown, when many companies fail. But the real test of whether this is all just a bubble will be the success of companies like Anyscale. As long as companies like that keep growing, so will the AI industry, even if the hype dies down.
The View From a16z
Martin Casado, who helped lead a16z’s investment in Anyscale, and is one of the most influential thinkers on AI investing these days, probably does not disagree with this move (I asked him but didn’t hear back). But I found an old blog post from 2020 in which Casado opens up about grappling with the CEO decision when he was a founder. It’s an insider’s treatise on the pros and cons of hiring a professional executive to run a tech startup. Here’s one excerpt:
“I’ve also witnessed situations where bringing in an external CEO was disastrous. The CEO is the most visible and most leveraged role in the company, and a bad fit can leave massive wounds. But fit is not just a matter of personality or skills, it’s also a matter of timing in the lifecycle of a company. Bringing in a professional CEO is particularly difficult during the product phase of a company. Because finding product-market fit requires a broad understanding of the technology and the customer; macro trends (many of which influenced the idea maze in the first place); relevant influencers and partners. All of that is best done by the technical founding team.”