Technology

➚ MOVE FAST: High clouds. Google and Microsoft posted quarterly earnings that beat analyst expectations on solid demand for their cloud services, boosted by the AI boom. That helped offset an increase in expenses.

➘ BREAK THINGS: Headwinds. Meta’s stock fell 12% yesterday after it flagged that spending would continue to rise as part of the AI race. Unlike some of its rivals, it doesn’t have a cloud business and also offers its Llama models mostly for free, for now.

Technology is always bringing rapper Tupac Shakur back to life, whether it’s holograms or, more recently AI. Rapper Drake used a deepfake of Tupac’s voice in a song that insults rival Kendrick Lamar. (In case you haven’t followed their drama, Drake and Lamar are in a feud. Lamar is a Tupac fan and is close with his family.)

If the case does go to court, it could lead to a very interesting legal question. Can AI be used to, in essence, impersonate someone for artistic purposes? For instance, any person can listen to a bunch of Tupac’s songs and then try to mimic him. Is it any different if a computer does it?

Bigger isn’t always better in AI

THE NEWS

Competition is heating up between big technology companies building smaller AI models, which can be used in a broader array of devices like smartphones, cameras, and sensors, potentially allowing them to tap more users.

This week, Microsoft and Apple launched Phi-3 and OpenELM, respectively, new large language models that use less compute than the likes of OpenAI’s GPT-4. The moves come as the AI industry realizes that the size of models should be tailored to different applications, and keeps finding ways to make smaller, cheaper LLMs more capable.

“The approach that we’re taking in the Phi series is different,” Sébastién Bubeck, Microsoft’s vice president of generative AI research, told Semafor. “The rest of the industry seems to be mostly about scaling up, trying to add more data and keep making the model bigger.” Bubeck, however, wants to squeeze as much performance out of small models as possible.

For Microsoft, investing in smaller models means it can give customers more options beyond the larger systems it offers from its partnership with OpenAI. Those that can’t afford to use top tier models can use smaller alternatives like Phi-3.

For Apple, OpenELM is relatively slow and limited, which still leaves it behind in the AI race. But it can run on iPhones, an ecosystem the company is keen to develop.

The trick to building small but mighty models lies in the quality of the text used to train them. Researchers at Apple filtered text from publicly available datasets, keeping sentences that are made up of a wider variety of words and more complex.

Microsoft used a mixture of real data scraped from the web and synthetic data generated by AI to train Phi-3. Prompting models to produce data means developers can better control the text used for training. “The reason why Phi-3 is so good for its given size is because we have crafted the data much more carefully,” Bubeck said.

It’s unclear how much data is needed to make a model as powerful as possible, and what capabilities might arise as they improve. These small AI models show that there are a lot of performance gains to be made by training on higher quality data than just scraping from the internet.

“This kind of iterative process of finding the right complexity of data for model size is a journey that the community hasn’t really embarked on yet,” Bubeck said. “And this is why we released these models. It’s to empower all of the developers to use them to see how far you can go once you get into this data optimal regime.”

KATYANNA’S VIEW

AI has yet to make a huge impact on smartphones, and tech companies are quickly moving to explore the possibilities. There’s no better way to see what new AI products and apps can be built than opening it up for other developers to tinker with.

Even Apple, which is notoriously secretive about its technology, has released the source code and training instructions for its OpenELM system. In a paper, Apple researchers explained that the reproducibility and transparency of LLMs was vital to advance AI, and investigate its potential biases and risks. It might be a while yet before the technology arrives, however, given the memory and power constraints on mobile hardware.

Elon Musk is no stranger to legal proceedings, with his latest dalliance set for Monday in what could be a full day of questioning. This time, he is being deposed in a case brought by thousands of former employees of Twitter, who claim they deserve severance pay after being fired in the wake of Musk’s purchase of the platform, which he later renamed X.

His ownership and regular public missives on the social network have already resulted in legal troubles for him, including a defamation lawsuit brought by a California man accused of being an undercover agent in an neo-Nazi group. In that deposition, Musk said “I may have done more to financially impair the company [X] than to help it, but I certainly…do not guide my posts by what is financially beneficial.”

On the other side in his legal fight with OpenAI, which he helped found and then parted ways, Musk requested documents from Helen Toner. She left the ChatGPT maker’s board last November after helping to oust Sam Altman as CEO, only to see him quickly return after an employee revolt.

The American Association for the Advancement of Science published new data Tuesday on global R&D spending and by one measure, Silicon Valley should be slightly worried. The percentage of GDP the US government spends on R&D has dropped to pre-pandemic levels, putting it in 12th place globally.

While the US is in fourth place in overall R&D spending as a percentage of GDP, it’s the government portion that really fuels future economic growth because it funds new breakthroughs that aren’t commercially viable yet. Everything from the internet to lithium-ion batteries to Apple’s M3 microprocessors were built on major developments backed by Uncle Sam. In a close technology cold war with China, the amounts governments spend could also be an indicator of the outcome.

Government R&D spending has nearly halved since 1976, when it was 1.2%. China still trails the US by this measure, but Beijing’s spending has been increasing, from 0.3% in 2000 to 0.46% in 2021, even as its economy has grown much faster over that time period.

China’s boost in R&D spending seems to be manifesting in the form of new research. According to the AAAS data, the number of human researchers in China shot past the US around 2011 and the gap continues to widen. It is also producing more academic papers.

What the US has lost in reduced R&D funding, it has made up for with a capitalist system that rewards innovation like no other country in the world. But depriving that system of government-funded R&D research could miss the moonshot breakthroughs that created Silicon Valley.

• Mike Johnson saved Ukraine aid. Will donors care?
• In battle against Russia, US focuses on Moldova.
• Poor countries have a climate bomb on their balance sheets.