Pupil movements, body postures, and crushing noses are among the flickers of human expression that Meta wants to reap in building its metaverse, according to an analysis of dozens of patents recently issued to Facebook's parent company.
CEO Mark Zuckerberg has promised to spend $ 10 billion a year over the next decade on the hazy and highly hyped concept that signifies an immersive virtual world filled with avatars. Rivals like Apple and Microsoft are also pursuing similar goals, which Big Tech executives describe as part of the next development of the Internet.
The Financial Times has reviewed hundreds of applications to the U.S. Patent and Trademark Office, many of which were filed this month. They reveal that Meta has patented several technologies that use users' biometric data to help drive what the user sees and ensure that their digital avatars are animated realistically.
But the patents also indicate how the Silicon Valley Group intends to monetize its virtual world with hyper-targeted advertising and sponsored content that reflects its $ 85 billion-a-year existing ad-based business model.
This includes suggestions for a "virtual store" where users can buy digital goods or goods similar to real-world goods that have been sponsored by brands.
"For us, the business model of the meta-verse is trade-driven," Nick Clegg, Meta's head of global affairs, told FT during a recent interview. "Obviously, ads play a role in that."
The patents do not mean that Meta will definitely build the technology, but they do provide the clearest indication so far of how the company aims to make its immersive world a reality.
Some of the patents relate to eye and face tracking technology, typically assembled in a headset via small cameras or sensors, which can be used to enhance a user's virtual or augmented reality experience. For example, a person will be shown brighter graphics where their gaze falls, or ensure that their avatar reflects what they are doing in real life.
A Meta patent, issued on January 4, outlines a system for tracking a user's facial expression through a headset that will then "customize media content" based on those responses.
There is a "portable magnetic sensor system" to be placed around a torso for "body pose tracking". The patent includes sketches of a user carrying the device but appearing in virtual reality as a soldier complete with a sword and armor.
Another patent proposes an "avatar personalization engine" that can create three-dimensional avatars based on a user's photos, using tools, including a so-called skin replicator.
"Meta aims to be able to simulate you down to every skin pore, every hair, every micro-movement," said Noelle Martin, a legal reformer who has spent more than a year researching Meta's ambitions for human surveillance with the University of Western Australia.
"The goal is to create 3D replicas of people, places and things so hyperrealistic and tactile that they are indistinguishable from what is real, and then to convey any range of services... In truth they are engaged in a global human cloning program. "
The project has enabled the company, which in recent times has been hit by other scandals over moderation and privacy, to attract engineers from rivals such as Microsoft in the midst of a fierce battle for talent between the world's largest technology companies.
Since changing its name from Facebook to Meta in late October in a corporate rebranding, the company's stock price has risen about 5 percent to $ 329.21.
Critics remain skeptical of the vision, suggesting the effort is a distraction from the latest investigation, after whistleblower Frances Haugen last year publicly accused the company of prioritizing hate over profit.
"What do they want with more data, and how will they make sure it's secure?" said Celia Hodent, former director of user experience at Epic Games, who now works as an independent consultant.
Some patents seem to focus on helping Meta with its ambitions to find new sources of revenue due to concerns about declining interest among younger users for their core products for social networks like Facebook.
Zuckerberg has indicated that the company plans to keep the prices of its headsets down, but instead pull revenue in its metavers from advertising and by supporting the sale of digital goods and services in its virtual world.
One patent explores how to present users with personalized augmented reality advertising, based on age, gender, interest, and "how users interact with a social media platform," including their likes and comments.
Another seeks to allow third parties to "sponsor the appearance of an item" in a virtual store that reflects the layout of a retail store, through a bidding process similar to the company's existing advertising auction process.
The patents indicate how Meta could offer ads in its immersive world that are even more personal than what is possible within its existing web-based products.
Research shows that momentary direction and student activity may implicitly contain information about a user's interests and emotional state, for example, if a user's eyes linger over an image, it may indicate that they like it.
"Obviously you could do something similar [to existing ad targeting systems] in metaversen - where you do not sell eye-tracking data to advertisers, but to understand whether people engage in an ad or not, you need to be able to use data to know that, ”said Clegg.
Brittan Heller, a technology lawyer at Foley Hoag, said: "My nightmare scenario is that targeted advertising based on our involuntary biological responses to stimuli will start to appear in the meta-verse... Most people are not aware of how valuable it could be "Right now, there are no legal restrictions on that."
Meta said: "While we do not comment on specific coverage of our patents or our reasons for filing them, it is important to note that our patents do not necessarily cover the technology used in our products and services."
Further reporting by Henry Mance in London