Facebook has been working on augmented reality glasses. Images
Facebook envisions a future in which you learn to play drums or cook up a new recipe while wearing augmented reality glasses or other devices powered by artificial intelligence. To make this future a reality, the social network needs its AI systems to see through your eyes.
"This is the world where we would have portable devices that could be of use to you and I in our daily lives by providing information at the right time or helping us to retrieve memories "said Kristen Grauman, a senior researcher at Facebook. The technology could potentially be used to analyze our activities, she said, to help us find misplaced items, like our keys.
Receive the CNET Daily N newsletterews Learn about the biggest news in minutes. Delivered during the week.
That future is still a long way off, as evidenced by the Ray-Ban branded "smart glasses , which debuted in September with no AR effects. Part of the challenge is training the systems to 'AI to better understand the photos and videos that people capture from their perspective so that AI can help people remember important information.
Facebook says analyzing videos shot in first person is a challenge for computers. Facebook
Facebook announced it has partnered with 13 universities and labs that have recruited 750 people to capture more 2,200 hours of first-person video over two years The participants, who have lived in UK, Italy, India, Japan, Saudi Arabia, Singapore, United States,Rwanda and Colombia have made videos of themselves engaging in daily activities such as playing sports, shopping, watching their pets or gardening. They used a variety of wearable devices, including GoPro cameras, Vuzix Blade smart glasses, and ZShades video recording sunglasses.
Starting next month, researchers from Facebook will be able to request access to this wealth of data, which the social network has said is the world's largest collection of uned first-person videos. The new project, called Ego4D, provides insight into how a tech company could improve technologies like AR, virtual reality, and robotics to play a bigger role in our daily lives.
The company's work comes during a tumultuous time for Facebook. The social network has fcome under close scrutiny from lawmakers, advocacy groups and the public after the The "Wall Street Journal published a series of articles on how the company's internal research showed it was aware of the platform's misdeeds even if he publicly minimized them. Frances "Haugen , former Facebook product manager turned whistleblower, testified before Congress last week on the contents of thousands of pages of documents confidential which she recovered before leaving the company in May. She must testify in the UK and meet the the " semi-independent supervisory board of Facebook in the near future future.
Even before Haugen's revelations, Facebook's smart glasses raised concerns among critics who fear the device could be used to secretly register people. During its research into first-person video, the social network said it was responding to privacy concerns. Camera wearers could view and delete their videos, and the company blurred the faces of passers-by and the license plates that were captured.
Continue research in AI
Laundry and cooking are different in the video from different countries. Facebook
As part of the new project, Facebook said it has created five benchmark challenges for researchers. Benchmarks include episodic memory , so you know what ist happened when; forecasting, so that computers know what you are likely to do next; and the manipulation of hands and objects, to understand what a person is doing in a video. The last two benchmarks are understanding who said what, and when, in a video, and who are the partners in the interaction.
"This sets up a bar just to start " Grauman said. "This is usually quite powerful, as you now have a systematic way to evaluate the data.
Helping AI understand first-person video can be difficult , because computers typically learn from images taken from a viewer's third-person perspective. Challenges like motion blur and pictures from different angles come into play when you record yourself kicking a soccer ball or riding a roller coaster.
Facebook has statedre that he was considering expanding the project to other countries. The company said that the persification of the video footage is important because if the ar glasses help a person to cook curry or do the laundry, the assistant ia should understand that these activities may be different in different parts of the world.
Facebook said the video dataset includes a wide range of activity shot at 73 locations in nine countries. Participants included people of different ages, genders and professions.
The COVID-19 pandemic has also created limits for research. For example, more sequences in the dataset are for home activities such as cooking or crafts rather than public events.
Some of Facebook's partner universities include University of Bristol in UK, Georgia Tech in US, University of Tokyo in Japan and Universidad de los Andes inColombia.