Facebook's chief executive, Mark Zuckerberg, has long rued the day that Apple and Google beat him to building smartphones, which now underpin many people's digital lives. Ever since, he has searched for the next frontier of modern computing and how to be a part of it from the start.

Now, Zuckerberg is betting he has found it: the real world.

On Tuesday, Zuckerberg introduced what he positioned as the first mainstream augmented reality platform, a way for people to view and digitally manipulate the physical world around them through the lens of their smartphone cameras.

What that means today is fairly limited. Augmented reality is nascent – people can add simple flourishes on top of their photos or videos, like sticking a pixelated blue beard on a selfie, or adding puppy stickers in the front yard of a photo of their house.

But in Zuckerberg's telling, there are few boundaries for how this technology will evolve. He said he envisioned a world in which people could eventually point smartphone cameras at a bowl of cereal and have an app create tiny sharks swimming in the milk. Friends can leave virtual notes for one another on the walls outside their favourite restaurants, noting which menu item is the most delicious.

Apps like Pokemon Go, the breakout augmented reality hit that went viral globally last year, are just the beginning for Zuckerberg. One day, he mused, household objects could perhaps be replaced entirely by software.

"Think about how many of the things around us don't actually need to be physical," Zuckerberg said in an interview last week. "Instead of a $500 TV sitting in front of us, what's to keep us from one day having it be a $1 app?"

Facebook does not expect to build all of these software experiences itself. At its annual developer conference Tuesday, the company called for computer programmers to help by building augmented reality-based apps to work with what Facebook calls its Camera Effects Platform. Facebook announced new tools to aid developers and will begin the initiative with a small number of partners in a closed test.

Zuckerberg's goal is ambitious – perhaps overly so. Augmented reality efforts have flopped in the past, including Google's much-promoted attempt around spectacles with the technology, known as Google Glass. Facebook has previously gambled on other futuristic technologies – including virtual reality with a US$2 billion purchase of Oculus, the virtual reality goggles maker, in 2014 – but Zuckerberg has acknowledged that it has had difficulty finding traction.

He is also grappling with many issues that have the potential to distract Facebook. The company is under scrutiny for its position as an arbiter of mass media, and questions as to what role Facebook should play in policing content across its platform of nearly 2 billion regular users. That issue was thrust to the forefront this week after a man posted on his Facebook page a video of a murder he committed.

Still, Zuckerberg said he intended to create the next major app ecosystem that would work with Facebook's in-app camera. If successful, Facebook could be in a position similar to that of Apple, which relies on the hundreds of millions of apps in its store to keep users buying the company's smartphones and tablets every year. Facebook, in turn, wants developers to build experiences that entice people to visit its website and apps on a daily – if not hourly – basis.

"Just like Apple built the iPod and iTunes ecosystem before the iPhone, you want to make sure there's a set of content there, even if there's not everything," Zuckerberg said.

Facebook has been building toward this goal for some time. Zuckerberg has spent the past 18 months reorganising his company and its suite of consumer apps – Facebook, Instagram, WhatsApp and Messenger – around a new interface, focused almost entirely on the camera. Slowly, the company has downplayed the role of text inside its apps, instead encouraging people to take and send photos and videos to one another by accessing the in-app camera features.

In time, Facebook hopes that companies like Electronic Arts, Nike and Warner Bros. – which are part of the initial set of partners – will be the ones to bring immersive augmented reality experiences to Facebook's platform.

One early partner app is Giphy Thoughts, made by Giphy, a short-form video start-up that acts as a search engine for animated GIFs. With Giphy Thoughts, for instance, people can attach cartoon thought bubbles above the heads of people they view through their Facebook camera lens.

"It goes back to creative expression," said David Rosenberg, director of business development at Giphy. "Facebook Camera is just going to be this massive audience of people ready to make deeply personal content they can share with their friends."

Hours before Facebook's developer conference Tuesday, Snapchat – a professed "camera company" and Facebook's direct competitor for attention – announced its own take on augmented reality, offering new 3D lenses for use inside of its app. The effect is similar to some of the ideas Zuckerberg has described. Snapchat also offers Spectacles, glasses that allow users to record video from their faces.

Microsoft has its version of an augmented reality headset, called HoloLens, which it unveiled two years ago.

Another dark horse is Magic Leap, the secretive augmented reality start-up that has an enormous investment from Google and Alibaba and is working on hardware to offer a similar experience. The company has yet to unveil an official product.

For the near term, however, Zuckerberg sees the smartphone camera as the first step forward.

"We want to get to this world in the future where you eventually have glasses or contact lenses where you can mix digital or physical objects in the digital world," he said.

New York Times