Technology

In May, Facebook teased a new feature called 3D photos, and itjust what it sounds like. However, beyond a short video and the name, little was said about it. But the companycomputational photography team has just published the research behind how the feature works and, having tried it myself, I can attest that the results are really quite compelling.

In case you missed the teaser, 3D photos will live in your news feed just like any other photos, except when you scroll by them, touch or click them, or tilt your phone, they respond as if the photo is actually a window into a tiny diorama, with corresponding changes in perspective. It will work for both ordinary pictures of people and dogs, but also landscapes and panoramas.

It sounds a little hokey, and I&m about as skeptical as they come, but the effect won me over quite quickly. The illusion of depth is very convincing, and it does feel like a little magic window looking into a time and place rather than some 3D model — which, of course, it is. Herewhat it looks like in action:

I talked about the method of creating these little experiences with Johannes Kopf, a research scientist at FacebookSeattle office, where its Camera and computational photography departments are based. Kopf is co-author (with University College LondonPeter Hedman) of the paper describing the methods by which the depth-enhanced imagery is created; they will present it at SIGGRAPH in August.

Interestingly, the origin of 3D photos wasn&t an idea for how to enhance snapshots, but rather how to democratize the creation of VR content. Itall synthetic, Kopf pointed out. And no casual Facebook user has the tools or inclination to build 3D models and populate a virtual space.

One exception to that is panoramic and 360 imagery, which is usually wide enough that it can be effectively explored via VR. But the experience is little better than looking at the picture printed on butcher paper floating a few feet away. Not exactly transformative. Whatlacking is any sense of depth — so Kopf decided to add it.

The first version I saw had users moving their ordinary cameras in a pattern capturing a whole scene; by careful analysis of parallax (essentially how objects at different distances shift different amounts when the camera moves) and phone motion, that scene could be reconstructed very nicely in 3D (complete with normal maps, if you know what those are).

But inferring depth data from a single camerarapid-fire images is a CPU-hungry process and, though effective in a way, also rather dated as a technique. Especially when many modern cameras actually have two cameras, like a tiny pair of eyes. And it is dual-camera phones that will be able to create 3D photos (though there are plans to bring the feature downmarket).

By capturing images with both cameras at the same time, parallax differences can be observed even for objects in motion. And because the device is in the exact same position for both shots, the depth data is far less noisy, involving less number-crunching to get into usable shape.

Herehow it works. The phonetwo cameras take a pair of images, and immediately the device does its own work to calculate a &depth map& from them, an image encoding the calculated distance of everything in the frame. The result looks something like this:

Apple, Samsung, Huawei, Google — they all have their own methods for doing this baked into their phones, though so far itmainly been used to create artificial background blur.

The problem with that is that the depth map created doesn&t have some kind of absolute scale — for example, light yellow doesn&t mean 10 feet, while dark red means 100 feet. An image taken a few feet to the left with a person in it might have yellow indicating 1 foot and red meaning 10. The scale is different for every photo, which means if you take more than one, let alone dozens or a hundred, therelittle consistent indication of how far away a given object actually is, which makes stitching them together realistically a pain.

Thatthe problem Kopf and Hedman and their colleagues took on. In their system, the user takes multiple images of their surroundings by moving their phone around; it captures an image (technically two images and a resulting depth map) every second and starts adding it to its collection.

In the background, an algorithm looks at both the depth maps and the tiny movements of the camera captured by the phonemotion detection systems. Then the depth maps are essentially massaged into the correct shape to line up with their neighbors. This part is impossible for me to explain because itthe secret mathematical sauce that the researchers cooked up. If you&re curious and like Greek, click here.

Not only does this create a smooth and accurate depth map across multiple exposures, but it does so really quickly: about a second per image, which is why the tool they created shoots at that rate, and why they call the paper &Instant 3D Photography.&

Next, the actual images are stitched together, the way a panorama normally would be. But by utilizing the new and improved depth map, this process can be expedited and reduced in difficulty by, they claim, around an order of magnitude.

Because different images captured depth differently, aligning them can be difficult, as the left and center examples show — many parts will be excluded or produce incorrect depth data. The one on the right is Facebookmethod.

Then the depth maps are turned into 3D meshes (a sort of two-dimensional model or shell) — think of it like a papier-mache version of the landscape. But then the mesh is examined for obvious edges, such as a railing in the foreground occluding the landscape in the background, and &torn& along these edges. This spaces out the various objects so they appear to be at their various depths, and move with changes in perspective as if they are.

Although this effectively creates the diorama effect I described at first, you may have guessed that the foreground would appear to be little more than a paper cutout, since, if it were a personface captured from straight on, there would be no information about the sides or back of their head.

This is where the final step comes in of &hallucinating& the remainder of the image via a convolutional neural network. Ita bit like a content-aware fill, guessing on what goes where by whatnearby. If therehair, well, that hair probably continues along. And if ita skin tone, it probably continues too. So it convincingly recreates those textures along an estimation of how the object might be shaped, closing the gap so that when you change perspective slightly, it appears that you&re really looking &around& the object.

The end result is an image that responds realistically to changes in perspective, making it viewable in VR or as a diorama-type 3D photo in the news feed.

In practice it doesn&t require anyone to do anything different, like download a plug-in or learn a new gesture. Scrolling past these photos changes the perspective slightly, alerting people to their presence, and from there all the interactions feel natural. It isn&t perfect — there are artifacts and weirdness in the stitched images if you look closely, and of course mileage varies on the hallucinated content — but it is fun and engaging, which is much more important.

The plan is to roll out the feature mid-summer. For now, the creation of 3D photos will be limited to devices with two cameras — thata limitation of the technique — but anyone will be able to view them.

But the paper does also address the possibility of single-camera creation by way of another convolutional neural network. The results, only briefly touched on, are not as good as the dual-camera systems, but still respectable and better and faster than some other methods currently in use. So those of us still living in the dark age of single cameras have something to hope for.

Google has published a set of fuzzy but otherwise admirable &AI principles& explaining the ways it will and won&t deploy its considerable clout in the domain. &These are not theoretical concepts; they are concrete standards that will actively govern our research and product development and will impact our business decisions,& wrote CEO Sundar Pichai.

The principles follow several months of low-level controversy surrounding Project Maven, a contract with the U.S. military that involved image analysis on drone footage. Some employees had opposed the work and even quit in protest, but really the issue was a microcosm for anxiety regarding AI at large and how it can and should be employed.

Consistent with Pichaiassertion that the principles are binding, Google Cloud CEO Diane Green confirmed today in another post what was rumored last week, namely that the contract in question will not be renewed or followed with others. Left unaddressed are reports that Google was using Project Maven as a means to achieve the security clearance required for more lucrative and sensitive government contracts.

The principles themselves are as follows, with relevant portions quoted from their descriptions:

1. Be socially beneficial: Take into account a broad range of social and economic factors, and proceed where we believe that the overall likely benefits substantially exceed the foreseeable risks and downsides…while continuing to respect cultural, social, and legal norms in the countries where we operate.
2. Avoid creating or reinforcing unfair bias: Avoid unjust impacts on people, particularly those related to sensitive characteristics such as race, ethnicity, gender, nationality, income, sexual orientation, ability, and political or religious belief.
3. Be built and tested for safety: Apply strong safety and security practices to avoid unintended results that create risks of harm.
4. Be accountable to people: Provide appropriate opportunities for feedback, relevant explanations, and appeal.
5. Incorporate privacy design principles: Give opportunity for notice and consent, encourage architectures with privacy safeguards, and provide appropriate transparency and control over the use of data.
6. Uphold high standards of scientific excellence: Work with a range of stakeholders to promote thoughtful leadership in this area, drawing on scientifically rigorous and multidisciplinary approaches…responsibly share AI knowledge by publishing educational materials, best practices, and research that enable more people to develop useful AI applications.
7. Be made available for uses that accord with these principles: Limit potentially harmful or abusive applications. (Scale, uniqueness, primary purpose, and Googlerole to be factors in evaluating this.)

In addition to stating what the company will do, Pichai also outlines what it won&t do. Specifically, Google will not pursue or deploy AI in the following areas:

• Technologies that cause or are likely to cause overall harm. (Subject to risk/benefit analysis.)
• Weapons or other technologies whose principal purpose or implementation is to cause or directly facilitate injury to people.
• Technologies that gather or use information for surveillance violating internationally accepted norms.
• Technologies whose purpose contravenes widely accepted principles of international law and human rights.

(No mention of being evil.)

In the seven principles and their descriptions, Google leaves itself considerable leeway with the liberal application of words like &appropriate.& When is an &appropriate& opportunity for feedback What is &appropriate& human direction and control How about &appropriate& safety constraints

Itarguable that it is too much to expect hard rules along these lines on such short notice, but I would argue that it is not in fact short notice; Google has been a leader in AI for years and has had a great deal of time to establish more than principles.

For instance, its promise to &respect cultural, social, and legal norms& has surely been tested in many ways. Where can we see when practices have been applied in spite of those norms, or where Google policy has bent to accommodate the demands of a government or religious authority

And in the promise to avoid creating bias and be accountable to people, surely (based on Googleexisting work here) there is something specific to say For instance, if any Google-involved system has outcomes based on sensitive data or categories, the system will be fully auditable and available for public attention

The ideas here are praiseworthy, but AIapplications are not abstract; these systems are being used today to determine deployments of police forces, or choose a rate for home loans, or analyze medical data. Real rules are needed, and if Google really intends to keep its place as a leader in the field, it must establish them, or, if they are already established, publish them prominently.

In the end it may be the shorter list of things Google won&t do that prove more restrictive. Although use of &appropriate& in the principles allows the company space for interpretation, the opposite case is true for its definitions of forbidden pursuits. The definitions are highly indeterminate, and broad interpretations by watchdogs of phrases like &likely to cause overall harm& or &internationally accepted norms& may result in Googleown rules being unexpectedly prohibitive.

&We acknowledge that this area is dynamic and evolving, and we will approach our work with humility, a commitment to internal and external engagement, and a willingness to adapt our approach as we learn over time,& wrote Pichai. We will soon see the extent of that willingness.

Stitch Fix, one of last yearhigh-profile IPOs, has had a bumpy ride for the past few quarters — but it blew out expectations this afternoon for its most recent quarter, and the stock went absolutely nuts.

Therealso a ton of news coming out from the company today, including the hire of a new chief marketing officer as well as the launch of Stitch Fix Kids. All this is pretty good timing because the company appears to be cramming everything into one announcement that is serving as a very pleasant surprise to Wall Street, which is looking for as many signals as it can get that the subscription e-commerce company will end up as one of the more successful IPO stories. Shares of the company are up more than 14 percent after the release came out, where the company beat out expectations that Wall Street set across the board — which, while not the best barometer, serves as a somewhat public barometer as well as what helps determine whether or not it can lock up the best talent.

However, following the announcement, Stitch Fixstock came back down to Earth and is up around 4 percent.

Herethe final line for the company:

• Q3 Revenue: $316.7 million, compared to $306.4 million in estimates from Wall Street and up 29 percent year-over-year.
• Q3 Earnings: 9 cents per share, compared to 3 cents per share in estimates from Wall Street.
• Q4 Revenue Guidance:$310 million to $320 million, compared to Wall Street estimates of around $314 million.
• Cost of goods sold: $178.5 million, up from $139.7 million in Q3 last year.
• Gross Margin: 43.6 percent, up from 43 percent in Q3 last year.
• Advertising spend: $25.2 million, up from$21.3 million in Q3 last year.
• Active clients: 2.7 million, up 30 percent year-over-year (2.5 million last quarter).
• Q3 Net income: $9.5 million ($12.4 million in adjusted EBITDA).

Stitch Fix Kids will carry sizes 2T to 14, which will be across a diverse range of aesthetics &to give kids the freedom to express themselves in clothing that they feel great wearing,& the company said. Those Fixes will include 8 to 12 items that include market and exclusive brands. Stitch Fix launched Stitch Fix Plus in February last year.

&Our new Stitch Fix Kids offering is a testament to the scalability of our platform,& CEO and founder Katrina Lake said in a statement accompanying the release. &We&re excited for Stitch Fix to style everyone in the family and to create an effortless way for parents to shop for themselves and their children. Our goal is to provide unique, affordable kids clothing in a wide range of styles, giving our littlest clients the freedom to express themselves in clothing that they love and feel great wearing.&

Stitch Fix was widely considered a successful IPO last year, though it faced some challenges over the course of the front of the year. But as itexpanded into new lines of subscriptions, its customer base still clearly continues to grow, and the company is still finding newer areas to expand — including the upcoming launch of Kids that it announced today. Like many recent IPOs, Wall Street is likely going to look for continued growth in terms of its core business (meaning, subscribers), but Stitch Fix is showing that itable to not set cash on fire as fresh IPOs sometimes do.

Stitch Fixnew CMO,Deirdre Findlay, comes to the company from Google, where she oversaw the Google Home hardware products, which included Home and Chromecast. Prior to that, Findlay has a pretty extensive history in marketing across a wide variety of verticals beyond just tech, including working with Whirlpool Brands, Allstate Insurance, MillerCoors and Kaiser Permanente, the company said. While Stitch Fix is a digitally native company, itnot exactly an explicit tech company and requires expertise outside of the realm of just the typical tech marketing talent — so getting someone with a pretty robust background like that would be important as it continues to expand into new areas of growth.