Technology

Facebook has been a bit slow to adopt the voice computing revolution. It has no voice assistant, its smart speaker is still in development, and some apps like Instagram aren&t fully equipped for audio communication. But much of that is set to change judging by experiments discovered in Facebookcode, plus new patent filings.

Developing voice functionality could give people more ways to use Facebook in their home or on the go. Its forthcoming Portal smart speaker is reportedly designed for easy video chatting with distant family, including seniors and kids that might have trouble with phones. Improved transcription and speech-to-text-to-speech features could connect Messenger users across input mediums and keep them on the chat app rather than straying back to SMS.

But Facebookvoice could be drowned out by the din of the crowd if it doesn&t get moving soon. All the major mobile hardware and operating system makers now have their own voice assistants like Siri, Alexa, Google Assistant and Samsung Bixby, as well as their own smart speakers. In Q2 2018, Canalys estimates that Google shipped 5.4 million Homes, and Amazon shipped 4.1 million Echoes. AppleHomePod is off to a slow start with less than 6 percent of the market, behind Alibabasmart speaker, according to Strategy Analytics. Facebookspotty record around privacy might deflect potential customers to its competitors.

Given Facebook is late to the game, it will need to arrive with powerful utility that solves real problems. Herea look at Facebooknewest developments in the voice space, and how its past experiments lay the groundwork for its next big push.

Aloha voice

Facebook is developing its own speech recognition feature under the name Aloha for both the Facebook and Messenger apps, as well as external hardware — likely the video chat smart speaker itdeveloping. Code inside the Facebook and Messenger Android apps dug up by frequent TechCrunch tipster and mobile researcher Jane Manchun Wong gives the first look at a prototype for the Aloha user interface.

Labeled &Aloha Voice Testing,& as a user speaks while in a message thread, a horizontal blue bar expands and contracts to visualize the volume of speech while recognizing and transcribing into text. The code describes the feature as having connections with external Wi-Fi or Bluetooth devices. Itpossible that the software will run on both Facebookhardware and software, similar to Google Assistant that runs both on phones and Google Home speakers. [Update: As seen below, the Aloha feature contains a &Your mobile device is now connected Portal& screen, confirming that name for the Facebook video chat smart speaker device.]

Facebook declined to comment on the video, with its spokesperson Ha Thai telling me, &We test stuff all the time — nothing to share today but my team will be in touch in a few weeks about hardware news coming from the AR/VR org.& It unclear if that hardware news will focus on voice and Aloha or Portal, or if itmerely related to FacebookOculus Connect 5 conference on September 25th.

A source previously told me that years ago, Facebook was interested in developing its own speech recognition software designed specifically to accurately transcribe how friends talk to each other. These speech patterns are often more casual, colloquial, rapid and full of slang than the way we formally address computerized assistants like Amazon Alexa or Google Home.

Wong also found the Aloha logo buried in Facebookcode, which features volcano imagery. I can confirm that I&ve seen a Facebook Aloha Setup chatbot with a similar logo on the phones of Facebook employees.

If Facebook can figure this out, it could offer its own transcription features in Messenger and elsewhere on the site so users could communicate across mediums. It could potentially let you dictate comments or messages to friends while you have your hands full or can&t look at your screen. The recipient could then read the text instead of having to listen to it like a voice message. The feature also could be used to power voice navigation of Facebookapps for better hands-free usage.

Speaker and camera patents

Facebook awarded patent for speaker

Facebookvideo chat smart speaker was reportedly codenamed Aloha originally but later renamed Portal, Alex Heath of Business Insider and now Cheddar first reported in August 2017. The $499 competitor to the Amazon Echo Show was initially set to launch at FacebookF8 in May, but Bloomberg reported it was pushed back amid concerns that it would exacerbate the privacy scandal ignited by Cambridge Analytica.

A new patent filing reveals Facebook was considering building a smart speaker as early as December 26th, 2016 when it filed a patent for a cube-shaped device. The patent diagrams an &ornamental design for a speaker device& invented by Baback Elmieh, Alexandre Jais and John Proksch-Whaley. Facebook had acquired Elmiehstartup Nascent Objects in September of that year and henow a technical project lead at Facebooksecretive Building 8 hardware lab.

The startup had been building modular hardware, and earlier this year he was awarded patents for work at Facebook on several modular cameras. The speaker and camera technology Facebook has been developing could potentially evolve into whatin its video chat speaker.

The fact that Facebook has been exploring speaker technology for so long and that the lead on these patents is still running a secret project in Building 8 strengthens the case that Facebook has big plans for the voice space.

Patents awarded to Facebook show designs for a camera (left) and video camera (right)

Instagram voice messaging

And finally, Instagram is getting deeper into the voice game, too. A screenshot generated from the code of InstagramAndroid app by Wong reveals the development of a voice clip messaging feature heading to Instagram Direct. This would allow you to speak into Instagram and send the audio clips similar to a walkie-talkie, or the voice messaging feature Facebook Messenger added back in 2013.

You can see the voice button in the message composer at the bottom of the screen, and the code explains that to &Voice message, press and hold to record.& The prototype follows the recent launch of video chat in Instagram Direct, another feature on which TechCrunch broke the news thanks to Wongresearch. An Instagram spokesperson declined to comment, as is typical when features are spotted in its code but aren&t publicly testing yet, saying, &Unfortunately nothing more to share on this right now.&

The long road to Voicebook

Facebook has long tinkered in the voice space. In 2015, it acquired natural language processing startup Wit.ai that ran a developer platform for building speech interfaces, though it laterrolled Wit.ai into Messengerplatform teamto focus on chatbots. Facebook also began testing automatically transcribing Messenger voice clips into text in 2015 in what was likely the groundwork for the Aloha feature seen above. The company also revealed its M personal assistant that could accomplish tasks for users, but it was only rolled out to a very limited user base and later turned off.

The next year, Facebookhead of Messenger David Marcus claimed at TechCrunch Disrupt that voice &is not something we&re actively working on right now,& but added that &at some point itpretty obvious that as we develop more and more capabilities and interactions inside of Messenger, we&ll start working on voice exchanges and interfaces.& However, a source had told me Facebooksecretive Language Technology Group was already exploring voice opportunities. Facebook also began testing its Live Audio feature for users who want to just broadcast sound and not video.

By 2017, Facebook was offering automatic captioning for Pages& videos, and was developing a voice search feature. And this year, Facebook began trying voice clips as status updates and Stories for users around the world who might have trouble typing in their native tongue. But executives haven&t spoken much about the voice initiatives.

The most detailed comments we have come from Facebookhead of design Luke Woods at TechCrunch Disrupt 2017 where he described voice search saying it was, &very promising. There are lots of exciting things happening…. I love to be able to talk to the car to navigate to a particular place. Thatone of many potential use cases.& Italso one that voice transcription could aid.

Itstill unclear exactly what FacebookAloha will become. It could be a de facto operating system or voice interface and transcription feature for Facebooksmart speaker and apps. It could become a more full-fledged voice assistant like M, but with audio. Or perhaps it could become Facebookbridge to other voice ecosystems, serving as FacebookAlexa Skill or Google Assistant Action.

When I asked Woods &How would Facebook on Alexa work,& he said with a smile &Thata very interesting question! No comment.&

Creators of the 1Sheeld, a tool designed to connect smartphones to Arduino boards, have created something even more interesting. Their latest product, the Elkrem, is a smart kit for creating blockchain IoT devices and they have raised $250,000 from Endure Capital and Consensys to build the project.

The founders are Amr Saleh and Islam Mustafa launched the 1Sheeld at TechCrunch Disrupt 2013 and sold tens of thousands of units in 120 countries. Now they&re building a new tool based entirely on blockchain.

&Elkrem is a Blockchain hardware development board. It allows Blockchain developers to integrate Dapps with hardware prototypes in an easy way without having deep knowledge in hardware development, and also allows electrical engineers and hardware developers to connect Blockchain to their hardware projects without having deep knowledge of how the Blockchain works,& said Saleh. &So they both can trigger actuators through smart contracts and log sensors data to smart contracts as well.&

The board is similar to an Arduino and has two processors, storage, and WiFi model. One processor runs a specialized Linux variant with interfaces to Ethereum, IPFS, Swarm, Whisper, Bitcoin, Status.im, and others. The other processor can do anything else you throw at it.

&Our edge is faster development, faster prototyping and faster go to market,& said Saleh. &The board allows you to send private, decentralized IoT messages using peer-to-peer communication&

What does all this mean Basically ita little board that makes it far easier to manage your Blockchain efforts. It uses a library called Koyn to let you accept payments in Bitcoin with a single line of code and they even built a few cool projects including a Bitcoin-enabled candy machine and an electrical outlet that you can rent with Bitcoins. The team plans to go live on Kickstarter later this year.