Microsoft is offering new tools to help developers build interactive bots that understand natural language, the company announced at its Build conference today.
There are two key components, which are available in preview and are both part of the larger Cortana Intelligence Suite. “The first, Microsoft Cognitive Services, is a collection of intelligence APIs that allows systems to see, hear, speak, understand and interpret our needs using natural methods of communication,” Microsoft said. “The second, the Microsoft Bot Framework, can be used by developers—programming in any language—to build intelligent bots that enable customers to chat using natural language on a wide variety of platforms including text/SMS, Office 365, Skype, Slack, the Web and more.”
Though Microsoft’s own “Tay” bot became a public relations nightmare, the company demonstrated how artificial intelligence applications built with Microsoft technology can be useful in the real world. Most impressive right now is “Seeing AI,” an application to help blind people navigate the world, built by a blind Microsoft software engineer named Saqib Shaikh.
As seen in the video above, Shaikh uses Seeing AI with both a smartphone and the Pivothead smart glasses to get information about his surroundings. While outside, Shaikh taps the side of his glasses to take a picture of a man doing a skateboard trick, and a voice tells him, “I think it’s a man jumping through the air doing a trick on a skateboard.”
Later, Shaikh is having a conversation with a few people and is able to get a description of their faces and emotions. As a blind person, Shaikh said it can be difficult to determine whether people he’s talking to are listening intently or bored to the verge of sleep. But after taking a picture with his sunglasses, he is told that he is talking to a “40-year-old man with a beard, looking surprised,” and a “20-year-old woman, looking happy.”
“The app can describe the general age and gender of the people around me and what their emotions are, which is incredible,” Shaikh said.
In another scene, Shaikh uses the smartphone app in a restaurant to find out what’s on the menu. “Years ago, this was science fiction. I never thought it would be something you could actually do, but artificial intelligence is improving at an ever faster rate,” he said.
Seeing AI is described by Microsoft as a “research project under development,” with no word on public availability.
Seeing AI was written using the intelligence APIs from Microsoft Cognitive Services. Microsoft also demonstrated what its new Bot Framework can do by showing a conversational bot designed for ordering Domino’s Pizza. In the demo, a PC user talks to a Domino’s bot in a chat window.
The bot needs to be able to understand and respond to natural language. In one example, a user asks the bot, “can you deliver three large pepperoni pizzas to my crib?” Since the bot doesn’t initially understand that a “crib” is someone’s house, the bot’s human designer has to tag the word and designate it as a location. This human intervention is required to help the bot learn, but Microsoft Distinguished Engineer Lili Cheng promised that it will not be difficult.
“You don’t need to be a data science expert, and in fact you don’t even really need to know how to code to help your bot improve,” she said. Cheng’s demo included some tools that aren’t yet available in the Bot Framework, so any pizza-ordering bot you build today might not be as intelligent as the one on Microsoft’s stage. (We think you’ll figure out some way to order pizza, though.)
In related news today, Microsoft released its “Skype Bot Platform,” which includes an SDK, API, and other tools. “With this platform, developers can build bots that leverage Skype’s multiple forms of communication, including text, voice, video and 3D interactive characters,” Microsoft said.