Microsoft sees a future full of bots – ‘Intelligent’ conversation with machines
In the movie ‘Her’, the protagonist falls in love with his operating system. The software knows and understands him completely, and communicates with him in a way that he considers ideal. Anyone who hears Microsoft talk about the personal assistant Cortana during its Build conference can draw careful parallels with the Hollywood film.
Microsoft CEO Satya Nadella emphasized during his opening keynote that Cortana is “a personal digital assistant that knows you, knows your world and is always with you, on all your devices.” Paul Bloom, product lead on the Cortana team, said it more explicitly during a session later: “Cortana understands the user on a deep level. The experience with Cortana is different for every user. Cortana understands their agenda, preferences, interests and context , and uses that information to provide personalized experiences.”
The fact that Microsoft still has something to learn about the development of artificial intelligence was apparent prior to Build. It seemed like such a good idea to unleash a chatbot on Kik, Twitter and GroupMe, but in no time the ‘virtual girlfriend’ started making racist and offensive statements, spamming users and stating that she had smoked weed where the police attended. Also on a new attempt, Tay, an acronym for ‘thinking about you’, turned out to be completely off the rails. Nadella acknowledged that Microsoft needs to go back to the drawing board. “We want technology that offers the best of humanity, not the worst.”
Despite that incident, Microsoft made it clear that artificial intelligence, bots, and communicating with computers in natural language are the future. In fact, the company presented a number of maxims: natural language is the new user interface, bots are the new apps, digital assistants are meta-apps annex new browsers and all interaction must involve intelligence. Microsoft calls the intelligent possibilities to communicate digitally with each other and with computers with a marketing term ‘conversations as a platform’. This platform should extend beyond Windows; Microsoft also wants to play a role in this area with iOS and Android.
One of the first elaborations of that vision concerns an improvement of Cortana in Windows. Cortana will then sit on your lock screen waiting for a conversation with you, suggesting questions you could ask, for example. If you ask such a question, the assistant can answer it, regardless of whether you are logged in. In addition to asking questions, you can also play music. The same feature should come to Cortana on Android. Microsoft did not make clear what exactly can be done from the lock screen. It does not seem desirable that unauthorized persons can access information on your system such as your calendar, but Microsoft did not reveal much about the security restrictions.
Cortana does get extensive integration with the user’s calendar. The assistant can automatically schedule appointments based on emails and send reminders about appointments. Microsoft also demonstrated how you, as a user, can reply to a message received via Android via the PC, after which Cortana automatically recognizes an appointment in that message as a double meeting. The assistant then suggested a change and immediately adjusted the consequences of the changes for the agenda.
For the new appointment, the user was offered suggestions regarding location, dining options and possible activities. For those suggestions, Cortana consults experts: apps that make personal recommendations about food, for example, and possibly also offer an ordering option. Developers of those apps will have the option to have their app actively respond to conversations with Cortana. For example, a question about a location the user visited during a previous trip, such as a restaurant where he once had a good meal, can provide directions, taxi order or call option.
To emphasize the new “intelligent” features of Cortana, Microsoft has changed the name of its underlying development platform for the assistant from Cortana Analytics Suite to Cortana Intelligence Suite. The suite is the link between data and the final apps and services, and would add ‘intelligence’, with which Microsoft naturally tries to monetize. During Build, two new components were added to this suite: Microsoft Cognitive Services and Microsoft Bot Framework.
Microsoft Cognitive Services consists of 22 APIs that developers can use to give their apps speech recognition, image recognition, translation, and search capabilities. For example, developers can use an emotion API for their app to recognize whether the user is happy, sad, surprised, or annoyed. This can be useful to gain insight into the response of users to services, but can also be used, for example, to have avatars respond to the user.
The Bot Framework gives developers the tools to develop chatbots for Slack, Skype, Office 365, WeChat, Line, email and SMS, among others. A BotBuilder sdk has been made available on GitHub for this purpose.
Such chatbots have existed for some time for Slack and Telegram, among others, and the conversational form has also made its appearance in news apps, with the arrival of the Quartz app . In its simplest form, such a bot receives a text message, performs an action and sends a message back. Microsoft makes it possible to connect bots to various services via a Bot Connector. A link with Cortana ensures that this personal assistant can bring in a bot if the conversation shows that such a service might be desired. Moreover, thanks to the availability of advanced cognitive APIs, these bots can acquire features that were previously only accessible to large software companies such as Microsoft itself.
Microsoft demonstrated an extensive development of the bot platform with Skype. Cortana is available in Skype at the top left of the screen, waiting for conversations and chats to analyze them and see if suggestions need to be made. In the demonstration, Cortana informed the user that a Cups and Cake bot had notified the assistant that an order was ready for pickup. Cortana then asks if the order needs to be tracked and provides an estimated delivery time. So Cortana replaces the interaction with the bot by going through the steps with a dialogue.
Conversely, developers of bots for Skype can actively let Cortana make suggestions, for example by suggesting hotels in a conversation about an upcoming trip and bookings through the hotel bot. Cortana has already passed the details to that bot, so that the booking can be processed faster. Within Skype, users can see which bots are present by clicking on a robot logo and they can also search for new bots here.
During Build Skype, Microsoft demonstrated bots that can recognize images in photos, manipulate images such as putting faces on other people, post news messages based on conversations, play music, and so on. The idea is that all sorts of functionality for which users still need separate apps will soon be provided via communication tools. That starts with text, but doesn’t stop there, the company claims. In the long term, the user must be given the same conversation options via audio and image. Skype showed a preview with animated bots.
Microsoft claims that the combination of bots and Cortana makes chats more productive and efficient, but that seems to depend a lot on how the bots are programmed. An overly intrusive commercial bot can seriously frustrate conversations, especially if there are more. Anyone who is harassed during a conversation about the weather with bots trying to sell him trips to the sun will soon want to turn them off. There are still many open security and privacy issues that Microsoft left unanswered during Build. Is it visible enough which bots listen in on group chats? How transparent are the operation and access to personal data? Is Microsoft going to monitor all bots?
Microsoft emphasized that developers, and therefore users, must prepare for a future where artificial intelligence plays a major role in virtually every digital experience people have. The beginning of this was made visible during Build. The company expects that we are about to start communicating with our systems in a different way. “Until now, machines have been interacting on their terms,” says Greg Sullivan of the Windows & Devices Group. “Type a letter wrong in a url and you won’t get to the page you want,” he gives as an example. “What will come next is that the interaction with computers will be more like that with humans.”
Microsoft is not alone in this vision, but by focusing so emphatically on its personal assistant in combination with bots, the company makes it clear that it does not want to miss the boat. When it comes to mobile apps, Microsoft still plays a marginal role, so it makes sense for Microsoft to push bots forward as new apps. The question is whether the company is not acting hastily, because many security and privacy issues are still open, and the failed experiment with chatbot Tay shows that artificial intelligence is barely in its infancy.