Google Assistant Home Control Actions – Introduction

During Google I/O 2017, Google launched its’ Assistant voice-enabled AI platform for iPhone. Google also released more Google Actions developer tools, including a console to configure “Home control” actions. Until now, developers like us had to fill out a form and wait for people from the Google Home Control Actions team enable a development skill.

Google Home Control actions functionally serve the same purpose as Amazon Alexa Smart Home skills – allowing users to control their smart devices, such as plugs, lights, and thermostats with voice. Much like Alexa, Google supports account linking with OAuth. Google Home Control action, also sends one of the predefined intents. Just like with Alexa Smarthome, Home Control does not support custom utterances.

Coding for processing Google actions may be hosted on Google Cloud, or anywhere else. Commands are sent as HTTP requests. Essentially, Home Control actions processing software for Google Cloud is a custom Express.JS application middleware. Our client decided to host the development code on Google Cloud and the production version on AWS Lambda. To move our Google Cloud project to AWS Lambda hosting, we used an existing piece of software – AWS Serverless Express.

Your wrapper AWS Lambda code might look like:

'use strict';

import { actionHandler } from './google_home_control';
const atob = require('atob');
const awsServerlessExpress = require('aws-serverless-express');
const awsServerlessExpressMiddleware = require('aws-serverless-express/middleware');
const express = require('express');
const bodyParser = require('body-parser');
const cors = require('cors');
const compression = require('compression')
const app = express();

app.use(compression());
app.use(cors());
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(awsServerlessExpressMiddleware.eventContext());

app.post('/', actionHandler);

const server = awsServerlessExpress.createServer(app);

export function lambda_handler(event, context, callback) {
    console.log("LAMBDA STARTED: ");
    console.log("LAMBDA EVENT: ", event);
    console.log("LAMBDA CONTEXT: ", context);
    console.log(`EVENT BODY IS BASE64: ${event.isBase64Encoded}`);
    
    try {
        if(event.isBase64Encoded) {
            event.body = atob(event.body);
            event.isBase64Encoded = false;
        }
        awsServerlessExpress.proxy(server, event, context);
    }
    catch(error) {
        console.log("ERROR IN awsServerlessExpress: ", error);
    }
}

Home Control actions API comes down to three (3) intents: SYNC – called when an account is linked using OAuth, QUERY to read the state of the devices and EXECUTE to perform control commands on behalf of the users. The SYNC intent is roughly a functional equivalent of Alexa Smart Home DISCOVERY. Unlike Alexa Smarthome API version 2, invoking different control intents depending on the desired action, Google Home control has only one control intent with specific action requests being passed in the payload, like "command": "action.devices.commands.OnOff". The device capabilities are called traits in Google terminology, for example, to support power control, devices must have action.devices.traits.OnOff.

If you have questions or need help with Amazon Alexa skills or Google Assistant actions, reach out to our experts.

Please follow and like us:

Amazon Alexa vs Google Assistant vs Microsoft Cortana

During the 2017 Build event, Microsoft unveiled the promised Cortana Assistant -powered speakers and released Cortana to developers looking to introduce voice skill. We have hands-on experience with Amazon Alexa “custom” and “smarthome” skills. Our engineers were also lucky to participate in Google Assistant “Home control” actions Early Access Program, and built regular Google Assistant actions. We, however, have not had a chance to get our hands on Cortana before the official release. Let’s look at all three platforms, based on the publicly available information.

Amazon Alexa
Alexa is the first voice-enabled speaker on the market. Skills are easy to configure, through an intuitive web interface. Amazon has great documentation and paid developer support is available. There are tons of examples for programmers building both custom and smart home skills. The code could be hosted on AWS Lambda, but it is not a must. Hosting on AWS allows invoking skill processing code by ARN. If the skill is simple, with a few tricks such as keeping lambda warm with CloudWatch events, you might be able to host an extremely quick autoscaling skill while fitting onto the AWS Lambda free tier. Alexa currently supports the US and UK English, as well as German.

Google Assistant
Google Assistant is available on Google Home and Google Pixel phones, but it is coming to every Android smartphone and other surfaces. Actions execution endpoints could be hosted on Google Cloud, or elsewhere. Google Assistant actions are still less mature than Alexa skills. Documentation is less thorough, probably because Google started later in the game. Google Assistant Home control actions are still in the Early Access Program. We won’t discuss specifics until Home control has been officially released. There are however Home control actions running in production. Most major smarthome hardware manufacturers have either already introduced their products on Google Home, or in the process. Google’s one apparent advantage is many more languages than Alexa. Further, Google Assistant is designed to integrate with other Google services, starting from the calendar. We believe that Google Assistant has a lot of potential. Time will tell how Google will execute leveraging its’ strengths and making actions development partner-friendly.

Microsoft Cortana
Based on the publicly available information, there is no Alexa smart-home or Google Assistant “Home control” -style API, yet. Cortana supports more languages than Alexa, yet fewer than Google Assistant. Several manufacturers partnered with Microsoft and announced their Cortana speakers, yet none have been released. It appears that Cortana skills have to be hosted on Microsoft Azure and nowhere else.

Conclusion
Microsoft is a bit late to the party, but this is probably only the beginning. There are rumors of an Apple Siri speaker, which may not pan out, or we might see one come out of Cupertino.
We believe that the main battle might take place in the cockpit of your car, where speakers make even more sense.
In the meantime, please stay tuned. Once Google releases “Home control” to the public, we will tell you how we managed to build a complex product for Google Assistant, in just one week, leveraging our Alexa experience and assets.

Please follow and like us:

Voice-enabling smart-home devices with Amazon Alexa (Part III)

In the previous installment of our Alexa skills development exploration, we got to the point of building smarthome and custom skills for Amazon Echo. We used tools babel and webpack to transpile modern ES6/ES7 JavaScript into an older JavaScript version supported by Amazon Lambda. Now, our project produces a single output with either the smart-home or the custom Alexa skill. We are able to manually compress the transpiled JavaScript and upload it to Amazon Web Services using the Lambda console.
We, however, want the deployment done automatically. We will configure serverless framework files to:
– compress our Lambda code,
– create a new “stack” on the AWS side, if necessary,
– copy the Lambda to Amazon and set its’ invocation trigger.

Our main serverless configuration file is serverless.yml. It has definitions split between function-level configurations and ones relevant to the specific stage and locality. It looks like:

service: ${file(./${env:DEPLOY_FILE_NAME}):service}

provider:
 name: aws
 custom:
   globalSchedule: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):globalSchedule}
   roleName: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):roleName}
   profileName: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):profileName}
 plugins:
   - pluginHandler
   - serverless-alexa-plugin
 runtime: nodejs4.3
 cfLogs: true
 stage: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):stage}
 region: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):region}
 memorySize: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):memorySize}
 timeout: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):timeout}
 keepWarm: false
 useApigateway: false

package:
 exclude:
   ${file(./${env:DEPLOY_FILE_NAME}):exclude}
functions:
 lambdaHandler:
   handler: ${file(./${env:DEPLOY_FILE_NAME_STAGE}):handler}
   events:
     ${file(./${env:DEPLOY_FILE_NAME}):events}

When settings apply to the function regardless of the deployment stage, we keep them in DEPLOY_FILE_NAME.
For example, our custom skill configuration looks like the below:
service: alexa-CustomSkillAdapter
exclude:
 - src/**
 - test/**
 - webpack/**
 - dist/**
 - build/**
 - node_modules/**
 - UiTest/**
 - bower.json
 - CHANGELOG.md
 - deploy.sh
 - jsconfig.json
 - karma.conf.js
 - LICENSE
 - package.json
 - pluginHandler.js
 - README.md
 - serverless_settings/**
 - settings_eu_customskill.yml
 - settings_eu_smarthome.yml
 - settings_us_customskill.yml
 - settings_us_smarthome.yml
 - .idea/**
 - .npmignore/**
 - .jshintrc
 - event.json
 - lambda_function_smart_home.js
 - documentation.docx
events:
 - alexaSkill

The serverless framework documentation covers what parameters are supported. We just want to draw your attention to the events setting. Our lambda functions do not use storage or databases, and only require events that trigger the Alexa intents processing. For custom skill, this event type is alexaSkill. At the time this article is being written, serverless does not support Alexa Smart Home events, so the smarthome skill trigger needs to be set manually. The AWS CloudFormation lack of support for Smart-Home events as AWS Lambda triggers is due to a unimplemented feature, already on AWS roadmap – EventSourceToken property for the AWS::Lambda::Permission. However, if you want to automatically set the trigger now, before the feature is implemented by Amazon Web Services, it is still possible using a simple shell script. Using the aws command line tools, you could set the trigger by executing the below command:

$ aws lambda add-permission --function-name $MY_FUNCTION_NAME --action lambda:InvokeFunction --principal $PRINCIPAL --region $AWS_REGION --event-source-token $ALEXA_SKILL_ID --statement-id 8

We have now covered the design and implementation of an Alexa skills project, for connected devices. Using babel, webpack and serverless, we were able to create a single Node.JS project used to produce and automatically deploy both custom and smart-home Amazon Alexa skills. The builds script allow specifying different configurations depending on stage and locality.
If you have questions or need assistance with your Amazon Alexa projects, please contact us and maybe we could help. If you are interested in controlling smart-home devices with your voice, stay tuned. We are about to publish a post on building Google Assistant Home Control Actions.

Please follow and like us:

Voice-enabling smart-home devices with Amazon Alexa (Part II)

In the previous post, we discussed setting-up Amazon Alexa skills and an environment for building and deploying an autoscaling service to host processing of the skill intents.
Let’s start coding our AWS Lambda functions to handle events sent to us by Alexa skills.
In our lambda_function_smart_home.js, we handle events sent to us by the smarthome skill. The code is below.

'use strict';

import HandlerProvider from './alexa/smarthome/handler_provider.js';
import * as utils from './alexa/common/utils.js';
import logging from './alexa/common/logging.js';
import * as exception from './alexa/common/exception.js';

var handle_event_async = async function(event, context, callback)
{
   // For more details on the format of the requests served by this lambda function check here:
   // https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills-kit-interface-reference#request-format
   logging.info(event);
   try {
       // Prevent someone else from configuring a skill that sends requests to this function
       var session = event.session;
       utils.validate_application_id(session.application.applicationId);
       var provider = new HandlerProvider(event);
       var handler = provider.get_handler();
       var res = await handler.handle_event(event);
       console.log(JSON.stringify(res));
       callback(null, res);
   } catch (error) {
       console.log(error);
       if(error instanceof exception.SmartHomeException) {
           logging.error(error.get_error_response());
           callback(null, error.get_error_response());
       } else {
           logging.error("Unable to return a sensible response to the user due to error: 1");
           var errorUnhandled = new exception.DriverInternalError();
           callback(null, errorUnhandled.get_error_response());
       }
   }   
}

export function lambda_handler(event, context, callback) {
   handle_event_async(event, context, callback);   
}

In our function, actual intents are handled by the Smart Home handler class.

Let’s init our npm project, if you have no done it yet, and add packages and scripts necessary to compile our code.
First, we put together .babelrc for transpilation, below:

{
 "presets": [
   ["latest", { "modules": false }]
 ],
 "plugins": ["babel-plugin-add-module-exports"]
}

We will need the babel exports plugin for our tests, once the skill is ready. Now, we got the add some packages to our node project. We used the below version:
"devDependencies": {
   "babel-cli": "^6.18.0",
   "babel-core": "^6.21.0",
   "babel-eslint": "^7.1.1",
   "babel-loader": "^6.2.10",
   "babel-plugin-add-module-exports": "^0.2.1",
   "babel-polyfill": "^6.20.0",
   "babel-preset-latest": "^6.16.0",
   "babel-runtime": "^6.20.0",
   "webpack": "2.2.1"
}

Now, the packages are in place. Let’s add scripts to build, test and deploy our project. We added the following commands to our package.json file.
scripts": {
   "test": "./node_modules/mocha/bin/mocha --require babel-polyfill --no-timeouts --colors",
   "clean": "rimraf lib dist coverage",
   "build:smarthome:us:dev": "./node_modules/.bin/webpack --env.skill=smarthome --env.locale=us --env.stage=dev --config webpack/webpack.config.dev.js",
   …
   "build:smarthome:all": "npm run build:smarthome:us:all && npm run build:smarthome:eu:all",
   "build:customskill:all": "npm run build:customskill:us:all && npm run build:customskill:eu:all",
   "build": "npm run build:smarthome:all && npm run build:customskill:all",
   "build:debug": "./node_modules/.bin/webpack src/app.js dist/debug/app.js --config webpack/debug/webpack.config.debug.js",
   "deploy": "sh ./deploy.sh"
}

The last script deals with serverless. We found it easier, in our case, to have a separate shell script to invoke the serverless framework executable. This allowed us to both: use cached credentials for deployment from CI tools, and to do an interactive deployment when invoked manually, with some of the parameters omitted.

Now, our code will compile and produce a single JavaScript file suitable for AWS Lambda deployment. To work locally, without having to deploy to Amazon to try every code modification, we used the app.js file that looks like the below:

'use strict';

//import {lambda_handler} from './lambda_function_custom_skill';
import {lambda_handler} from './lambda_function_smart_home';

let applianceId = "e96b94ba-da2b-.................";
let createAmazonHelpEvent = function(access_token)
{
   return {
     "session": {
       "sessionId": "SessionId.db5adc55-878f-43ac-a58d-..........",
       "application": {
         "applicationId": "amzn1.ask.skill.f27c2889-d3f5-....-....-........."
       },
       "attributes": {},
       "user": {
         "userId": "amzn1.ask.account.A……………………………………………………………………………………….",
         "accessToken": access_token
       },
       "new": true
     },
     "request": {
       "type": "IntentRequest",
       "requestId": "EdwRequestId.b9caccff-34c7-4bcb-....-............",
       "locale": "en-US",
       "timestamp": "2016-11-03T19:56:51Z",
       "intent": {
         "name": "AMAZON.HelpIntent",
         "slots": {}
       }
     },
     "version": "1.0"
   };
};

let request = require('request');
…
let {username, password} = getCredentials("us");

request.post(url, function (error, response, body) {
 if (!error && response.statusCode == 200) {
   let user = JSON.parse(body);
   lambda_handler(createEvent(user.access_token), null, (error, result)=>{
    console.log(result);
    console.log("DONE - lambda_handler");
   });
 }
});

In this article, we covered building barebone Amazon Alexa skills with modern JavaScript. While the code built as described in the previous installment and this post should be fully functional, provided that developers implement intent handlers, we still have not covered in detail the serverless framework configuration. We will do in our next article.

Please follow and like us:

Voice enabling smart-home devices with Amazon Alexa

skills_api_logoOur client is a smart-home devices manufacturer. The company decided to voice-enable their products, starting with an ability to interact with the Amazon Echo family of devices, powered by Amazon Alexa technology.

The client had extensive expertise in smart devices field. Their competent IT department wanted an enterprise-grade solution – easy-to-maintain and built for scale.

We were going to build two Alexa skills above utilizing the Smart Home Skill and Custom Skill APIs. Smarthome API do not support and hence do not require setting-up custom utterances. Smarthome skills are also easier to invoke. However, the lack of customization limits the ability to expose some of finer features offered by specific target hardware. In the time that we have worked with Amazon on Alexa skills, Smarthome API has been enhanced several times. We expect that eventually, it will accommodate more smart-home devices and usage scenarios.

Building a basic skill is fairly easy. We used a configuration with account-linking and an OAuth endpoint authenticating devices with access tokens. While a startup may not need this, we had to accommodate a well-established development process with the product being built for multiple stages such as development, beta, production and having to deal with different API and authentication endpoints depending on the stage and locality – our product needed to support different regions.

Maintaining all of this manually would create a huge problem once the product moved from the initial development to subsequent releases. Therefore, the client had an established system in place, with automated builds and continuous integration.

Alexa skills send events, when users issue commands. One of the Alexa skill configuration parameters is the endpoint processing those events. While various cloud “resources” could be used to host the code processing Alexa requests, programmers typically use Amazon’s own Lambda service. In addition to being an auto-scaling solution, using lambda simplifies the skill configuration and alleviates the need to setup the security certificate.

lambda nodejs-logo

endpoint

Amazon Lambda supports several programming languages including Java, JavaScript and Python. Our language of choice was JavaScript, because much the client’s backend was already running on Node.JS. If we were using Python, we might have opted for the zappa package. However, with Node.JS, we selected the serverless framework for configuration and deployment automation. We like serverless because of support for Amazon Lambda, as well as Google CloudFunctions and Microsoft Azure. That would come in handy when adding Google Home and Microsoft Cortana Assistant.

home_framework_v1_1

With Node.JS and serverless, we almost had a complete toolkit. The remaining problem was the Amazon’s lag behind Node.JS release cycles. At the time of the writing, Amazon Lambda supports only Node.JS version 4.3. Recently, it was 4.2 and the latest upgrade happened about the same time version 7 came out. Our missing ingredients were babel to transpile the code written in the new ES6/ES7 JavaScript syntax to the version Lambda accepts, and webpack to build all of our code, including modules, into a single JavaScript file.

Our project structure will be:

A source directory with Alexa client, IoT-backend client, common functionality directory and one directory for each skill type – smarthome and customskill. In the top source directory, we create three JavaScript files: smarthome_lambda_handler.js and customskill_lambda_handler.js for the two skills, and app.js to run/debug our code locally, without having to upload it to AWS Lambda.

Now, we need to configure webpack to build our code. We create a webpack directory off root of the project tree and populate it with webpack 2.2 configurations. We create one webpack configuration for each development and production builds.

We used a base config shown below:

'use strict';

var webpack = require('webpack');

module.exports = {
    output: {
        library: 'projectpure',
        libraryTarget: 'umd'
    },
    resolve: {
        extensions: ['.json', '.jsx', '.js']
    },
    module: {
        rules: [
            { test: /\.cfg$/, loader: 'raw-loader'},
            { test: /\.js$/, loaders: ['babel-loader'], exclude: /node_modules/ },
        ]
    },
    target: 'node',
    externals: [ 'aws-sdk'  ],
    plugins : [
        new webpack.ContextReplacementPlugin(/moment[\/\\]locale$/, /en/)
    ]
};

Actual build-type configurations inherit from the base, so our development config looks like:
'use strict';

var webpack = require('webpack');
var CopyWebpackPlugin = require('copy-webpack-plugin');
var baseConfig = require('./webpack.config.base');

module.exports = function(options) {
    let skill  = options.skill;
    let locale = options.locale;
    let stage  = options.stage;
    let skillEntry = 'lambda_function_smart_home.js';
    if(options.skill !== 'smarthome') {
        skillEntry = 'lambda_function_custom_skill.js';
    }
    baseConfig.entry = ['babel-polyfill', `./src/${skillEntry}`];
    var config = Object.create(baseConfig);
    config.output.filename = skillEntry;
    config.output.path = `./dist/${locale}/${skill}/${stage}`;
    config.plugins = config.plugins.concat([
        new webpack.DefinePlugin({
            'process.env.NODE_ENV': JSON.stringify('development')
        }),
        new CopyWebpackPlugin([{ from: `./src/config/${locale}/${stage}/config.cfg` }])
    ]);
    return config;
}

Now, our project can be built. Let’s add code to it, and produce a basic skill then deploy to AWS Lambda using serverless framework. See how we did this, in the next post.

Please follow and like us: