Notes on Sentiment Sweep

After incorporating sentiment analysis APIs into four previous skills, I was curious how they stacked up against each other on the same utterance.  My four previous skills used sentiment analysis in the form of a challenge: say something with a target sentiment and magnitude and score points based on how close you get. Rather than complicating this up with targets and points, I adapted the utterance processing I used with another skill, Emotional Radar, which takes free-form sentences one at a time and reports back with the analysis. The result is “Sentiment Sweep“.

Sentiment Platforms

My four sentiment game skills use Amazon Comprehend, Microsoft Azure, Expert.ai and Symbl.ai. I had written the game with Comprehend last year, and adapted it with the latter three APIs for hackathon entries.  I encountered issues when using Symbol.ai, as it required two API calls, one to process the utterance and one to analyze it (on top of a call to get the API key first for the other two calls). Despite coding for parallelism among the four APIs (see below), the two calls to Symbol.ai would choke the Lambda backend as often as not, causing the skill to fail ungracefully. So I swapped out the Symbol.ai API (sorry), and swapped in the API from ParallelDots. I had used the ParallelDots emotion analysis API for Emotional Radar, so I was ready to go with it.

Parallelism

This was the first time I endeavored to do parallel API calls, so needed to figure how to do that in JavaScript. This is where I discovered the Promise.allSettled() method:

try {
        [
          DSreturn,
          PDsents,
          //SYMBLsents,
          EXPERTsents,
          COMPREHENDsents,
          AZUREsents,
        ] = await Promise.allSettled([
          callDirectiveService(handlerInput),
          pd.sentiment(query, "en"),
          //getSymblResults(handlerInput, query),
          getExpertResults(handlerInput, query),
          getComprehendResults(query),
          getAzureResults(handlerInput, query),
        ]);
}

Note the Symbl.ai calls regrettably commented out. Note also I do a parallel call to callDirectiveService, which I use to play a little sound to keep the user happy while I make the API service calls.

async function callDirectiveService(handlerInput) {
  // Call Alexa Directive Service.
  const requestEnvelope = handlerInput.requestEnvelope;
  const directiveServiceClient =
    handlerInput.serviceClientFactory.getDirectiveServiceClient();
  const requestId = requestEnvelope.request.requestId;
  const endpoint = requestEnvelope.context.System.apiEndpoint;
  const token = requestEnvelope.context.System.apiAccessToken;
  // build the progressive response directive
  const directive = {
    header: {
      requestId,
    },
    directive: {
      type: "VoicePlayer.Speak",
      speech:
        `<speak><audio src=${process.env.waitsound} /></speak>`,
    },
  };
  // send directive
  return directiveServiceClient.enqueue(directive, endpoint, token);
}

Sentiment Scales

The four APIs return measurements on different scales, which I had to normalize somewhat to offer results. I chose to report a sentiment and an absolute magnitude for that sentiment on a scale of 0-100 for each of the APIs.

• Amazon Comprehend returns a value of 0-100 for each of POSITIVE, NEGATIVE, NEUTRAL and MIXED
• Microsoft Azure returns a value of 0-1 for each of POSITIVE, NEGATIVE and NEUTRAL
• Expert.ai returns a value of 0-100 for POSITIVITY and NEGATIVITY (and I interpolate a 0 value as NEUTRAL)
• ParallelDots returns a value of 0-1 for each of POSITIVE, NEGATIVE and NEUTRAL

The Outlier

I haven’t done statistical analysis on this, but anecdotally I am seeing Expert.ai needing some coaxing to have high magnitude scores on some utterances. For instance, the utterance “Happy happy joy joy” maxes Comprehend, Azure and Expert.ai at 100, with ParallelDots lagging just a bit at 92.

However the utterance “I am very happy today” scores high with Comprehend (100), ParallelDots (97) and Azure (98) but straggles in with Expert.ai (33). Go figure.

Try It