In the last article of the series we started building a game Rx.js and Web Speech just like I did at the JavaScript hackathon at Active Dublin 2016.

The game is a variation of the popular typing game where words fall off the screen and you need to type them before they get to you. The slight variation consists in changing the game mechanics from typing to speaking. In our game, letters will fall off the screen and the player will need to say a word starting with that letter to earn points, destroy the letter and avoid certain annihilation.

At the end of the last article we had completed the beginnings of a game with a beautiful black background and some letters falling off the screen. Let’s continue building our game by adding some player interaction through the Web Speech API!

The Web Speech API

The Web Speech API let’s you incorporate voice interaction right into your web applications. You can either take advantage of the Speech Synthesis API to make your computer talk to your users or the Speech Recognition API to teach your programs to hear and recognize your users enabling new and cool interactions.

Prior to this hackathon I had experimented a little with the Speech Synthesis API but I had done nothing at all with Speech Recognition. Because of that and the super-uber-mega-tight schedule of the hackathon I chose the easy way forward and used the annyang library to cover my speech recognition needs.

Using Annyang to Gather Player Input

The annyang library offers a super simple API to recognize a person’s speech. You define commands using text patterns and tie these patterns to callback functions. For instance, if you want your application to do something when the user says cucumber you define a command like this:

var commands = {
  cucumber: _ => console.log('haha he said cucumber'),
}

If you want to extract part of the user’s speech you can do it as well. Imagine that you want to order a pizza with arbitrary condiments, you could type the following:

var commands = {
  'order pizza with *ingredients': ingredients => {
    console.log(`the user order a pizza with ${ingredients}`)
    magicPizza.createOrder(ingredients)
  },
}

Where the *ingredients part of the command would be recognized by the Speech Recognition API and passed as an argument to your function.

In our particular case we want to be able to understand words that we can later match with the letters that are falling down the screen. For instance, if an l appears on the screen, the player could start shouting lentils! lenses! lotion! to eliminate it and earn… let’s say… 100 points.

We can set up a simple command to recognize anything spoken by the player like this:

var commands = {
  '*word': word => {
    console.log('user said: ', word)
    // TODO: handle player word in some way
  },
}

And you hook up the commands and get the speech recognition started with these two simple lines:

// Add our commands to annyang
annyang.addCommands(commands)

// Start listening.
annyang.start()

But how can we incorporate this into the game? We need to somehow wrap this user input in an observable. There are different ways to achieve this but the one I went for was using a Subject.

A Subject is a special Rx.js object that doubles as an observer and an observable. When it plays the role of an observer it can subscribe to other observables or be pushed new values like the words pronounced by a player. And as an observable it can emit these values and you can combine it with other observables or subscribe to it.

You can create a subject via the Rx.Subject function:

var userWords$ = new Rx.Subject()

Once created we can update commands to push values into the subject (taking advantage of its observer facet):

/* VOICE STUFF */
var commands = {
  '*word': word => {
    console.log('user said: ', word)
    userWords$.onNext(word)
  },
}

Now we need to include these words into the game loop. Each time a player says a word, we want to compare that word with the letters that are falling off the screen and if there’s a match, we want to eliminate that letter and increase the user’s score.

We can create a new observable newWordAndGame$ to represent the status of the game every time a user says a new distinct word:

var newWordAndGame$ = game$
  .combineLatest(userWords$, (gameObjects, userWord) => {
    return { gameObjects, userWord }
  })
  .distinctUntilChanged(o => o.userWord)

This new observable combines the userWords$ and the game$ observables. If you don’t remember how the game$ observable looked like here you have a refresher:

var game$ = gameLoop$.combineLatest(background$, letters$, (_, b, l) => {
  var gameObjects = [b, ...l]
  return gameObjects
})

The distinctUntilChanged operator ensures that new values are only emitted when the player says a new word. We can subscribe to the resulting observable and perform our word matching to see whether or not we can eliminate a letter from the screen and give the player some points:

newWordAndGame$.subscribe(gameObjectsAndWords => {
  matchWords(gameObjectsAndWords)
})

The matchWords method will implement the matching logic. It could look like this:

function matchWords({ gameObjects, userWord }) {
  console.log(
    `matching ${userWord} with ${gameObjects.letters.map(l => l.letter)}`
  )
  let letterMatched = findLetterMatchingWord(gameObjects, userWord)
  if (letterMatched) {
    console.log('matched word')
    removeLetter(letterMatched, gameObjects)
    updateScore(100)
  }
}

Where findLetterMatchingWord would retrieve the first letter of those in the screen that is the starting letter of the word said by the player (for instance, if the player says violoncello that would be the first v within our letters array):

function findLetterMatchingWord(letters, word) {
  return gameObjects.letters.find(l =>
    userWord.toLowerCase().startsWith(l.letter)
  )
}

The removeLetter function would remove the matched letter from our collection of letter game objects:

function removeLetter(letter, letters) {
  let idx = letters.indexOf(letter)
  letters.splice(idx, 1)
}

And the updateScore function would update the score. We will leave it empty for the time being since we don’t have any score yet in our game:

function updateScore(points) {
  // update score for great justice
}

Since we need a convenient way to access the letters within our game objects (The gameObjects.letters property doesn’t exist yet at this stage) we update the game$ observable to expose them:

var game$ = gameLoop$.combineLatest(background$, letters$, (_, b, l) => {
  var gameObjects = { letters: l, all: [b, ...l] }
  return gameObjects
})

This means that we’ll have to make a small update in our subscribe method to conform to the new API:

game$.subscribe(gameObjects => {
  gameObjects.all.forEach(g => {
    g.update()
    g.draw()
  })
})

If you run the game now you should be able to interact with it as a player. As soon as you open the game in your browser you’ll be prompted to Allow or Block access to the microphone. Allow it and when you see a letter falling down the screen say a word that starts with that letter. The letter should disappear.

If that doesn’t happen check the console and see if there are any errors. You can also add a visual help to see the words that you utter directly within the game. Add the following element to your html markup:

    <ul class="matches">
    </ul>

And update the userWordsAndGame$.subscribe to append the uttered words to the matches list:

userWordsAndGame$.subscribe(gou => {
  matchWords(gou)
  appendToDom(gou.userWord)
})

function appendToDom(word) {
  var ul = document.querySelector('.matches')
  var li = document.createElement('li')
  li.textContent = word
  ul.appendChild(li)
}

Now you should be able to verify the words or sentences recognized by the Web Speech API more readily as they’ll appear on the screen beside the canvas.

Why didn’t We Include userWords$ in the Original game$ Observable?

Now you may be thinking… Why didn’t we make the userWords$ be a part of the game$ observable like we did with the previous ones?

Well if we had done so it would’ve been more difficult to know when a user had uttered a word. Since combineLatest emits a value any time a combined observable emits a value we would have had new values being pushed on every game tick, every letter, every word uttered, etc…

Separating word handling into a different observable makes it easier to compose it with the distinctUntilChanged operator and unequivocally know when the user utters a new word.

Adding Scores

Let’s continue by adding a way to manage scores. We need to be able to both compute scores and have a way to display them within the game.

We will create a new game object to represent the idea of a Score:

class Score extends GameObject {
  constructor(value) {
    super(50, 50, 0)
    this.value = value
  }
  draw() {
    var canvas = document.getElementById('game')
    var ctx = canvas.getContext('2d')
    ctx.fillStyle = 'yellow'
    ctx.fillText(this.value, this.x, this.y)
  }
  add(score) {
    this.value += score.value
    return this
  }
}

And now that we have a way to represent scores we need to include it in the game. Just like we did with the player words, we will create a subject so that we can push in new scores as the user words match the falling letters:

var ScoreSubject = new Rx.Subject()

We can now implement the updateScore method from before:

function updateScore(points) {
  ScoreSubject.onNext(score)
}

And we will add it to our game loop so it can be drawn in the screen. In order to do that we create the score$ observable that takes the score values and maps then into Score game objects:

var score$ = ScoreSubject.startWith(0)
  .map(v => new Score(v))
  .scan((prev, cur) => prev.add(cur), new Score(0))

We use the scan operator much in the same way that we did in the first part of the series but in this case to accumulate scores.

The next step is to include the score in our game loop:

var game$ = gameLoop$.combineLatest(
  background$,
  letters$,
  score$,
  (_, b, l, s) => {
    var gameObjects = { letters: l, all: [b, s, ...l] }
    return gameObjects
  }
)

And now if you start your game, you’ll be able to see a 0 standing in the top left-most corner. If you say a word that matches any of the letters in the screen you should be able to see how your score increases. So say a word and your score goes up… 200!???

Wait… There’s Something Weird Happening? Every time I say a word I get 200 points! Wat!??

So, you wanted the player’s score to go up 100 points whenever he uttered the right word, but somehow, any time you say a word that eliminates a letter you are getting 200 points. Moreover, if you look closely it looks like there’s 2 words falling down every 2 seconds instead of only one. What’s happening here?

Well what is happening is that we have created a cold observable with two subscribers when what we really wanted was a hot observable.

Cold? Hot? Waaaaat!?

Cold and Hot Observables

The type of observables we have been creating in this game are what are known in Rx.js jargon as cold observables. Cold observables only start running when someone subscribes to them. That is, they are inert, dead, they don’t do anything at all until someone calls suscribe on them.

Moreover, the values pushed by a cold observable are not shared accross subscribers. If you, for instance, have a cold observable that produces a sequence of integers with two different subscribers, the whole sequence of integers will be pushed to both subscribers regardless of when they subscribed to the observable:

// push a total of 3 numbers every 500ms
var cold$ = Rx.Observable.interval(500).take(3)

cold$.subscribe(value => console.log(`cold 1: ${value}`))

// subscribe 700 ms after
setTimeout(_ => cold$.subscribe(value => console.log(`cold 2: ${value}`)), 700)

// => cold 1: 0
// => cold 1: 1
// => cold 2: 0
// => cold 1: 2
// => cold 2: 1
// => cold 2: 2

// both subscribers get all values
// even when subscriber 2 was late to the party

You can find this example in JsFiddle if you’re curious and want to tinker yourself.

If now that you know about cold observables you take a second look at our game you’ll realize that we have two subscribers that are subscribed to the game$ observable. As a result of that, we have double the letters being created and double the scores being accumulated.

What we want is to use the same sequence regardless of the number of subscribers. What we want is a Hot observable. Hot observables, in opposition to cold ones, emit values even before they have active subscribers. You can convert cold into hot observables in different ways and, in this case, we will use the share operator.

We can modify the previous example to illustrate its use:

// push a total of 3 numbers every 500ms
var hot$ = Rx.Observable.interval(500)
  .take(3)
  .share()

hot$.subscribe(value => console.log(`hot 1: ${value}`))

// subscribe 700 milliseconds after
setTimeout(_ => hot$.subscribe(value => console.log(`hot 2: ${value}`)), 700)

// => hot 1: 0
// => hot 1: 1
// => hot 2: 1
// => hot 1: 2
// => hot 2: 2

// values are pushed regardless the number of subscribers
// the value 0 had already being emitted when
// the subscriber 2 subscribed and therefore
// it missed it

So! Just like in this example, we can use the share operator in our game$ observable to fix the problem:

var game$ = gameLoop$
  .combineLatest(background$, letters$, score$, (_, b, l, s) => {
    var gameObjects = { letters: l, all: [b, s, ...l] }
    return gameObjects
  })
  .share()

Now you should be able to go back to game and see how everything works as it is supposed to. Yeeey!!!

Concluding

Great! Now we have a game that you can play! Cool right?

We have built on top of what we had done in the previous article adding player interaction through voice commands and we are able to affect the state of the game by removing letters and increasing the player score. We’ve also learnt some new Rx.js concepts like subjects and hot and cold observables.

There’s still some stuff left though, for one, the game never finishes. It doesn’t matter how many letters fall off the screen you can continue playing, and that sucks, because one of the fun and exciting things about games is that you can lose.

So in the next article of the series we will focus on a way to represent player lives and add some thrill to the game. We will wrap the series by adding some graphics, doing some refactoring and reflecting over the whole business.

Would you Like to Learn More About The Web Speech API and Rx.js?

Take a look at these great articles:

• Introduction to the Web Speech API at Google Developers
• Web Speech API at MDN
• More on Subjects
• More on Hot vs Cold Observables