Mastering the Arcane Art of JavaScriptmancy for C# Developers - Chapter 8: Enumerables, Iterables, Iterators and Generators

A young wizard practices the fine wizardring arts with mixed success throwing fireballs at her master.

An in-depth look into JavaScript enumerables and ES6/ES2015 iterables, iterators and generators

The Mastering the Arcane Art of JavaScript-mancy series are my humble attempt at bringing my love for JavaScript to all other C# developers that haven’t yet discovered how awesome this language and its whole ecosystem are. These articles are excerpts of the super duper awesome JavaScript-Mancy book a compendium of all things JavaScript for C# developers.

In the last chapter of these series we discussed how you can use LINQ-like functionality in JavaScript with different approaches like the array.prototype methods, ES2015 generators, third-party libraries and comprehensions. I showed you how you can use generators to add deferred evaluation to operating on collections but I never got around explaining what generators actually are. I mean to fix that oversight right now and right here!

Let’s begin at the beginning…

Once Upon a Time There Were Enumerables

From the olden times JavaScript has had a concept called Enumerability, the concept of being able to enumerate the properties within an object by using the for/in loop.

Ideally you would be able to use the for/in loop to traverse the properties of an object, or the values of an array (via its indexes):

// From the olden times JavaScript has had data structures that are enumerable
// namely arrays and objects

console.log('*** Enumerating items within an array ***')
console.log(
  "let's say I have an array of ingredients for my potions and spells"
)
var ingredients = ['mandragora', 'troll nail', 'dragon scale', "gorgon's eye"]
console.log(ingredients)

// you can enumerate all the ingredients in the array
// by using a for/in loop
console.log(
  '\nyou can enumerate all the ingredients within the array with a for/in loop:'
)
for (idx in ingredients) {
  console.log('Item at index ' + idx + ': ', ingredients[idx])
}
// => Item at index 0: mandragora
// => Item at index 1: troll nail
// => etc...

// note how the for/in loop let's you enumerate through the index of an array
// and not through the actual values

Well, no problems so far, everything seems to work as you would expect aside for the teeny tiny fact that you can only iterate over the indexes of an array and not the items themselves.

However, you may have heard that using the for/in loop is not a good idea. Let’s find out why. We’ll start by adding a new property to our ingredients array.

// you may have heard that the for/in loop is not very save to enumerate through
// items of an array. That's because if you were to add a property to the array
// like this:
ingredients.maxNumberOfIngredients = 666

// and you try to enumerate it:
console.log(
  '\nIf we enumerate after adding a property to the array object itself:'
)
for (idx in ingredients) {
  console.log('Item at index ' + idx + ':', ingredients[idx])
}
// => Item at index 0: mandragora
// => ...
// => Item at index maxNumberOfIngredients: 666
// WAAAAT
console.log('WAAAAT')
console.log(
  'in addition to enumerate through the different items \nwithin an array we also iterated over the new added property!!!'
)
// so in addition to enumerate through the different items within an array, we
// also iterated over the new added property!!!!

Ok, red flag number #1: if you augment the array willy nilly with a new property the for/in loop traverses it. That’s not what we wanted, is it?

If we switch the context to vanilla JavaScript objects that behavior makes more sense. You enumerate through the properties of an object using the for/in loop, and if you augment an object with new properties, you’d like to be able to enumerate over those new properties as well:

// Before we address **that issue**, let's take a look at enumerating objects
// we can enumerate properties within an object in a similar fashion that we
// we enumerate arrays
console.log('\n\n*** Enumerating properties within objects ***')
console.log("let's say I have a spell of destruction object:")
var spellOfDestruction = {
  name: 'spell Of destruction',
  damage: 9999,
  duration: 'instant',
  toString: function() {
    return this.name
  },
}
console.log(spellOfDestruction)
// and we'll enumerate its properties with the for/in loop

console.log(
  '\nyou can enumerate all the properties within an object with a for/in loop:'
)
for (prop in spellOfDestruction) {
  console.log('Value of property ' + prop + ': ', spellOfDestruction[prop])
}
// => Value of property name
// => spell of destruction
// => etc...

// and as you would expect, if we add new properties to this object
// they'll be enumerated as well
console.log('\nif we add new properties theyll be enumerated as well:')
spellOfDestruction.minMagicLevel = 99
spellOfDestruction.mana = 1000
for (prop in spellOfDestruction) {
  console.log('Value of property ' + prop + ':', spellOfDestruction[prop])
}
// => Value of property name: spell of destruction
// => ...
// => Value of property minMagicLevel: 99
// => etc...

Ok, so how does this all work? If it is based on properties of objects, as it appears from the examples, Why do we enumerate through indexes of an array but not other methods like push, pop, forEach, etc?

It turns out that properties within objects in JavaScript can be configured to a certain extent and one of these configurations, the enumerable property descriptor, allows us to define whether or not a property is enumerable via the for/in loop. When we use an object literal or when we augment an object with properties using the dot notation the properties are defined as enumerable by default.

In order to specify a property as not-enumerable we need to drop down one layer of abstraction and use low-level methods of Object.prototype: defineProperty and defineProperties:

// It turns out that we can define properties as enumerable or non enumerable
// when we create and augment objects.
// By default, a new property will be enumerable:
console.log('\n\nIt turns out that properties are enumerable by default')
console.log('The ingredients.maxNumberOfIngredients is enumerable:')
// we can verify it with the Object.propertyIsEnumerable method
console.log(ingredients.propertyIsEnumerable('maxNumberOfIngredients'))
// => true
console.log('If you check one of the array.prototype methods like push:')
console.log(ingredients.propertyIsEnumerable('push'))

// But you can use low-level object definition functions within Object.prototype
// to alter that behavior
Object.defineProperty(
  /* object */ ingredients,
  /* property */ 'maxNumberOfIngredients',
  /* descriptor */ { enumerable: false }
)
// and you try to enumerate it:
console.log(
  '\nIf we enumerate after modifying the property to not be enumerable:'
)
for (idx in ingredients) {
  console.log('Item at index ' + idx + ':', ingredients[idx])
}
// => Item at index 0: mandragora
// => ...
// => Item at index 3: gorgon's eye
// Fixed!!

// The Object.prototype.keys method let's you check which properties within
// an object are enumerable
console.log(
  'These are the enumerable properties of the ingredients array: ',
  Object.keys(ingredients)
)
// => These are the.... array: [0, 1, 2, 3]

// you can also use define property to create new properties
var elementsOfMagic = ['wind', 'earth', 'fire', 'water']
Object.defineProperty(elementsOfMagic, 'secretElements', {
  enumerable: false,
  value: ['aether', 'iron'],
})

By defining properties as non-enumerable we can somewhat guarantee that you can use the for/in loop to enumerate items within an array or properties within an object.

Another less apparent problem with enumerables and the for/in loop happens when we augment an object’s prototype. For instance, if we extend Array.prototype with a new method that is enumerable, we will break the for/in behavior for all arrays in the current environment:

// And when you thought this was all of it, JavaScript prototypical
// inheritance is thrown into the mix
console.log('\n\n*** Enumerating properties within objects and prototypes ***')

// what happens if we augment the Array.prototype willy nilly?
Array.prototype.countMissisipi = function() {
  return this.length + ' Mississipi!'
}
// Now we have broken the for/in loop for all arrays in the current environment
console.log(
  '\nIf we enumerate the ingredients array after adding a method to its prototype:'
)
for (idx in ingredients) {
  console.log('Item at index ' + idx + ':', ingredients[idx])
}
// => Item at index countMissisipi: .....
// => BOOOOM!
console.log('BOOOOOM!')

// we can still fix it with the defineProperty method
Object.defineProperty(Array.prototype, 'countMissisipi', {
  enumerable: false,
})

So as you can see from these examples enumerability and the for/in loop are not the most reliable way to traverse an array or an object. The for/in loop is still interesting for stuff like reflection or object introspection, but for iterating over arrays you’ll problaby need to limit yourself to the Array.prototype.forEach, other array FP functions or a vanilla for loop.

Well… that was until ES2015 (ES6) came out…

ES2015 Brings Real Iterables and Iterators

ES2015 comes with two new data structures: maps and sets (as well as their weak versions). And suddenly it becomes even more apparent that we need a reliable and straightforward way to traverse these data structures and our old friend, the array. Luckily for us, ES2015 not only comes with these new data structures but also solidifies the concept of iterating over a collection with iterables, iterators and the new for/of loop. (Which are analog to the C# IEnumerable and IEnumerator interfaces)

Any JavaScript object becomes iterable if it implements the iterable interface (like implementing IEnumerable), not in the sense that we implement an interface in C# but in the more native to JavaScript duck-typing sense. That is, if an object has a “GetIterator” method that returns a valid iterator then the object is iterable. And I say “GetIterator” between quotes because the name of the method itself is defined by a constant: Symbol.iterator. If we take a look at an array, map and set in ES2015 (ES6) we can verify that indeed they are iterable:

// run on Babel.js (https://babeljs.io/repl/)
// Let's take a look at iterables and iterators in ES2015/ES6

// Any object that implements the "GetIterator" method will be iterable
// and as such we will be able to traverse it using a for/of loop
// The actual name of the "GetIterator" method is the constant Symbol.iterator
console.log('*** Arrays as iterables ***')
let inventory = ['sword', 'shield', '100 coppers']

console.log('hello')
console.log(Symbol.iterator.toString())
// array default iterator
console.log(inventory[Symbol.iterator])
// => function ArrayValues()

console.log('\n\n> see inventory')
console.log('Traveller, you have these items in your inventory:')
for (let item of inventory) {
  console.log(item)
}
// =>  > see inventory
// => Traveller, you have these items in your inventory:
// => sword
// => shield
// => 100 coppers

console.log('\n\n*** Maps as iterables ***')
let shop = new Map([
  ['swords', ['sword of truth', 'copper sword']],
  ['shields', ['silver shield', 'tower shield']],
])
// map default iterator
console.log(shop[Symbol.iterator])
// => function entries()

console.log('\n\n> look at shop')
console.log('The shop is scarcely lit but you see...')
for (let [itemName, items] of shop) {
  console.log(`...these many ${itemName}: ${items}`)
}
// => > look at shop
// => The shop is scarcely lit but you see...
// => ...these many swords: sword of truth,copper sword
// => ...these many shields: silver shield,tower shield

console.log('\n\n*** Sets as iterables ***')
let exits = new Set(['outside', 'wooden door', 'stairs going up'])
// Set defaul iterator
console.log(exits[Symbol.iterator])
// => function values()

console.log('\n\n> exits')
console.log('You can see the following exits:')
for (let exit of exits) {
  console.log(`${exit}`)
}
// => > exists
// => You can see the following exists:
// => outside
// => wooden door
// => stairs going up

Vanilla objects, on the other hand, are not iterable:

console.log('\n\n*** Objects as iterables ***')
let swordOfTruth = {
  description: `Behold the Mighty Sword Of Truth! ....`,
  damage: 100,
  magicDamage: 1000,
  weight: 30,
  state: 'well kept',
  toString: () => this.description,
}
// Objects have no default iterator!!
// and therefore are not iterable by default
console.log(swordOfTruth[Symbol.iterator])
// => undefined

try {
  for (let prop of swordOfTruth) {
    console.log(prop)
  }
} catch (ex) {
  console.error(`*Error*: ${ex.message}`)
  // => swordOfTruth[Symbol.iterator] is not a function
}

As you can appreciate from the examples above, the Symbol.iterator property within an iterable object will return its default iterator. In the case of arrays and sets it is an iterator over the values while in the case of maps it is a key/value pair iterator. In addition to the default iterators, maps also provide specific iterators for keys and values:

console.log('\n\n***Map non-default iterators***')

// maps also offer other iterators over keys and values
console.log(shop.keys)
// => function keys()
console.log(shop.values)
// => function values()

console.log('the shop has this types or articles:')
for (let type of shop.keys()) {
  console.log(type)
}
// => the shop has these types of articles:
// => swords
// => shields

In summary, ES2015 brings forth the concept of iterability over elements within a collection and leverages it with all its data structures old and new: arrays, maps and sets. An iterable object is that which implements a GetIterator method located in its Symbol.iterator property. Vanilla JavaScript objects are not iterable because they don’t have an iterator (they don’t implement the necessary interface to be iterable).

How can we make any arbitrary object iterable? Let’s dive a little deeper into iterables and find out.

ES2015 Iterables

To make any arbitrary object iterable we need to implement a GetIterator method. Let’s say we have the majestic swordOfTruth from previous examples:

let swordOfTruth = {
  description: `Behold the Mighty Sword Of Truth! ....`,
  damage: 100,
  magicDamage: 1000,
  weight: 30,
  state: 'well kept',
  toString: () => this.description,
}

It is a vanilla JavaScript object and thus not iterable:

try {
  for (let prop of swordOfTruth) {
    console.log(prop)
  }
} catch (ex) {
  console.error(`*Error*: ${ex.message}`)
  // => swordOfTruth[Symbol.iterator] is not a function
}

If we implement the getObjectPropertiesIterator method to, let’s say, iterate over the object properties and set it as its Symbol.iterator property, the object becomes iterable:

console.log(
  "\n\n Let's implement a GetIterator method then to traverse the object properties"
)

function getObjectPropertiesIterator() {
  // Note that I'm piggy-backing on an array iterator here
  // the Object.getPropertyNames method returns an array with the properties of the object
  // and I am reusing the iterator of that array to make any object iterable
  let objectProperties = Object.getOwnPropertyNames(this)
  return objectProperties[Symbol.iterator]()
}
swordOfTruth[Symbol.iterator] = getObjectPropertiesIterator

// now I should be able to iterate over the properties of the swordOfTruth
// with a for/of loop
console.log('This is the Sword of Truth!')
for (let prop of swordOfTruth) {
  console.log(`${prop}: ${swordOfTruth[prop]}`)
}
// => description: Behold the Mighty Sword of Truth!
// => damage: 100
// ...
// => toString: function(){...}

The cool and super interesting thing about this new abstraction/generalization of iterability in JavaScript is that it enables us to reuse iterators in a very straightforward fashion. That is, you write a function to create an iterator for an object and then you can reuse it in other parts of you system to you heart’s content. For instance, we could make all objects iterable by default:

// http://jsfiddle.net/vintharas/qunycLhb/ Test it on jsFiddle
Object.prototype[Symbol.iterator] = getObjectPropertiesIterator

// and if we create a new arbitrary object
var stoneOfTears = {
  description: 'A common blue stone with the shape of a diamond',
}
for (let prop of stoneOfTears) {
  console.log(`${prop}: ${stoneOfTears[prop]}`)
}
// => description: a common blue stone with the shape of a diamond

I don’t know if you noticed but, in this previous example, I took a little shortcut and reused an array iterator by extracting it from an array of properties of an object. But we are not limited to reuse built-in iterators, we can even implement our very own iterators if we need that additional degree of freedom and control. Let’s see how.

ES2015 Iterators

If we take a look at a built-in iterator we will be able to appreciate that it has a single next method:

// http://jsfiddle.net/vintharas/jd9vs611/ Test in on jsFiddle
// run on Babel.js (https://babeljs.io/repl/)
// Let's take a look at iterables and iterators in ES2015/ES6

console.log('\n\n*** Taking a deeper look at iterators ***')
console.log(`\n If we take a look at a built-in iterator
we can see that it has a method called next:`)

let minions = ['troll', 'orc', 'goblin']
let it = minions[Symbol.iterator]()
console.log(it.next)
// => function next

If you call this next method you’ll get the next element in the array:

// http://jsfiddle.net/vintharas/jd9vs611/ Test in on jsFiddle
console.log(`\n what happens if we call it?`)
let nextItem = it.next()
console.log(nextItem)
// => {"value": "troll", "done": false}

console.log(`\n when we call the next method we get an object with the next value in the collection
and an status that indicates that there are still elements to traverse within the array`)
console.log(it.next())
// => {"value": "orc", "done": false}
console.log(it.next())
// => {"value": "goblin", "done": false}
console.log(it.next())
// => {"done": true}

So, How do we implement a custom iterator? By implementing a next method that returns the same information as the built-in iterator above and thus fulfills the iterator protocol/interface/contract expected of an iterator:

// http://jsfiddle.net/vintharas/jd9vs611/ Test in on jsFiddle
// The class below represents an iterator that iterates over all non-function properties
// of an object
class ObjectPropertiesIterator {
  constructor(obj) {
    this.properties = Object.getOwnPropertyNames(obj).filter(
      p => typeof obj[p] !== 'function'
    )
    this.index = 0
  }
  next() {
    if (this.index === this.properties.length) return { done: true }
    else {
      this.index++
      return { value: this.properties[this.index - 1], done: false }
    }
  }
}

// and an equivalent implementation in ES5
function ObjectPropertiesIterator(obj) {
  this.properties = Object.getOwnPropertyNames(obj).filter(
    p => typeof obj[p] !== 'function'
  )
  // note that we would need to add a polyfill for the filter
  // array prototype function that is not present in ES5
  this.index = 0
}

ObjectPropertiesIterator.prototype.next = function next() {
  if (this.index === this.properties.length) return { done: true }
  else {
    this.index++
    return { value: this.properties[this.index - 1], done: false }
  }
}

Once we have defined a suitable iterator we can set it as the default iterator in our vanilla JavaScript object:

// http://jsfiddle.net/vintharas/jd9vs611/ Test in on jsFiddle
let swordOfTruth = {
  description: `Behold the Mighty Sword Of Truth! ....`,
  damage: 100,
  magicDamage: 1000,
  weight: 30,
  state: 'well kept',
  toString: () => this.description,
  [Symbol.iterator]: getObjectPropertiesIterator,
}
function getObjectPropertiesIterator() {
  // Now we create our very own iterator
  return new ObjectPropertiesIterator(this)
}

console.log('The sword of truth!')
for (let prop of swordOfTruth) {
  console.log(`${prop}: ${swordOfTruth[prop]}`)
}

And again, as we saw in the previous section, by having an explicit representation for an iterator, we can reuse it in other parts of our application as needed.

A Better Way to Create Iterators: ES2015 Generators

And finally we get to the magic generators. From what you saw in the last section, creating new iterators can be a bit of a hassle: you need to create a class with a next method, actively handle and keep track of the current state of the iterator, compose the result of the next method… Wouldn’t it be nice if there was a quicker and sweeter way to create an iterator? Well, that’s what generators are for. A generator is not other thing that a factory of iterators. Generators are easily spotted by the * after the function keyword and the yield keyword:

// http://jsfiddle.net/vintharas/o1bgt628/ Test it on jsFiddle
// run on Babel.js (https://babeljs.io/repl/)
// Let's take a look at generators in ES2015/ES6

console.log(
  '\n\n*** Needing to create an iterator? Generators to the rescue!! ***'
)

// the asterisk and the yield keyword mark the function below as a generator
function* getObjectPropertiesIterator() {
  for (let prop in this) {
    if (typeof this[prop] !== 'function') {
      // the yield keyword returns the property and
      // stops the execution of this function at this place
      yield prop
    }
  }
}
console.log(`${getObjectPropertiesIterator()}`)
// => [object Generator]

let swordOfTruth = {
  description: `Behold the Mighty Sword Of Truth! ....`,
  damage: 100,
  magicDamage: 1000,
  weight: 30,
  state: 'well kept',
  toString: () => this.description,
  [Symbol.iterator]: getObjectPropertiesIterator,
}

console.log('The sword of truth!')
for (let prop of swordOfTruth) {
  console.log(`${prop}: ${swordOfTruth[prop]}`)
}

And Some Weird Things You Can Do With Generators

Because the result on calling next on an iterator created by a generator is generated on demand, generators can be used to create infinite sequences in a very cheap and straightforward manner. Consider the following generator used to spawn an infinite sequence of devilish imps in a game of increasing difficulty for our hypothetical hero of yore:

// http://jsfiddle.net/vintharas/wkb96j00/ Test it on jsFiddle
// run on Babel.js (https://babeljs.io/repl/)
// Let's take a look at generators in ES2015/ES6

console.log('*** Using Generators to create infinite sequences ***')

// imps!!
function* impsExclamationExclamation() {
  let fn1 = 1,
    fn2 = 1
  while (true) {
    let current = fn2
    fn2 = fn1
    fn1 = fn1 + current
    yield `spawned ${current} imps!`
  }
}

var imps = impsExclamationExclamation() // imps!!!!
console.log(imps.next().value)
// =>  spawned 1 imps!
console.log(imps.next().value)
// => spawned 1 imps!
console.log(imps.next().value)
// => spawned 2 imps!
console.log(imps.next().value)
// => spawned 3 imps!
console.log(imps.next().value)
// => spawned 5 imps!
console.log(imps.next().value)
// => spawned 8 imps!

console.log('oh hero, now you are in a pickle!')

Additionally, you can configure generators via its arguments:

// http://jsfiddle.net/vintharas/wkb96j00/ Test it on jsFiddle
// imps!!
function* impsExclamationExclamationWithOffset(minimumNumberOfImps) {
  let fn1 = 1,
    fn2 = 1
  while (true) {
    let current = fn2
    fn2 = fn1
    fn1 = fn1 + current
    if (current >= minimumNumberOfImps) {
      yield `spawned ${current} imps!`
    }
  }
}

var impsWithOffset = impsExclamationExclamationWithOffset(
  /*minimumNumberOfImps*/ 50
)
console.log(impsWithOffset.next().value)
// =>  spawned 55 imps!
console.log(impsWithOffset.next().value)
// => spawned 89 imps!
console.log(impsWithOffset.next().value)
// => spawned 144 imps!
console.log(impsWithOffset.next().value)
// => spawned 233 imps!
console.log(impsWithOffset.next().value)
// => spawned 377 imps!
console.log(impsWithOffset.next().value)
// => spawned 610 imps!

console.log('oh hero, now you are in a pickle!')

And affect the iteration of the sequence by passing arguments to the next method:

// http://jsfiddle.net/vintharas/wkb96j00/ Test it on jsFiddle
// imps!!
function* impsExclamationExclamationWithRespit() {
  let fn1 = 1,
    fn2 = 1
  while (true) {
    let current = fn2
    fn2 = fn1
    fn1 = fn1 + current
    // whatever we pass to the next function is
    // stored in the breathe variable
    var breathe = yield `spawned ${current} imps!`
    if (breathe) {
      fn1 = 1
      fn2 = 1
    }
  }
}

var impsWithRespit = impsExclamationExclamationWithRespit()
console.log(impsWithRespit.next().value)
// =>  spawned 55 imps!
console.log(impsWithRespit.next().value)
// => spawned 89 imps!
console.log(impsWithRespit.next().value)
// => spawned 144 imps!
console.log(impsWithRespit.next().value)
// => spawned 233 imps!
console.log(impsWithRespit.next().value)
// => spawned 377 imps!
console.log(impsWithRespit.next(/* breathe */ true).value)
// => spawned 1 imps!

console.log('fiuuuuu')

Concluding

In the last chapter of the series I brushed over generators and started using them without preamble nor clarification while talking about linq. I hope that with this article I have done generators justice and improved your understanding of iterators and generators by having brought them into context.

In summary we saw:

How previous versions of JavaScript had the concept of enumerability to traverse object properties and array indexes using the for/in loop.
The limitations and unreliability of using enumerables and the for/in loop and the side-effects of for/in when augmenting prototypes
How ES6 brings two new data structures: maps and sets and the new concept of iterability and iterators that allows us to safely iterate over items of an array/set or key/value pairs of a map.
How to make any object iterable and how to implement custom iterators
How ES6 generators simplify the creation of iterators and how to use generators to generate infinite sequences of devilish imps

Hope you have enjoyed the article! Up next, more functional programming, function composition and building a vocabulary of beautiful abstractions. Take care!

Interested in Learning More JavaScript? Buy the Book!

Are you a C# Developer interested in learning JavaScript? Then take a look at the JavaScript-mancy book, a complete compendium of JavaScript for C# developers.

Have a great week ahead!! :)

Do you want to Learn Some More About Iterators and Generators?

Written by Jaime González García , dad, husband, software engineer, ux designer, amateur pixel artist, tinkerer and master of the arcane arts. You can also find him on Twitter jabbering about random stuff.Jaime González García

barbarianmeetscoding

WebDev, UX & a Pinch of Fantasy

Mastering the Arcane Art of JavaScriptmancy for C# Developers - Chapter 8: Enumerables, Iterables, Iterators and Generators

Once Upon a Time There Were Enumerables

ES2015 Brings Real Iterables and Iterators

ES2015 Iterables

ES2015 Iterators

A Better Way to Create Iterators: ES2015 Generators

And Some Weird Things You Can Do With Generators

Concluding

Interested in Learning More JavaScript? Buy the Book!

More Articles In This Series

Do you want to Learn Some More About Iterators and Generators?