Did you ever get that feeling when a random song pops into your brain and you can’t get rid of it? Well, that happened to me recently and I couldn’t even remember the title of the damn song! In this talk, I want to share with you the story of how I was able to recover the details of the song by navigating some music-related APIs using JavaScript, Node.js and the magic of async iterators!

1. Finding a lost song
with Node.js & async iterators
Luciano Mammino ( )
@loige
Dublin, Node.js Meetup
2021-04-29
loige.link/async-it
1

2. loige.link/async-it
Get these slides!
loige 2

3. Photo by on
Darius Bashar Unsplash
A random song you haven't listened to
in years pops into your head...
3

4. It doesn't matter what you do all day...
It keeps coming back to you!
Photo by on
Attentie Attentie Unsplash 4

5. And now you want to listen to it!
Photo by on
Volodymyr Hryshchenko Unsplash 5

6. But, what if you can't remember
the title or the author?!
Photo by on
Tachina Lee Unsplash 6

7. THERE MUST BE A WAY TO REMEMBER!
Photo by on
Marius Niveri Unsplash 7

8. Today, I'll tell you how I solved this problem using
- Last.fm API
- Node.js
- Async Iterators
Photo by on
Quinton Coetzee Unsplash 8

9. Let me introduce myself first...
9

10. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
9

11. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin )
9

12. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin )
nodejsdp.link
Co-Author of Node.js Design Patterns 👉
9

13. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin )
nodejsdp.link
Co-Author of Node.js Design Patterns 👉
20% eBook discount on Packt
20NODEDUBLIN
9

14. Let me introduce myself first...
I'm Luciano ( 🍕🍝) 👋
Senior Architect @ fourTheorem (Dublin )
nodejsdp.link
Co-Author of Node.js Design Patterns 👉
Connect with me:
(blog)
(twitter)
(github)
loige.co
@loige
lmammino
20% eBook discount on Packt
20NODEDUBLIN
9

15. We are business focused technologists that
deliver.
| |
Accelerated Serverless AI as a Service Platform Modernisation
Do you want to ?
work with us
loige 10

16. There was this song in my mind...
loige 11

17. I could only remember some random
parts and the word "dark" (probably
in the title)
loige 12

18. 13

19. 14

20. loige 15

21. Luciano - scrobbling since 12 Feb 2007
loige 15

22. Luciano - scrobbling since 12 Feb 2007
loige
~250k scrobbles... that song must be there!
15

23. loige 16

24. loige
~5k pages of history &
no search functionality! 😓
16

25. loige
But there's an API!
https://www.last.fm/api
17

26. loige 18

27. loige
Let's give it a shot
curl "http://ws.audioscrobbler.com/2.0/?
method=user.getrecenttracks&user=loige&api_key
=${API_KEY}&format=json" | jq .
19

28. loige 20

29. It works! 🥳
Let's convert this to JavaScript
loige 21

30. import querystring from 'querystring'
import axios from 'axios'
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json'
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
loige 22

31. loige 23

32. loige
We are getting a "paginated" response
with 50 tracks per page
23

33. loige
We are getting a "paginated" response
with 50 tracks per page
but there are 51 here! 🤔
23

34. loige
We are getting a "paginated" response
with 50 tracks per page
but there are 51 here! 🤔
(let's ignore this for now...)
23

35. loige
We are getting a "paginated" response
with 50 tracks per page
but there are 51 here! 🤔
How do we fetch the next pages?
(let's ignore this for now...)
23

36. loige 24

37. let page = 1
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
page
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
break // it's the last page!
}
loige 25

38. let page = 1
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
page
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
break // it's the last page!
}
loige 25

39. let page = 1
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
page
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
break // it's the last page!
}
loige 25

40. let page = 1
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
page
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
break // it's the last page!
}
loige 25

41. let page = 1
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
page
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
break // it's the last page!
}
loige 25

42. let page = 1
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
page
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
console.log(response.data)
if (page === Number(response.data.recenttracks['@attr'].totalPages)) {
break // it's the last page!
}
loige 25

43. loige 26

44. loige 26

45. Seems good!
Let's look at the tracks...
loige 27

46. // ...
for (const track of response.data.recenttracks.track) {
console.log(
track.date?.['#text'],
`${track.artist['#text']} - ${track.name}`
)
}
console.log('--- end page ---')
// ...
loige 28

47. loige 29

48. loige
* Note that page size
here is 10 tracks per
page
29

49. loige
* Note that page size
here is 10 tracks per
page
Every page has a song with undefined time...
This is the song I am currently listening to!
It appears at the top of every page.
29

50. loige
* Note that page size
here is 10 tracks per
page
Sometimes there are duplicated tracks
between pages... 😨
29

51. The "sliding windows" problem 😩
loige 30

52. loige
...
tracks (newest to oldest)
31

53. loige
...
tracks (newest to oldest)
31
Page1 Page2

54. loige
...
tracks (newest to oldest)
31
Page1 Page2
...

55. loige
...
tracks (newest to oldest)
31
Page1 Page2
...
new track

56. loige
...
tracks (newest to oldest)
31
Page1 Page2
...
Page1 Page2
new track

57. loige
...
tracks (newest to oldest)
31
Page1 Page2
...
Page1 Page2
new track
moved from page 1 to page 2

58. loige 32

59. Time based windows 😎
loige 33

60. loige
...*
tracks (newest to oldest)
34
* we are done when we get an empty page (or num pages is 1)

61. loige
...*
tracks (newest to oldest)
34
Page1
* we are done when we get an empty page (or num pages is 1)

62. loige
...*
tracks (newest to oldest)
34
Page1
t1
* we are done when we get an empty page (or num pages is 1)

63. loige
...*
tracks (newest to oldest)
34
Page1 before t1
(page 1 "to" t1)
t1
* we are done when we get an empty page (or num pages is 1)

64. loige
...*
tracks (newest to oldest)
34
Page1 before t1
(page 1 "to" t1)
t1 t2
* we are done when we get an empty page (or num pages is 1)

65. loige
...*
tracks (newest to oldest)
34
Page1 before t1
(page 1 "to" t1)
t1 t2
before t2
(page 1 "to" t2)
* we are done when we get an empty page (or num pages is 1)

66. let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
limit: '10',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
console.log(
`--- ↓ page to ${to}`,
`remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---`
)
for (const track of tracks) {
console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`)
}
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
} loige 35

67. let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
limit: '10',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
console.log(
`--- ↓ page to ${to}`,
`remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---`
)
for (const track of tracks) {
console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`)
}
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
} loige 35

68. let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
limit: '10',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
console.log(
`--- ↓ page to ${to}`,
`remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---`
)
for (const track of tracks) {
console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`)
}
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
} loige 35

69. let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
limit: '10',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
console.log(
`--- ↓ page to ${to}`,
`remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---`
)
for (const track of tracks) {
console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`)
}
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
} loige 35

70. let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user: 'loige',
api_key: process.env.API_KEY,
format: 'json',
limit: '10',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
console.log(
`--- ↓ page to ${to}`,
`remaining pages: ${response.data.recenttracks['@attr'].totalPages} ---`
)
for (const track of tracks) {
console.log(track.date?.uts, `${track.artist['#text']} - ${track.name}`)
}
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
} loige 35

71. loige 36

72. loige
The track of the last timestamp becomes the
boundary for the next page
36

73. We have a working solution! 🎉
Can we generalise it?
loige 37

74. We know how to iterate over every
page/track.
How do we expose this information?
loige 38

75. const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// callbacks
reader.readPages(
(page) => { /* ... */ }, // on page
(err) => { /* ... */} // on completion (or error)
)
loige 39

76. const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// event emitter
reader.read()
reader.on('page', (page) => { /* ... */ })
reader.on('completed', (err) => { /* ... */ })
loige 40

77. const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// streams <3
reader.pipe(/* transform or writable stream here */)
reader.on('end', () => { /* ... */ })
reader.on('error', () => { /* ... */ })
loige 41

78. import { pipeline } from 'stream'
const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// streams pipeline <3 <3
pipeline(
reader,
yourProcessingStream,
(err) => {
// handle completion or err
}
) loige 42

79. const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// ASYNC ITERATORS!
for await (const page of reader) {
/* ... */
}
// ... do more stuff when all the data is consumed
loige 43

80. const reader = LastFmRecentTracks({
apikey: process.env.API_KEY,
user: 'loige'
})
// ASYNC ITERATORS WITH ERROR HANDLING!
try {
for await (const page of reader) {
/* ... */
}
} catch (err) {
// handle errors
}
// ... do more stuff when all the data is consumed loige 44

81. How can we build an async iterator?
🧐
loige 45

82. Meet the iteration protocols!
loige
loige.co/javascript-iterator-patterns
46

83. The iterator protocol
An object is an iterator if it has a next() method.
Every time you call it, it returns an object with
the keys done (boolean) and value.
loige 47

84. function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
} loige 48

85. function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
} loige 48

86. function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
} loige 48

87. function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
} loige 48

88. function createCountdown (from) {
let nextVal = from
return {
next () {
if (nextVal < 0) {
return { done: true }
}
return {
done: false,
value: nextVal--
}
}
}
} loige 48

89. const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }
console.log(countdown.next())
// { done: false, value: 2 }
console.log(countdown.next())
// { done: false, value: 1 }
console.log(countdown.next())
// { done: false, value: 0 }
console.log(countdown.next())
// { done: true } loige 49

90. Generator functions "produce" iterators!
loige 50

91. function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}
loige 51

92. function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}
loige 51

93. function * createCountdown (from) {
for (let i = from; i >= 0; i--) {
yield i
}
}
loige 51

94. const countdown = createCountdown(3)
console.log(countdown.next())
// { done: false, value: 3 }
console.log(countdown.next())
// { done: false, value: 2 }
console.log(countdown.next())
// { done: false, value: 1 }
console.log(countdown.next())
// { done: false, value: 0 }
console.log(countdown.next())
// { done: true, value: undefined } loige 52

95. The iterable protocol
An object is iterable if it implements the
@@iterator* method, a zero-argument function
that returns an iterator.
loige
*Symbol.iterator
53

96. function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
})
}
}
loige 54

97. function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
})
}
}
loige 54

98. function createCountdown (from) {
let nextVal = from
return {
[Symbol.iterator]: () => ({
next () {
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
})
}
}
loige 54

99. function createCountdown (from) {
return {
[Symbol.iterator]: function * () {
for (let i = from; i >= 0; i--) {
yield i
}
}
}
}
loige 55

100. const countdown = createCountdown(3)
for (const value of countdown) {
console.log(value)
}
// 3
// 2
// 1
// 0
loige 56

101. OK. So far this is all synchronous iteration.
What about async? 🙄
loige 57

102. The async iterator protocol
An object is an async iterator if it has a next()
method. Every time you call it, it returns a
promise that resolves to an object with the keys
done (boolean) and value.
loige 58

103. import { setTimeout } from 'timers/promises'
function createAsyncCountdown (from, delay = 1000) {
let nextVal = from
return {
async next () {
await setTimeout(delay)
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
}
} loige 59

104. import { setTimeout } from 'timers/promises'
function createAsyncCountdown (from, delay = 1000) {
let nextVal = from
return {
async next () {
await setTimeout(delay)
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
}
} loige 59

105. import { setTimeout } from 'timers/promises'
function createAsyncCountdown (from, delay = 1000) {
let nextVal = from
return {
async next () {
await setTimeout(delay)
if (nextVal < 0) {
return { done: true }
}
return { done: false, value: nextVal-- }
}
}
} loige 59

106. const countdown = createAsyncCountdown(3)
console.log(await countdown.next())
// { done: false, value: 3 }
console.log(await countdown.next())
// { done: false, value: 2 }
console.log(await countdown.next())
// { done: false, value: 1 }
console.log(await countdown.next())
// { done: false, value: 0 }
console.log(await countdown.next())
// { done: true } loige 60

107. const countdown = createAsyncCountdown(3)
console.log(await countdown.next())
// { done: false, value: 3 }
console.log(await countdown.next())
// { done: false, value: 2 }
console.log(await countdown.next())
// { done: false, value: 1 }
console.log(await countdown.next())
// { done: false, value: 0 }
console.log(await countdown.next())
// { done: true } loige 60

108. loige 61

109. import { setTimeout } from 'timers/promises'
// async generators "produce" async iterators!
async function * createAsyncCountdown (from, delay = 1000) {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}
loige 62

110. import { setTimeout } from 'timers/promises'
// async generators "produce" async iterators!
async function * createAsyncCountdown (from, delay = 1000) {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}
loige 62

111. import { setTimeout } from 'timers/promises'
// async generators "produce" async iterators!
async function * createAsyncCountdown (from, delay = 1000) {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}
loige 62

112. The async iterable protocol
An object is an async iterable if it implements
the @@asyncIterator* method, a zero-argument
function that returns an async iterator.
loige
*Symbol.asyncIterator
63

113. import { setTimeout } from 'timers/promises'
function createAsyncCountdown (from, delay = 1000) {
return {
[Symbol.asyncIterator]: async function * () {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}
}
}
loige 64

114. import { setTimeout } from 'timers/promises'
function createAsyncCountdown (from, delay = 1000) {
return {
[Symbol.asyncIterator]: async function * () {
for (let i = from; i >= 0; i--) {
await setTimeout(delay)
yield i
}
}
}
}
loige 64

115. const countdown = createAsyncCountdown(3)
for await (const value of countdown) {
console.log(value)
}
loige 65

116. const countdown = createAsyncCountdown(3)
for await (const value of countdown) {
console.log(value)
}
loige 65

117. Now we know how to make our
LastFmRecentTracks an Async Iterable 🤩
loige 66

118. function createLastFmRecentTracks (apiKey, user) {
return {
[Symbol.asyncIterator]: async function * () {
let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user,
api_key: apiKey,
format: 'json',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
yield tracks
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
}
}
}
}
loige 67

119. function createLastFmRecentTracks (apiKey, user) {
return {
[Symbol.asyncIterator]: async function * () {
let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user,
api_key: apiKey,
format: 'json',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
yield tracks
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
}
}
}
}
loige 67

120. function createLastFmRecentTracks (apiKey, user) {
return {
[Symbol.asyncIterator]: async function * () {
let to
while (true) {
const query = querystring.stringify({
method: 'user.getrecenttracks',
user,
api_key: apiKey,
format: 'json',
to
})
const url = `https://ws.audioscrobbler.com/2.0/?${query}`
const response = await axios.get(url)
const tracks = response.data.recenttracks.track
yield tracks
if (response.data.recenttracks['@attr'].totalPages <= 1) {
break // it's the last page!
}
const lastTrackInPage = tracks[tracks.length - 1]
to = lastTrackInPage.date.uts
}
}
}
}
loige 67

121. const recentTracks = createLastFmRecentTracks(
process.env.API_KEY,
'loige'
)
for await (const page of recentTracks) {
console.log(page)
}
loige 68

122. Let's search for all the songs that contain the
word "dark" in their title! 🧐
loige 69

123. async function main () {
const recentTracks = createLastFmRecentTracks(
process.env.API_KEY,
'loige'
)
for await (const page of recentTracks) {
for (const track of page) {
if (track.name.toLowerCase().includes('dark')) {
console.log(`${track.artist['#text']} - ${track.name}`)
}
}
}
}
loige 70

124. loige 71

125. loige
OMG! This is the song! 😱
...from 8 years ago!
71

126. For a more serious package that allows you to
fetch data from Last.fm:
loige
npm install scrobbles
72

127. Cover picture by on
Thanks to Jacek Spera, , , ,
for reviews and suggestions.
Rod Long Unsplash
@eoins @pelger @gbinside
@ManuEomm
-
loige.link/async-it loige.link/async-it-code
for await (const _ of createAsyncCountdown(1_000_000)) {
console.log("THANK YOU! 😍")
}
loige
nodejsdp.link
20% eBook discount on Packt
20NODEDUBLIN
73