Tiwio, a local soundtrack for your travels

You're walking through Paris in the rain. Spotify puts on a song connected to Paris, then another that fits the weather. Your phone stays in your pocket. That's the idea behind Tiwio, a project I started with a few friends.

It started over a few too many ciders. The cider was aged in whisky barrels. Extremely tasty, extremely unwise as the precursor to a project. We thought we were geniuses with what we were designing. The commitment was probably a bit larger than we'd have made sober.

I'd love to put this on while walking somewhere new. There are probably musicians from that place I haven't heard of, and I like the thought of finding them along the way. I'm interested in what the surroundings could choose for me.

The plan is for Tiwio to work in the background and arrange the music through Spotify, so we can choose the songs without building another player to play them. There's a small phone interface for login, a map and preferences, but the experience I care about happens after you've stopped looking at it.

What does a place sound like?

“Play something from Paris” hides the first difficult question. Does that mean the artist was born there? That the song was recorded there? That its lyrics mention the city? Those are different relationships, and we need data for whichever one we choose.

Our enrichment pipeline starts with artist and song names. It fetches lyrics through a service backed by Genius and looks for words matching city names, leaving out words shorter than four letters before attaching coordinates to the matches. We can later search those coordinates by distance.

Local music, well, songs that mention the place. Where the music was made is another question entirely. That difference matters to the original idea. A song can mention somewhere the singer dislikes or somewhere they never visit, and a place name can be an ordinary word, so matching its letters doesn't explain why they're in the song. It gives us a candidate.

The coordinates give us something to query, so we can start trying the idea. I have to remember where they came from though, we might have found one word in a verse, and now I'm asking the song to represent a whole city.

I'd like to bring in data about where musicians are from and where they're based, alongside the lyrics. That feels closer to the local music I want Tiwio to find. We're looking for sources we can integrate, but getting access to well-curated location data can be difficult.

I can download Wikipedia or some larger database and cross-reference. More sources, more names to match, another database. But I'd love to prioritise smaller musicians. The vibe would be better, I think.

Weather gets a similar treatment. The enrichment side looks for weather words in lyrics. At request time the serving side can compare those tags with nearby weather, connecting the world outside with the words in a song without understanding how that song feels.

A cheerful song about leaving Paris could follow you round Paris all afternoon because the computer found the word Paris and considers its job done.

How far away is relevant?

Once a song has coordinates attached, we need a way to favour nearby places. Our city query uses six overlapping distance searches. Each search carries a different weight:

The important word is overlapping. A place 300 metres away sits inside all six rings, so its song can collect contributions from all six searches rather than the smallest ring it fits into. A place 12 kilometres away only qualifies for the last two.

The query code collects the hits, groups them by song and adds their scores, including weather and selected genre or language preferences, then keeps the top fifteen songs.

Several city coordinates can belong to one song document, and a ring matches it once if any qualify, so two nearby names in one song don't earn two copies of that ring's weight.

That distinction is easy to miss when turning the idea into a little loop.

I could happily spend an afternoon adjusting those numbers. We also need better location data. Only that can say whether the song belongs to Paris or mentions it once.

Here is an example with fictional songs, adding matching ring weights for distance and one point for a matching weather tag, while our actual weather query uses Elasticsearch's text relevance scoring.

The example gives tied songs a stable order by identifier.

Nearby wins here

In this model the song connected to a place 300 metres away gets 11.75 distance points, while the one connected to a place 12 kilometres away gets 0.75. A matching weather tag adds one. Rain can separate nearby songs without lifting the distant one above them, which means I'd have to change the balance if I wanted distant rain to beat nearby sunshine. Nearby wins by quite a lot. Moving the point around lets me try that out, and the outer rings still leave some room for songs from the wider area.

Keeping the phone in a pocket

On login the phone client finds or creates a private playlist named Tiw.io, clears its tracks and requests songs, with another request scheduled every five minutes.

When song results arrive, the client takes their Spotify track identifiers and replaces the upcoming part of its Tiw.io playlist. It tries to keep the current track, reload the playlist and seek back to the playback position it has read.

Suppose the playlist contains three tracks and the second is playing. A new location gives us two fresh suggestions. The intended change looks like this:

Before:  first track → current track → old suggestion
After:   first track → current track → new suggestion → another suggestion
                          ↑
                  keep the playback position

The part you've already heard stays in the list, and the upcoming part changes. Spotify handles playback. Our job is to choose what follows and leave the listener's current song alone.

That last part matters. And moving past a coordinate shouldn't invite a restarted chorus. We want the next songs to change while the current one gets to finish being a song.

The implementation makes several calls to do this. It reads the playlist, removes its tracks, reads the player and writes the new list, and keeping the music comfortable is already more work than choosing the highest score. There is still work here around interrupted requests and changes to the player while those calls are in flight. Interrupted requests, a listener pressing Next, another update.... Which track is current now?

Anyway, will it stay awake? A five-minute timer in the client only works if the mobile operating system keeps the app awake to run it. Background operation is part of the intended product. It is one of the things that needs finishing and testing before someone can leave this running.

Two systems and a file

The phone client watches location and sends coordinates, genre choices and language choices over a WebSocket. Messages go both ways.

On the server the request takes a longer route than its result suggests:

Phone → WebSocket API → Kafka → song query → Elasticsearch
Phone ← WebSocket API ← Kafka ← song results
                                  |
                                  └→ weather request → saved weather

The weather request starts alongside the song lookup, but the lookup uses nearby weather that's already stored rather than waiting for this request to finish. No weather yet? The query can run without its contribution. I want the weather to matter. On a new location there might not be any to use, and sending a request isn't the same as having the answer ready for the song query.

The serving side needs an answer promptly. Enrichment has a different job. It can take time to fetch lyrics, ask Spotify for metadata and attach the fields that later searches need.

We give those jobs separate services and storage. The enrichment side uses MongoDB to keep records of the work. The serving side uses Elasticsearch to search the prepared data. Each side has its own Kafka setup.

There's a practical reason to separate slow preparation from a listener's request. Waiting for every lyric lookup while somebody wants music would make a very different sort of listening experience.

How do they meet? The connection between the two sides is much more ordinary than their internals:

Artist and title
      ↓
Enrichment services → MongoDB
                         ↓
                     CSV export
                         ↓
                   manual transfer
                         ↓
                  Jenkins upload job
                         ↓
                   Elasticsearch → song queries

Exporting produces a CSV file. The loading job accepts an uploaded file, starts a pod and copies the file into it, which is how prepared data gets into the serving system for song queries.

We've got Kafka on both sides and we still have to move a CSV file between them, which is where we've got to for now.

I can get the prepared songs into the search without first building another service. There's a lot I'd like to connect up automatically, but the upload job lets us try the data while that work waits.

Since moving to the Netherlands, I've spent much of my spare time learning infrastructure, first Docker and then Kubernetes. Tiwio gives that learning an application to organise around. Containers need something to run. A music idea supplies plenty.

Jenkins builds the images and deploys the services, and I enjoy following that route from a successful build to a running service, then seeing a message reach my code. Getting the whole thing connected is satisfying in its own right.

Our services are named after digestion (ingest, digest and absorb) “because that is apparently hip right now”, according to our README. The same README begs, in capital letters, for someone to help me find a better naming scheme for the Kafka topics.

The next work is better location data and making the phone app dependable in the background. I want the music to follow the walk while the phone stays in my pocket.