The lyrics have arrived. Which of our services can use them?
I'm trying to work that out for Tiwio. There isn't one pipeline definition to read so I'm opening components and checking their inputs. Cities needs lyrics, weather words needs lyrics and language does too. Mood wants the artist and title plus lyrics before returning its answer.
Its answer is always “Happy”.
Happy, but waiting for lyrics. The result is constant (I can't help liking that), yet the input check still gives it a place in the order. By the third component, a picture would be welcome. The Tiwio account covers the larger project. Here I'm following one song from its name to searchable metadata.
Follow the lyrics
Each Scala service uses Akka Streams to connect its processing steps. Kafka carries messages between services through named streams called topics. MongoDB stores song metadata and records of the work.
Let's give an invented song the lyrics “Rain over Paris”. We begin with only its artist and title. The importer makes those fields available and publishes an announcement to Kafka, where lyrics and Spotify metadata hear it through separate consumer groups so one reading it doesn't take it from the other.
Each component checks the announced field names against its inputs. The shared base class calls this isMatch. Lyrics needs artist and title and can ask for work. Cities waits for lyricsText, which doesn't exist yet.
The component's first stream checks those names and requests work on another Kafka topic. The request includes its component's name, version and reply topic. I can follow that little stream by itself. An announcement passes the input check and leaves as a request without fetching a word of the song. Fetching the words waits for a reply on another topic.
Only names, no words yet
An announcement carries field names rather than values.
In my head the lyrics component can get on with it. All it can actually do is ask for work. Where do I look if the reply never arrives?
Has this song already been processed?
The database manager consumes requests through its own Akka stream. It checks MongoDB for work already running or completed for that song and component, and if a record blocks the request the component gets no reply. Otherwise it inserts an InProgress record and gathers the metadata onto the component's reply topic. An arrow from artist and title straight to lyrics would leave that decision out. The inputs can be ready while a stored job record stops anything coming back. That's why I'm following the messages as well as listing the dependencies.
Only when the reply arrives does the component's second stream receive the values, so for lyrics it can finally take the artist and title, fetch the text and publish its result to the update topic. The database manager saves it, marks the record Done and announces the available fields again.
Now the announcement includes lyricsText, so cities can request it and return Paris through the same request/reply/update cycle, while weather words can independently return rain. Neither component has to call the other or know that it exists.
Put the missing picture together
I like adding a consumer without teaching the producer to call it, but cities needs lyrics and that relationship is spread across its input check, Kafka topics and the database gate. Following a topic gets me to another service and more code to find out why nothing came back. Topics to open, checks to read, job records.
The version gate has a particularly irritating example. For completed records, the query blocks on a stored version less than or equal to the requested one. Ask for cities 1.0.1 after 1.0.0 has finished and the old result blocks the new job.
The rerun comparison is backwards.
The check and insertion are separate database operations. Whether concurrent requests can reserve the same job depends on the database constraints and conflict handling too.
I need those details beside the input check to explain a waiting job.
Collecting the input checks gives me this list. The services don't read it as configuration yet:
lyrics: [artist, title]
spotify_metadata: [artist, title]
cities: [lyricsText]
weather_words: [lyricsText]
language: [lyricsText]
mood: [artist, title, lyricsText]
Lyrics and Spotify can start from the same artist and title. Once lyrics finishes, cities, weather words and language all become eligible, and mood gets its final input too. In our example cities and weather words add separate fields so they can overlap and leave us with both Paris and rain. Shared inputs alone can't show a relationship that makes their order matter, so that would need describing as well.
I'd keep the description near the code because a separate drawing would be another thing to update every time I change a component. I've only written the inputs here. Outputs are the missing half. Lyrics produces lyricsText, and cities needs it. With both halves written down, a checker could connect them or tell me nobody produces a required field.
I'd put job records beside those connections. The connections say what can run, while the records say what has run and which version did it and what is still in progress.
That's the view I'm after, without opening half the services to find out why Happy is waiting.