You upload a song, the progress bar reaches the end, and you'd expect to be done. Except I've still got to convert it into the files the site needs. The recording has arrived but you can't listen to it yet.
The recording needs other versions. Different browsers want different playback formats and the shop wants the full song. I'd like a thirty-second preview before you buy it. One upload, then playback versions, a lossless copy, previews.... I can't ask the progress bar to wait for all of that and pretend it's still uploading.
PHP can do it, but it's busy
First I put it all in the upload request. The file is already there. It's an easy place for the next job too, convert the recording I received in this request, make the other versions, and send them to storage without deciding where another process fits or when to start it. That means five files with different playback formats, a lossless copy and a preview. Each conversion reads and decodes the recording again and then encodes another version. PHP can keep going after the tab closes and I've given it up to two hours. I still want the song to finish even if the person gets bored of waiting.
On the web server.
While it's busy making five copies of a recording, somebody else wants to open a band's page. Giving PHP two hours doesn't give me another machine. Conversion needs another place to happen.
The request can end first. Receive the recording, save it in a pending folder, then leave a small JSON ticket beside it with the song and user, so the worker can find a job without opening the recording first. A worker can come along later and do the conversion.
A scheduler called cron starts the worker at regular times. The worker takes a lock so the next run can avoid starting another scan while this one is still busy.
What is waiting?
With made-up names the folder looks like this:
pending/
song-17
song-17.json
I can scan those tickets to see what needs doing without opening every recording. I can look in the folder myself when I'm wondering where an upload has got to.
Let another machine take over
It's a queue made out of files.
The worker reads each ticket, converts the recording and uploads the results. Then it removes the ticket. All that happens after the browser request has ended.
That gets the browser request out of the way. I'm still making the same server receive the recording and convert it though. The folder is quite handy but I'd like another machine to pick up the work in it.
The recording is going into storage anyway so I can let the browser send it straight to Amazon S3. PHP can deal with whose song it is and another machine can handle the conversion.
The browser shouldn't get my storage credentials. My server can sign a permission slip for this particular upload instead. The browser sends it along with the recording and S3 checks it before accepting the file.
The signing secret stays on my server. PHP still keeps track of the upload but no longer has to receive the audio itself.
The separate encoder can then pick up the pending recording. It probes the file with ffmpeg to find out what's in it and sends that information back to the site. I want to approve the song before it gets added to the catalogue so the encoder asks what to do next.
Are we allowed to convert it yet?
The site answers with a number:
| Number | What the encoder should understand |
|---|---|
| 0 | The upload hasn't been inspected yet |
| 1 | The probe details have arrived. Wait for approval |
| 2 | Approved. Go ahead and convert it |
| 3 | Declined. Remove the pending source |
| 4 | Already handed out for processing |
The first visit stores probe details. Now it's waiting for approval. Approval gives the encoder a newly assigned song ID, title and band name, and it needs those to name the files and add their metadata.
I've separated those values with colons, which means I now have to remove colons from song titles and band names before sending them to the encoder. Band names, colons, rules about how someone can name a song. A small text protocol, and already I've given myself opinions about punctuation.
Status 4 gets ahead of itself. On approval the API replies with permission to convert and sets the stored status to 4. The encoder hasn't finished yet. It might not even have started its first conversion. The code calls that “fully processed”. I've handed out the work, I've called it finished, and the stored status tells me nothing about whether the files exist or whether a conversion failed. Two workers could also see the approval before either one changes the status and both start the job. Worse, a worker that gets status 4 removes the pending source, even if another worker is still using it.
One worker mustn't delete the recording from under another. Who has it? Have they finished with it?
The current encoder makes M4A, Ogg and lossless FLAC files plus two thirty-second previews.
After the commands run, the script removes the pending source and its probe report. It also cleans up declined recordings, which I don't want sitting there waiting to be picked up again.
That cleanup doesn't properly check that every output was created successfully. An empty pending folder could mean the work finished or that I've deleted the source after something went wrong. I can check the output files.
The web server can concentrate on pages now, which is what I wanted. I'm not quite done with the encoder. Before it throws the source away, I want it to check the FLAC and other files really exist. Then I can tell the person who uploaded the song something more useful than “the upload finished”.