I have one client's application which needs cloud storage, another uses S3, another project needs local disk, and all three bring me much the same list of jobs.
Put, get, remove.
I'd sooner keep improving one storage implementation than start again for each client. The application needs a file. Surely it can ask for one without a fresh conversation about storage every time?
The methods look easy enough to agree on. It's what each backend means by them that takes the work. I'm trying to get those differences out of the application, which means I have to deal with them here.
“Give me the hash of this file” sounds like one job wherever the file lives. Download some bytes, identify them and move on.
The trouble starts when I try to avoid the download. The object already has an ETag in its metadata, and I can read that without downloading the file. Can I use that as the hash?
Some S3 ETags are the MD5 of the object data, but multipart uploads and some encryption arrangements produce ETags that aren't. S3's object reference spells out those cases.
How was the object created? That decides whether I can take the shortcut.
Compare the promise before the API
Suppose the application wants to avoid processing unchanged file content. There are at least two possible contracts:
| Contract | What the caller may conclude |
|---|---|
| Provider version token | This provider reports the same object version or validator |
| Content digest under a named algorithm | These bytes produced this digest |
For a cache inside one provider the cheaper version token can answer my question, while comparing content across stores calls for the named digest. GetHash leaves the caller guessing which one I've decided to return.
Check what the provider promises
Would another provider be tidier? Google Cloud Storage has checksum metadata too, but composite objects don't provide an MD5 digest, and its metadata documentation describes what's available for each kind of object.
The adapter can use a provider's checksum if it's guaranteed to be the right one for this version of the object, otherwise it must read the bytes and calculate the digest. I'd prefer to skip the download, but I need the answer GetHash promised. A 32-character hex ETag looks so much like an MD5 that I want to be done here. I can rule out the usual multipart form by checking its spelling. But an ETag can still look like an MD5 and mean something else. Annoying. Check the spelling, check how the object was created, check the provider's promise. I only take the shortcut where the provider promises the digest I need.
An adapter can also store a checksum in a separate <key>.md5 file to avoid repeating the download, but suppose GetHash returns that saved checksum before reading the main object's metadata. Which version of the object does the digest describe? If Put replaces the file without removing or updating its saved checksum, the next GetHash returns the old digest, and now I have to remember the little companion file whenever I replace its owner.
A stale digest is an impressively tidy answer to the wrong file.
Put the replacement rules behind Put
I'm shaping the storage interface around operations such as put, get, stat, copy and remove, with provider-specific implementations underneath. For basic file operations I can start this small:
type Storage interface {
Put(context.Context, string, io.Reader) error
Get(context.Context, string) (io.ReadCloser, error)
Remove(context.Context, string) error
}
Even with these three methods, I've got decisions to make. Does Put replace an existing object? How does the caller recognise “not found”? Who closes the reader? When does a successful write become visible to a reader?
Those answers belong in the contract, or the application still has to ask whether it's using S3 or local disk before it knows what a result means. And I've hidden the provider's imports, but left its rules for everyone else to remember.
Local disk makes me question assumptions that feel natural in the cloud, but callers should still use logical keys and the shared error categories. Handing them a filesystem path would undo part of the reason I made this interface.
On local disk, even “replace this file” needs more thought. If the upload fails halfway through, what happens to the file that was already there? I can open the destination with os.Create and copy the incoming reader straight into it. There isn't much code to look after. If the copy succeeds, the file is where the caller asked me to put it. Open an existing report.pdf with os.Create and it truncates the old file. If the incoming reader fails halfway through, Put returns an error, leaving part of the replacement at that path, and the previous report has already gone.
I'd expect the caller to retry a failed upload. With this version, I'd also be asking it to recover a report I've already deleted. That's more work than I want a failed upload to create for the application.
So I write the replacement to a temporary file in the destination directory, check that copying and closing both succeeded and only then rename it into place. If copying or closing fails, I remove the temporary file and leave the existing report alone.
On a POSIX filesystem that same-directory rename gives me atomic replacement. Someone opening report.pdf sees either the old file or the completed replacement. They don't open the half-written upload. Go's Rename documentation calls out the platform differences.
This belongs behind Put. The application shouldn't have to arrange a safe replacement each time it uploads a report. The saved checksum has to follow the replacement too. Its sidecar stays put when the main file is renamed.
Then Get brings another question. How often should I fetch a file several callers want?
One download, several readers
Several requests can ask for one stored image before its download finishes. If every Get starts a download, I fetch identical bytes several times. I want to share that result with the waiting requests.
Go's singleflight package lets one call do the work while overlapping calls with the same key wait for its result. I group those calls by provider, repository and object key, so requests for different stores don't get mixed together. This shares a download while it's in progress. Ask again later and I'll fetch it again.
But Get returns a reader. Handing everyone that reader gives them one shared position in the stream. One caller reads the opening bytes, another carries on after them, and they can also close the underlying reader on one another. Everyone wanted all the bytes.
I share the result by buffering the downloaded bytes and giving each waiting caller a fresh reader using io.NopCloser(bytes.NewReader(data)). They share the data, but each reader starts at the beginning and advances independently. Closing one doesn't close the others.
The cost of sharing
That means keeping the whole object in memory and finishing the download before handing out the readers. For small images I'm happy to make that trade, but I'd rather download a large file twice than hold the whole thing in memory.
With singleflight enabled I check the reported object size first. Small enough objects take the buffered path while larger ones stream directly from the provider. That lets me choose which objects to buffer. Many different small files can still add up in memory.
Coordinating this above the adapters saves each backend from implementing it. The provider supplies bytes while the shared service decides when callers can share a download and gives each an independent reader.
The interface is still only a few lines. Behind it, I'm keeping track of file replacement, saved checksums and what several callers can share. Getting those rules to line up takes more thought than the method list suggests.
I still want this in one place, though. A client application can start with the work I've already done, rather than inherit another set of rules for moving files around. Then I can get on with what it needs the files for.