The repository says three replicas. The cluster says five. Both are perfectly happy to tell me so.
Which one am I supposed to trust? The command succeeded. The file hasn't changed. Neither is lying, and now I have to work out which number we meant to keep.
I'm starting to use Argo CD and Flux because they keep asking after deployment. There's a declaration I can read, and a controller comparing it with the cluster while I get on with something else. That ongoing attention is what draws me to them.
“Infrastructure as data” is how I like to think about it. I can read the result we want in a file and review a change before it reaches the cluster. Then the controller keeps comparing it with what is there. I'd like that to carry on after the deployment command finishes.
A green tick has a short memory
Take a small example. A deployment pipeline applies a Kubernetes Deployment with replicas: 3. The apply command succeeds, Kubernetes has its instruction to work towards three copies, and that successful command is enough for everyone to go and make tea.
Then someone changes the Deployment directly:
kubectl scale deployment web --replicas=5
Kubernetes now works towards five replicas. It's following its instructions. The repository still says three because the command changed the cluster's declaration without changing the file.
That difference is drift. The next pipeline run might put the value back. Until then, the green tick is a souvenir. It hasn't a clue which number the team intends to keep.
The scale command did what somebody asked it to do. I'm now looking at a file that says something else, with no further pipeline run to put it back. That's the gap I'm trying to fill.
Terraform and Puppet already let us describe resources and configuration. What keeps my attention here is maintaining that declaration after the deployment has finished.
Kubernetes already has controllers that compare a requested state with what exists. If a pod disappears, the relevant controllers work to replace it. They reconcile the difference, meaning they observe and compare and act again.
I'd love to leave that job to Kubernetes, except Kubernetes currently has an instruction to run five. Replace a missing pod and it will helpfully restore five, because it can't read my mind or the other instruction sitting in our repository.
A GitOps controller supplies that connection. It reads the manifests in Git, compares them with the cluster's resources, and applies changes according to its policy. Kubernetes then does its own work to bring the running workloads into line.
I find it useful to follow both jobs separately. If Kubernetes restores a missing pod it is still following five, so the GitOps controller has to put three into the Deployment before Kubernetes can work towards that instead.
After our manual edit the three numbers are three in Git, five in the Deployment and five running. With automatic repair enabled, the GitOps controller puts the Deployment back to three. Kubernetes then works towards three running replicas. A commit changing Git to four starts at the other end. The Deployment and running replicas follow it.
Those changes happen over time and API requests or pods can fail. I can follow the Git value, the Deployment value and the running count together to see where the work has stopped. Sync status and pod health answer different questions.
In Argo CD, automatic repair of manual changes needs self-healing enabled. This illustrative Application watches the apps/web directory of an example repository:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: web
namespace: argocd
spec:
project: default
source:
repoURL: https://example.com/team/infrastructure.git
targetRevision: master
path: apps/web
destination:
server: https://kubernetes.default.svc
namespace: default
syncPolicy:
automated:
selfHeal: true
Deleting resources is a separate policy. This example leaves automatic pruning off because a missing file and a manual replica edit call for different decisions. Grouping both under “automatic” would hide that choice from us.
A failed sync still needs attention. Argo CD won't automatically retry the same commit with the same source settings after that sync has failed. I need to see that failure, even with self-healing switched on.
Flux checks for new commits and periodically reapplies the manifests. It can return to the same Git revision to repair a difference in the cluster. I don't have to wait for another release to put three back.
Make the record part of the work
For a release the build still compiles code, runs tests and pushes an image, and the deployment change becomes a commit updating the image reference. The controller reads the change from Git, so CI no longer needs a cluster credential to apply this deployment.
Flux can take on that image update too. With automation enabled, it watches image registries and updates the manifests under my policy, those updates become commits, and I still get a recorded change to inspect even when a person hasn't typed it. Choosing which images it may pick becomes part of the release decision.
The commit tells me which release we intend to run, and changing that file can now change production. Repository write access has become deployment access. If Flux pushes directly to the watched branch, those updates won't wait for a pull-request review.
The small comparison
Strip away Git, YAML and the Kubernetes API, and the comparison fits in a few lines of Go:
package main
import "fmt"
func reconcile(desired, actual map[string]int) {
for name, replicas := range desired {
if actual[name] != replicas {
fmt.Printf("repair %s: %d -> %d\n", name, actual[name], replicas)
actual[name] = replicas
}
}
}
func main() {
desired := map[string]int{"web": 3}
actual := map[string]int{"web": 3}
actual["web"] = 5
fmt.Println("manual edit:", actual["web"])
reconcile(desired, actual)
fmt.Println("after reconcile:", actual["web"])
desired["web"] = 4
reconcile(desired, actual)
fmt.Println("after desired change:", actual["web"])
reconcile(desired, actual)
}
Save it as reconcile.go and run go run reconcile.go:
manual edit: 5
repair web: 5 -> 3
after reconcile: 3
repair web: 3 -> 4
after desired change: 4
The last call does nothing because the numbers already agree. There isn't much code in that comparison (part of why I like it). The real controller has API requests that fail and need another attempt, but the controller keeps coming back to this little comparison.
Who owns the number?
My three replicas shouldn't be fixed in Git if an autoscaler is choosing the number, and Argo CD lets me ignore differences in selected fields, including /spec/replicas. That handles the comparison, but a sync can still write a field present in the manifest, so for an autoscaled Deployment I leave spec.replicas out of the file.
Flux documents this too, so GitOps can maintain the rest while the autoscaler chooses how many copies to run.
Fields to leave alone, images to allow, emergencies.
Two helpful robots can spend all day undoing each other's work.
An emergency manual change needs an explicit pause or policy change so reconciliation doesn't undo it. The pause needs to be visible because the controller also stops applying new commits, and when we're ready for it to take over the declaration needs to say what should run.
The deployment command can finish. I want that ongoing job running while I make the tea.