I've seen more two-day tickets for environment variables than you can imagine.
A developer needs a new setting, but infrastructure owns the deployment files, so the change starts with a ticket, and it often lands with me. I ask what the setting does, edit the YAML and arrange the deploy, then go back to ask whether the application looks right. Everyone has done their job. Somehow, changing one setting has become a small correspondence course. I've spent two years on the operations side, first around car telemetry and now the systems behind the Dutch ports, so I know that seat in the middle very well. I don't want to make a career out of passing settings back and forth. I'd prefer to help the team make the change themselves.
I call this developer governance. Configuration, releases and alerts belong to the application team. Infrastructure owns the cluster and shared services and the rules around their use. Developers first.
Ownership matters.
Being responsible for a service while having to ask another team for every change is miserable, the person answering for it needs a useful way to change it.
I understand how the controls end up with infrastructure. Kubernetes has plenty of details to get wrong, and operators spend time learning them. Keeping the files with the people who understand them feels sensible.
But then the developers get less practice. Next time, keeping the files with infrastructure feels even more sensible.
I can keep answering tickets and the team will keep needing to send them. Another setting to change, another release to arrange.... I don't like where that leaves us. I'd prefer to spend some time going through the files together, even if the first change takes longer.
We can break that circle by putting application changes into the team's own work:
| Work | Infrastructure-owned arrangement | Developer-first arrangement |
|---|---|---|
| Application settings and deployment files | Infrastructure edits them | Application team maintains them |
| Releases and replica counts | Infrastructure runs the change | Application team uses the shared pipeline and quotas |
| Application alerts and dashboards | Infrastructure maintains them | Application team maintains them, with help |
| First response to an application alert | Infrastructure asks the developers | Application team investigates, with infrastructure support |
| Cluster and shared services | Infrastructure team | Infrastructure team |
| Access rules, quotas and shared checks | Infrastructure team | Infrastructure team |
That gives a developer authority over their application within agreed limits. Cluster administration stays separate.
Give the owner files they can read
The environment-variable change should start in the application's repository. A developer edits it and a teammate reviews it before the pipeline checks and applies it. Keeping the deployment files beside the code lets both changes happen in the same review.
I'm fussy about the format. I like Helm when I'm packaging shared software for people with different settings. For one team's little application I start wondering who the template is helping.
The developer needs to find an environment variable. Plain Kubernetes YAML, please.
Consider this fragment inside a Helm chart's container definition:
{{- with .Values.env }}
env:
{{- toYaml . | nindent 2 }}
{{- end }}
That template reads and indents an env value from another file:
env:
- name: GREETING
value: Welcome to production
To see where the setting lands, the developer follows it from the values file through this template into the Deployment. The chart's values form another interface with names and rules the chart author has to document.
I can read this and I could explain it. But now we've paused an application change to explain whitespace and where a value belongs. If I hand over controls that still need me standing beside them, I haven't got very far.
Kustomize starts with ordinary resource files and applies changes to them. I can keep a base Deployment and Service, then give each environment a small overlay containing its differences. Kubernetes 1.14 has brought Kustomize into kubectl, so this works with the client we already use.
For the same production change, a patch identifies the Deployment and container, then adds the setting:
apiVersion: apps/v1
kind: Deployment
metadata:
name: greeting
spec:
template:
spec:
containers:
- name: web
env:
- name: GREETING
value: Welcome to production
The production overlay includes that patch alongside its replica setting, while development keeps using the base without that production patch. I can open the production patch, recognise the Deployment's field names, look at the combined output, and review what we're about to apply without first explaining a template language. That is one less set of names to explain. I check the combined output with kubectl kustomize, or helm template for the chart. Either way the reviewer needs to see the final Deployment we'll apply. For this application I choose the base and overlay. Helm keeps its place in the shared software I package and install.
Give them a view of the result
Suppose they've made the change and the pipeline is green. Now what? I'd look at pods and events and logs then check whether the application is doing what I expected. The developer needs that view too. Otherwise we've moved the request out of my queue and put the next question straight back into it.
Halfway through following a request is a rotten time to discover you need another ticket. Arrange that access with the team before they need it.
Logs can contain sensitive data. Who needs to see what? Permission to inspect a namespace needn't include its secrets, so I want useful access designed alongside dashboards that answer the team's questions.
Faster tickets keep me in the middle of every change, but with files and access to the running application the developer can follow their own edit through to the result.
Give them the keys, well, give them a way to learn how to use them too.
There is plenty left for me to do. I can provide a starting repository, sensible resource limits and health checks and a pipeline that checks manifests before deployment. I can arrange team access and quotas, keep the shared services working, and sit with someone through their first release. Starting repositories, quotas, someone beside you when the thing fails.
That last part matters as much as the files. If a deployment fails we work through it together, if an alert is useless we change it together, and the application team gradually needs less help because it understands more of what it owns.
Werner Vogels's “you build it, you run it” puts operational responsibility with the developers. I agree with that direction. Said on its own the line can sound like someone handing over a pager and leaving the room (which would undo the helping part).
I'd rather pull up a chair.
We still need to agree who responds first and when infrastructure joins an incident. I'd want to work that out with the team while we set up the access, then stay around for the first releases. There will be plenty to talk about when the thing won't start. At least the person who wrote it can open the files and help me find out why.