I provided consulting expertise last week to a company who was experiencing technology outages with negative customer impact. One of my first questions was how they handle code freezes.
From the response, both body language and words, I could tell this was a sensitive subject. For context, as a CIO I know this is sensitive for a lot of organizations so I took it in stride.
The conversation went something like this:
How do you handle code freezes?
We really can’t afford them because we have to make changes right until the last minute to get as many features in as possible, and we multiple deployments a week to be responsive.
How’s that working for you?
Silence. And then a laugh.
Technical changes are prone to error. To reduce error, I suggest a freeze. The freeze doesn’t need to be weeks or months, but there needs to be a moment in time that you have some way to either confirm the changes will work as expected or take a risk that it won’t. There are times the risk is acceptable, but not usually with financial transactions or personal data.
In a digital world where your customers will experience your issues immediately, you need to be good at either a) freezing code and deploying cleanly, or b) identifying issues with a plan you are ready to execute for backing out immediately.
All techies who are held accountable for the stability of the platform and application’s and services will try to implement some aspect of code freeze. Every business person who wants to be nimble and responsive and dare I say the word agile will push back on the whole concept of code freeze.
Make a choice. Take the time to freeze, confirm the build, deploy, and confirm the deployment. Or take a risk and live with the fall-out. Just make the decision with eyes wide open.
I’ve led stability programs in two Fortune 500 companies when system outages were severely impacting credibility and bottom line financials. One learning from both experiences was the price of good governance is almost always less than the price of a remediation program.
Keep talking about it. And listen to both sides to understand the real risks and options. The answer is always in the balance.