During my career in software development I've encountered many different corporate cultures. A common one is what I like to call a "culture of blame". In blame cultures, everything revolves around finding someone else to blame. When a project is late, product managers blame developers. When there are bugs, developers blame QA. When tests fail, QA blames development. When there are production problems, operations blames developers. When QA can't get test environments to work, QA blames operations. When software is pushed before it's ready, developers blame product managers. It's all about plausible deniability and a "circle of blame".

Where I work now has a very different culture. I call it the "just fix it" culture. If you break the build, "just fix it". If tests fail, "just fix it". If there's a production issue, "just fix it". You see where this is going... Instead of finding someone to blame, or finding ways to defer the problem, "just fix it", and move on. There's no need for retrospectives (where everyone sits in a meeting room deciding who to blame), nor really any need to discuss it beyond what is required to fix it and get back on track.

Whether it's a "culture of blame", or one of "fix it", the reality is that the same person has to fix it. By skipping all the crap in the middle, we go from problem to solution without delay and diversion. Instead of pretending that we can be perfect, the trick is to accept that things break. It's not that we want things to break, it's about accepting reality. The poor person didn't mean to break it - he or she is mortified about it, in fact. By accepting a "just fix it" culture, we just ask that they fix it, and move on. In a "culture of blame" we end up discouraging this person from being productive because blame is intimidating, and it scares people into being afraid to commit. When people are afraid to commit, they stop being productive.

In the end, software development is all about productivity. You want developers to be productive. You want QA to be productive. You want operations to be (you guessed it) productive. How do we make everyone productive? It's not by pointing fingers, and it's not by assigning blame. We all know when we screw up, and we don't like doing so. Deep down inside, we all fundamentally want to succeed. This is pretty obvious, right? Nobody likes failure. So why do so many companies build a culture around failure instead of success? It's a good question, right?

So what's the secret?

Well, from the above, it's pretty obvious - stop playing the blame game. If Fred checks in code that breaks the build, Fred should know about it as soon as everyone else does. If he doesn't have visibility into this, then you need to improve your build system and notifications. It's perfectly OK to tell Fred "hey, did you see that the build failed?". Fred didn't plan on breaking it, and he wants to fix it. Notify, but don't blame. Fred will fix it. If Fred's not around, then someone else has to fix it. This is the exact same situation that you could be in with a "culture of blame" anyway, and the end result is the same, that somebody will need to fix it. Again, instead of worrying about who to blame, let's "just fix it", and move on.

An agile release process is key as well. If you only release a few times a year (or less), then blame is more likely, because the release process is expensive and infrequent. We release every week. If your code isn't ready, then your code doesn't go out this week. It's that simple, and the worst case is your code ships the following week instead. Very rarely we have skipped a release because the release branch has had issues, but again we don't blame anyone - we "just fix it" and release a different day, instead. Having the ability to do rolling upgrades without bringing down the site is key.

Simple, right?

It is simple, but there's complexity in the details, and it requires discipline and support from management - and an acceptance that things do go wrong. And yes, it requires that your corporate culture change. Are you mature enough to handle it? That's all up to you!