What is wrong with DevOps?

Caveat

This is an opinion based on my observations and does not have to be true within your organization.

Where I come from

For many many years I was a frontend developer professionally and hobbyist when it came to linux servers because I also was interested in running my own software as well as broadening my spectrum of tech knowledge. I was running servers for the games I’ve played, ran projects of mine, experimented with video streaming, ran mail servers, chat servers and much more. At work I became to be known as the guy who could take a new task and somehow make it work, like setting up a whole solution for streaming talks to a selected audience including professional audio and video gear, streaming server, screen grabbing and all the cool stuff that made our companies presentations to stakeholders look professional. Also I took care of deploying our software and did what other companies have a full team dedicated to do so.

While working at a startup like business unit of a larger company, I had the opportunity to break with their established and quite bloated procedures of doing things. After all, the sole intend of our unit was to create prototypes of business ideas, present them to possible stakeholders and investors and collect real world user feedback on the business idea. If it got enough traction and financial commitment, everything would be rewritten by other teams using the established channels, tools, frameworks and such, or being implemented in the big scale app that already was there.

My ordeal started with me having to get a signed waiver for every bit of open source software I was going to use. This quickly was ended after the second time I printed out the full list of node modules for the Ionic Framework we’ve decided on using for our first prototype. Then I had to write proposals for using first Github for our code, since we needed some freedom in how we operate, not having to write a ticket every time we wanted a new repository and having to wait for a few weeks to get that one processed. A year later, we switched from Github and Travis to a self hosted Gitlab instance in order to stay even more flexible but also address some corporate security concerns we had having our code out there on a platform we didn’t control. This lead us to the possibility of creating refined deploy and testing pipelines with a little more flexibility and also I got to play with some cool new to me tooling whilst having the freedom of not having to go trough some ticket mumbo jumbo, then running over to the guys responsible for working on my tickets and rubbing their bellys in order to get stuff done without a month of delays for every step of the way.

Hey super, we’re super flexible now, we have a reverse proxy doing our SSL termination, creating subdomains as we need them, we have a way to deploy our application, our merge requests and spin up deployments for A/B testing and integration testing as we needed it. This was not done with Kubernetes as those were only prototypes with a couple of users, but with Docker-Compose and some nginx reverse proxy tool that automatically wrote configs and got certificates. All was good and dandy. But was it really?

When I left the company because the work environment grew toxic more and more, it fell apart rather quickly. Why? Because I was the person driving this, enabling the handful of developers a self service like infrastructure, enabling them to do ‘DevOps’. The other devs focused on their area of expertise, getting features ready to ship on a tight schedule. No one cared to pick up knowledge on how to maintain and enhance that infrastructure, so they phased out of the rapid prototyping and moved to a product scope in the process.

The myth of full stack

Every time I read companies looking for fullstack developers, I cringe a little. This concept of one individual being a good frontend developer, a good UI engineer and a good backend developer is a theoretical construct, I’ve only seen two persons in my professional career fulfill.

Each of those things is highly complex. So much so, that mastering one of those topics will take so much time and brain power, that there is hardly space for more than one of those.

Sure, as a frontend developer I had a good understanding of backend development in general, and if you would have shown me some Rails code, I would likely understand it. If you start to show me Java code, I start to feel uneasy, since I simply lack the experience of this highly opinionated language and never looked into it out of a strong distaste of bloatiness (yes, it’s ironic to write this, while having worked on Javascript projects with ’node_modules’ folders that included hundreds of thousand packages).

And while I was quite good with CSS and engineering UI elements and custom, highly flexible UI frameworks, I was not like my colleauge who not only never seemed to have to look up how the new fancy stuff works, but also knew all the pitfalls of different and sometimes arcane browsers.

In todays world, where everyone uses some sort of UI framework, and the Internet explorer really is mostly dead and you can get along supporting the major two rendering engines, this is not as important as it used to be.

Now my therory is: You can only be good in two things, master of one, and be okay in some more, depending on your career, your desire to learn a new discipline or previous job requirements.

Now, you have UI engineering, you have frontend engineering, backend engineering. What about databases, automation, containerization, networking, operations, backups, security, infrastructure and all those other things?

Another point to consider: Labour costs

Having a new person joining a team is a costly endeavour. Not only will that new person require extensive onboarding in which time the productivity of that single person is greatly reduced in comparison to someone who is on the project for a long time. In addition to that, experienced team members are giving focus on the onboarding as well, since they want that new person in the team to be up to speed, asap. The more complex your tech stack and your way of doing things, the longer this period will last. Even if you only hire (expensive) senior developers, and even if they know the tech stack you’re using, the tooling around your product, the workflow, the way you write tickets, the special solutions in your code, everything takes time to learn. Now throw that Ops part on top. There is no industry solution for doing things the “right” way. Every solution is taylored to the problems they want to solve, taylored to your workflow, your product workflow and so on. This inflicts an additional cost to your teams productivity.

Following the DevOps approach, you have to know this stuff as well. Can you really become good in this wide field as well without sacrificing your other skills, stay up to speed with the latest Javascript frameworks, or best practices and new developments in other areas, while also not working 100hrs a week and possibly keeping any form of work life balance and not burn out?

Hey guys, now we’re doing devops

I’ve seen this in a couple of projects and companies. There is some form of excitement in the beginning. The team is happy they won’t have to deal with those grumpy admins anymore, and possibility have less limitations in their infrastructure.

But very suddenly the initial excitement dwindles. Not only has someone to keep everything running, but also someone has to learn new things. Usually you’ll find that person who is willing to get on task and learn the new tools and techniques needed to accomplish the goal, but soon finds himself/herself alone in front of a new techstack, with his teammembers and project managers breathing down their necks, asking why this all takes so damn long. Let’s call this person Alice. She will accompany us trough this story for a little longer.

Alice is a developer, not an operations person. She always looked a bit beyond her Ruby on Rails world, but never was on the Ops side. Shure she knew how to SSH into a machine and restart the deployment, or how to create a docker container. But she didn’t architect pipelines, nor has any kubernetes or orchestration knowledge. Alice starts to read up on Kubernetes, and is overwhelmed by all those new concepts.

Alice then goes around and looks for other projects within the company to have successfully migrated to that new infrastructure or having implemented that new build/deploy pipeline, because she needs to finish. Stuff gets copy pasted, and voila, you end up with something that kind of works. (To my surprise I still find artifacts of my old code in several repositories. Code that was not the best and already has undergone several refactorings. But due to the copy paste, “let’s get stuff done” mentality, those refactorings never got incorporated and most likley never will).

Now you have one unsatisfied person in your team. The other team members are unwilling to get involved since the personal benefits are not that high, and project management already was unhappy with the outcome.

Your bus factor at this point in time is near one. It isn’t one, since the code copy pasted from that other project suggests that there is someone else within the company who knows how this works, but within your team it’s one. And that poor person (Alice) now is the teams devops person.

Since everything has to be reviewed before a merge, Alice asks for a coworker to get involved. The changes are massive and not even Alice fully understands what happens. But it works, doesn’t it? The reviewer understands way less of the new technologies than Alice, but they need to get this on the road, they’re already four weeks behind schedule since the topic was heavily underrated.The review isn’t very fruitful and the code isn’t the best either, but hey it works. Everyone can go back to what they were originally doing, since there’s a deadline to hold.

However Alice is a smart engineer and starts to get more and more involved in understanding what she had copy pasted, and starts to improve it from time to time.

Fast forward 6 months: The deployments fail. Alice since has moved on to a new project and the poor person who did the code review is now the new devops person responsible for fixing the failed builds, while production is down and the team urgently needs to push that hotfix.

I could paint this picture further and further.

Points are:

• There is a reason why ops teams exist. The expertise required is not trivial.
• DevOps is not a one or two person thing. The whole team ideally needs to know how everything works, leading to the problem of mental overload of the fullstack developer, even if an ops team in the background takes care of maintaining the infrastructure and giving guidance in using it. Someone has to adapt the project specific configurations/pipelines/etc.
• Ops only gets noticed if something fails. It’s a job that often does not get much appreciation. As a result, it often is overlooked in planning and budgets as well.
• Even if every team has one or two dedicated DevOps people and they interact over team boundaries, you’ve the same situation as before. You have devs working on their features. And you have ops people trying to fulfill the demands of those devs, while still trying to be a good developer.

I believe that the concept of DevOps is broken inherently, just like the concept of full stack developers is broken.

Understanding the ins and outs of operating production systems is nothing that can easily be combined with being a great frontend/backend/whatever developer. A frontend developer who knows a little backend development and can do minor adjustments to the API he is consuming with his frontend? Sure. A Frontend developer who can write a configuration file for a deployment? Sure! A backend developer who can change text or color or a button or add a new form field to reflect his changes in the backend?

But a backend developer who can run and maintain a kubernetes cluster including databases, caches, provision backups (and testing them!), while also optimizing the pipeline, updating libraries in the complex frontend application while being able to slap on a refactoring of the pattern library? Seriously, how likley is it to find such a person in your company? Let alone ideally having only devs with those capabilities?

There is no way!

How do we fix this? Where do we go from here?

One solution is to have a dedicated ops team that provides infrastructure and services such as databases, caches, and other services that can be provisioned trough some form of configuration. Those we call Cloud Engineers.

Another team is dedicated on creating and maintaining good, performant and usable pipelines. This team provides patterns and jobs that can be used by project teams to not having to reinvent the wheel. Lets call them DevOps Engineers since this seems to be one definition quite a few people can agree on.

And then we have the developers. They write code, right?

Links: