I’ve spent a number of years working in a few DevOps/DevSecOps roles to transform organizations into new ways of doing business. I love automation and cloud, and I particularly love infrastructure as code. DevSecOps transformation is not only about the tech, but it’s also about the people. Maybe even more so.
I’ve always said… tech is about people, not tech. Dev(Sec)Ops especially drives that point home. Part of this means that you need to have the freedom to organically grow your DevOps processes and adjust them when necessary. We started this current project under the mindset that any one terraformed AWS account would be equivalent to a single git repo. Over time, we succeeded in getting other people to adapt to infrastructure as code. We developed many branching mechanisms and whatnot to support deployment of resources into the same AWS account, but under different “environments” without using workspaces. Don’t ask, it was a customer requirement. It’s a customer requirement that we’re changing, but it’s a slow roll because it requires a re-architecture of the Amazon environment.
Anyway, we had a number of people making changes to the single large repo. This started to become a problem. There are a few files in the repo that contain important directives - like, which Terraform resources belong to which environments based on the git branch, etc. This meant that a lot of edits coalesced into those files and wham, you’ve got merge conflicts and overlapping changes. It started to suck.
It started to suck a lot.
Now, we’re shifting our model to break up the projects into multiple git repos/Terraform deployments. It seems daunting to have a lot of Terraform state files at first, but using the benefits of working multiple projects without overlapping deployments through CI/CD has already shown that this was the right decision. I was resistant to this at first, but the benefits of keeping the wheels greased far outweigh the negatives. People are able to get work done without worries of overwriting other infrastructures. However, this does get a little dangerous when some state files are not aware of other resources in the same environment. I suspect that’s the next process challenge we’ll face - and it’ll happen when someone accidentally wipes a server being managed from another Terraform repo. If it happens, that’s fine, we’ll just have to organically adjust the processes again.
Stay fluid, folks.