can someone please explain to me how to implement different stages (dev, testing, prod) in git or gitlab to implement a usefull and low risk CI/CD driven infrastructure as code environment? i only started using gitlab this year and im kinde self taught in that matter but im good with linux and have been a sysadmin for a few years now.
lemme explain where im coming from: im managing a serverfarm of around 500 on-premise virtual machines for my employer. we are using zero cloud based applications or systems. all systems are only for internal operations, no external customers in that regard.
in the last few months i implemented some playbooks that automate a portion of the servers via ansible. after some time i could centrally manage the servers, deploy new machines or rollout changes on various network devices with these playbooks.
Next step: i integrated the playbooks and roles that i created into a self hosted gitlab instance. some pipelines later i was able to execute playbooks and the self written roles via gitlab ci/cd. this allowed me to schedule playbook runs and allows multiple people to make changes with approval process.
but now comes the problem: i have no idea how to cleanly separate dev, testing and prod in playbooks and roles.
- how to implement different stages for my environment. at the moment im using centrally managed pipelines that include and exclude servers depending on their group membership (dev, testing, prod) depending on stage of the pipeline. the pipelines than use different ansible-playbook commandlines to limit against these groups. but this feels kinda wonkey and im not happy with it.
- i only have one inventory and set of group/host vars. at the moment dev, test and prod are only separated via group membership.
- all my playbooks run everything via roles that are stored in different repositories. these roles get imported on every pipeline run according to a requirements.yml which belongs to the playbook.this doesn’t support different stages of the downloaded roles at all. sure, i can define which branch of a role should be downloaded, but this sounds pretty bad to manage. it might be easier to incorporate the roles as submodules maybe?
- whats your workflow / way to propagate changes of roles and playbooks from dev to testing to production and how do you make sure that roles get reexecuted on all hosts that got the role in the past?
- if i create a playbook, lets say deploy_nextcloud which uses the role ntp_settings to configure ntp before installing the nextcloud. later i change the ntp role or a variable in group_vars/all.yml that was used by nextcloud. how do you make sure the change gets propagated to the server created by “deploy_nextcloud”?
would love to hear some of your experiences on that matter.
Comments
Apparatus · [2021-09-23] · 12 points
In regard to how to manage your environments, I think you should promote your playbook code through your environments base on git branches. So any new commits would first enter your
developmentbranch, that would kick off a job to apply your updated playbook to your development servers, or even first make a new set of development servers via terraform and then apply the playbook.You can group the servers in your different environments in your inventory file by the branch name. With the built-in variable in gitlab ci/cd
${CI_COMMIT_BRANCH}you could dynamically select the correct group in your inventory file for the ansible run with a single, reusable ci job.If you’re feeling extra ambitious do some automated testing on the systems. You can use something like inspec for this.
After your test job completes, or you’re done manually looking things over, merge the branch in to your next higher branch/environment and repeat until you’ve reached production.
This ensures that you’re always using the same code for each of your environments, and it’s only the inventory configuration and branch that distinguishes which environment it’s being applied to.
[deleted] · [2021-09-23] · 3 points
In regard to how to manage your environments, I think you should promote your playbook code through your environments base on git branches.
Here begins my first problem. Most of the time changes happen at role level not inside the playbook. So changes in a role won’t trigger the playbooks that are used to assign them to a group of hosts.
Do you you pre-configured branches like dev, testing, production for every role and playbook?
At the moment my default branch “master” executes against prod, everything non default branch only against hosts in group dev or test.
Im using a rule that checks if CI_COMMIT_BRANCH is master. Only then the playbook runs against prod. This kinda works but feels wrong, because the test branch of a playbook uses the same roles as would the master branch of that same playbook.
So when I’m changing a role, I need to either create a second branch for that role and change the requirements of the test branch of the playbook to point to the new role branch. But in that case I can’t just merge test to master for that playbook because of the different branches the requirements is pointing at. Or I just change the master branch of a role and avoid executing the master branch of a playbook until the changes executed successfully against test.
Apparatus · [2021-09-23] · 2 points
I think you should refer to your roles by a version tag. That way when you’re consuming a given role within a playbook, and promoting that version through your environments, you’re always getting the same role-code. I.E. it won’t shift out from under you as the role continues to be developed separately in its own repo.
You have a two-step process then. When you make changes to the role-code, you cut a new tag. Then you update the playbooks that consume that role with the new tag, by promoting that tag-change from dev to test to prod. All the higher environments are locked to the previous version of the role, until the commit with the role’s new tag has been successfully promoted from the lower environments.
This also allows you to keep playbooks that consume a given role at older versions of the role, if they’re not ready to be upgraded yet.
[deleted] · [2021-09-22] · 6 points
Up for visibility. I’ll be starting a role as a consultant in two weeks and I was hired to do this actually 🙃 I know a lot of ansible and worked with pipelines but never configured one myself. I don’t think it should be that hard but it’s good to have more info.
AT_DT · [2021-09-23] · 3 points
We do this and one risk is leaving the merges to the next branch/env for too long. And/or the forced inclusion it imparts. We end up with changes in a stage branch we don’t want to go to prod, but then other issues come up that DO need to go to prod. Makes for some messy merging and tracking of what’s where.
[deleted] · [2021-09-23] · 2 points
I honestly didn’t want to go the route of every role is it’s own repo for the fact that collections exist now, I don’t know if the structure is scalable that way to my liking (I know I sound like a jerk here). So. Instead. I created my own API endpoint. This handles the GitHub Payloads. Once I do a pull request from my branch into production, any files that have changed that are yml files go through a lint. After that they are dry ran just to make sure Ansible has all of the correct ‘includes’ and the only issue is it doesn’t know what hosts to really run on. Once it has reported back that is good. The final approval will include ‘Ansible run <playbook>’ that an engineer will write into GitHub comment section. In Ansible tower I have job 10 for example that is setup to run all jobs, but the CI/CD will make a job of that job workflow and put in <playbook> into the tags section. This will allow for dynamic role increase without needing to actually make a new API endpoint for every role. And only jobs in that job 10 that are in tags, be the only jobs that can be ran in production and are ‘battle ready’. So yeah. I just use branches vs environments, but I’m also into networking, so everything is production. The best I can do right now is A/B side testing.
RIPTrainJudo · [2021-09-23] · 2 points
The following is what we ended up doing at a place that was converting some Salt into Ansible. The structure of the repos is the same as the one used in their Salt content which includes all logic for app release automation so it is familiar to them and I find it works well after seeing the smoothness of the releases done with Salt in the background. Kind of a don’t fix what isn’t broken thing (except one thing):
For environment specific details it is one repo per env kept in a project. For roles it was originally one repo for app content and one repo for system specific content. One branch per environment. We started moving things over to one role per repo, or a collection if there is too much related logic to sanely keep in a role. These roles are git tagged for release versions to allow projects importing the roles flexibility on version while protecting environments. With the original implementation we brought from Salt there was a central merging branch like app-central and sys-central, the lowest env was merged into this after a deployment and then the next highest env was merged from the central branch as you moved up the envs.
For deployment it is Ansible Tower so that is how the deployments are triggered from GitLab into the environments after approvals. Having static inventory isn’t fun and if you can get good filtering from a dynamic inventory the above works great.