#gitops - April 2019

gitopsArchived
88 messages

Discuss continuous delivery of infrastructure
Archive: https://archive.sweetops.com/gitops/

Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
(video will be posted later)
btai
btaiover 7 years ago
So im trying to deploy to my k8s cluster w/ helm via my ci tool and I’d like for those helm deploys to kick off based off merges into special branches. For example, merging a PR into develop would kick off a helm upgrade to the dev cluster and merging a release PR into master would kick off a helm upgrade to the prod cluster. How are those of you following this pattern handling the fact that helm doesn’t redeploy pods using images on a specific tag (develop) even if there is a more recent version of that image?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
we've used a number of strategies
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
for develop you can use helm upgrade --recreate-pods option 😃
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
or just always tag yur docker images also with a git ref
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
then use the git ref for the image tag
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
--recreate-pods will cause a brief outage
btai
btaiover 7 years ago
so your repo has a tag per commit?
btai
btaiover 7 years ago
😥
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
we tag every docker image with a git ref
btai
btaiover 7 years ago
ah
btai
btaiover 7 years ago
i misread that
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
that way there is a 1:1 relationship between git and docker
btai
btaiover 7 years ago
so when you merge into master, that will re-build the docker image meaning that you would be technically deploying with a different image (although should be identical)?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
yes, but our process is different
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
we merge to master, we build
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
especially since we usually squash merge
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
we use release tags to deploy to environments
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
release tags therefore only retag a docker image. no rebuilding required.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
master is continuously delivered to staging
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
release tags go to production.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
1.2.3-foobar goes to the foobar environment
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
e.g. 1.2.3-prod
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
1.2.3-preprod
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
1.2.3 goes no where
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago(edited)
then for PRs we deploy each one into a new namespace on the staging cluster
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
e.g. pr1234 of the example repo gets deployed to pr1234-example namespace
btai
btaiover 7 years ago
so for your prod deploys, they are not being triggered via git
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
they totally are
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
100% git driven
btai
btaiover 7 years ago
ohh
btai
btaiover 7 years ago
youre saying
erik
erikover 7 years ago
btai
btaiover 7 years ago
by tagging a commit, you are pushing to specific environments?
btai
btaiover 7 years ago
so when you push that tag, it triggers the deploy
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
yes
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
retag the image
btai
btaiover 7 years ago
thank you, i believe thats the step i was missing
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
deploy it
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
in our case, we use github
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
so we use the UI for releases
btai
btaiover 7 years ago
i do to
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
but essentially it's tags
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
releases also have webhook events
btai
btaiover 7 years ago
so you create a release, which triggers a deploy in codefresh
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
we register those with codefresh
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago(edited)
yup
btai
btaiover 7 years ago
nice thats what i’ll do
btai
btaiover 7 years ago
how do you like codefresh?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
and then you can throw in an approval step
btai
btaiover 7 years ago
i believe you guys are big fans
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
so it queues it up for deployment, but someone higher up has to click approve
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
yes, we're big time fans of #codefresh
btai
btaiover 7 years ago
approval step is a codefresh thing or github?
btai
btaiover 7 years ago
codefresh correct
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
approval step is a codefresh step
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago

ask_for_permission:
    type: pending-approval
    title: Deploy release?

btai
btaiover 7 years ago
nice nice
btai
btaiover 7 years ago
we’re on circle right now, which isnt my favorite
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
yea, circle definitely has the marketshare
btai
btaiover 7 years ago
but i believe i can perform all of those still
btai
btaiover 7 years ago
with circle
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
and the two have been converging on functionality
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
but codefresh has tighter k8s integration and native helm support
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
plus kanban boards so youo can view your releases and what stage they are in
btai
btaiover 7 years ago
yeah ive built my own helm deploy script to run from circle
btai
btaiover 7 years ago
thats why i was interested in drone.io but right now its more affordable to pay for circle than run our own 3 node k8s cluster for drone
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago(edited)
yea, i dn't like hosting CI/CD
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago(edited)
it's foundational infrastructure, that when everything else is done right (IaC) it's just easier to treat the CI/CD as an axiom; it just exists and we don't need to worry about how.
btai
btaiover 7 years ago
same, but drone does run jobs as native k8s jobs which removes the need for managing agents which is nice
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
that's nice
btai
btaiover 7 years ago
but yeah, thinking about our CI/CD platform having uptime issues during a client-facing outage would suck
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago(edited)
i feel like we got enough to manage.... kiam, cert-manager, prometheus, grafana, teleport, kibana, fluentd, external-dns, keycloak, etc. . and for those there is no hosted option.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
but for CI/CD there is.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
😃
btai
btaiover 7 years ago
haha yeah
btai
btaiover 7 years ago
sysdig datadog 🤷‍♂️
btai
btaiover 7 years ago
but i get it
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
yea, true
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
sumologic, splunk
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
but daaaaaaang the prices $$$
btai
btaiover 7 years ago
yeah they are not cheap
btai
btaiover 7 years ago
thanks for the help, answered with the exact solution i was looking for @Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)over 7 years ago
thanks @btai! glad I could help