#helmfile - December 2018

December 2018
Daren
Darenalmost 8 years ago
Have you encountered issues with a chart that creates jobs? When running sync a 2nd time k8s errors with field is immutable
Shane
Shanealmost 8 years ago
@Daren This would not be a issue with helmfile or helm. Certain fields are not mutable in kubernetes. What field are you trying to modify?
Daren
Darenalmost 8 years ago
Im changing a value which is used as an ENV by the job
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
so we're deploying jobs regularly with our charts - mostly to run db migrations
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
we've not run into that
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
fwiw, our envs are referring to values in configmaps and secrets
Daren
Darenalmost 8 years ago
hmm maybe thats the issue
Daren
Darenalmost 8 years ago(edited)
Im doing it directly from values
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
:party_parrot: monochart :party_parrot: monochart :party_parrot: monochart
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
😃
Shane
Shanealmost 8 years ago
@Daren If you are creating a job directly via helm you likely should have it as part of a lifecycle hook.
Shane
Shanealmost 8 years ago
Jobs are not mutable and once it's create it should run and then complete
Daren
Darenalmost 8 years ago
Hm even when using a configmap it still fails on immutable
Daren
Darenalmost 8 years ago
I thought of doing init-container originally, but its tricky for this situation. Im deploying an official chart which requires that a couple scripts be run against the DB first. I wanted to avoid having to build a container just for that
Daren
Darenalmost 8 years ago
So I tried using a chart that runs a couple jobs using the official image.
erik
erikalmost 8 years ago
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
Here's a job being deployed several times a day that runs migrations
Daren
Darenalmost 8 years ago
Heres mine, its simple:
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ template "harbor-init.fullname" . }}-registry
  labels:
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: {{ include "harbor-init.name" . }}
    <http://helm.sh/chart|helm.sh/chart>: {{ include "harbor-init.chart" . }}
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: {{ .Release.Name }}
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: {{ .Release.Service }}
{{- if .Values.podAnnotations }}
  annotations:
{{ toYaml .Values.podAnnotations | indent 4 }}
{{- end }}
spec:
  template:
    metadata:
      labels:
        app: {{ template "harbor-init.fullname" . }}-registry
        release: "{{ .Release.Name }}"
    spec:
      restartPolicy: OnFailure
      containers:
        - name: registry
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          envFrom:
            - configMapRef:
               name: {{ include "harbor-init.fullname" . }}
          env:
            - name: PGPASSWORD
              valueFrom:
                secretKeyRef:
                  name: {{ include "harbor-init.fullname" . }}
                  key: password
          command: ["psql"]
          args:
            - "-f"
            - "/docker-entrypoint-initdb.d/initial-registry.sql"
Daren
Darenalmost 8 years ago
Im not doing the checksum annotations
Daren
Darenalmost 8 years ago
Does it matter?
Shane
Shanealmost 8 years ago
Nothing in a job's spec an change. If you want to do this on install I would recommend using a helm lifecycle hook for post-install
Shane
Shanealmost 8 years ago
with your current job if the image or tag changes it will cause helm to fail
Daren
Darenalmost 8 years ago
ohhh a helm hook, nice
erik
erikalmost 8 years ago
Shane
Shanealmost 8 years ago
^
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
thanks @Shane - i didn't know why @Igor Rodionov was doing this
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
but i guess that's why
Igor Rodionov
Igor Rodionovalmost 8 years ago
@Igor Rodionov has joined the channel
Shane
Shanealmost 8 years ago
The hooks allow you to delete the job so when it runs again it can run the job again. if you don't delete the job it will fail if the job ever changes
Shane
Shanealmost 8 years ago
Due to the immutability
Daren
Darenalmost 8 years ago
someone get this man a 🍺
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
🍻
Shane
Shanealmost 8 years ago
I always accept beer as payment
Daren
Darenalmost 8 years ago
If you only want the job to run once, do you need the delete hook?
Daren
Darenalmost 8 years ago
do not run on any subsequent sync
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
then i thikn you want
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago

"<http://helm.sh/hook|helm.sh/hook>": pre-install
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
or basically, something to indicate that it shouldn't get triggered on upgrades
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
i don't know the hooks off the top of my head
Daren
Darenalmost 8 years ago
👍️
Igor Rodionov
Igor Rodionovalmost 8 years ago
@Daren you can find more info here https://docs.helm.sh/developing_charts/#hooks
Igor Rodionov
Igor Rodionovalmost 8 years ago
oh.. you found already
Daren
Darenalmost 8 years ago
Is there a way to bundle some resources into an official chart?
Igor Rodionov
Igor Rodionovalmost 8 years ago
what chart?
Igor Rodionov
Igor Rodionovalmost 8 years ago
monochart ?
Daren
Darenalmost 8 years ago
Daren
Darenalmost 8 years ago
I mean does helmfile allow you to attach another resource to a chart you are installing
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
helmfile is basically a Makefile for helm 😃
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
so if you can do it with helm, you can do it with helmfile
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
now, i know i'm going to sound like a broken record - but...
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
we're doing something similar. let me explain.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
so we install granfa from official chart repos
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
but then we need some additional configmaps
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
in our helmfile we have one release for grafana and another release for the configmaps which get installed using our monochart
Igor Rodionov
Igor Rodionovalmost 8 years ago
you can not inject jobs into existing chart.
We have workaround of that with helmfile and monorchat
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
we have an example - will show you
Daren
Darenalmost 8 years ago
Sounds like my use case. Im installing the Harbor chart. But I need to run a script against the DB it will use beforehand
Igor Rodionov
Igor Rodionovalmost 8 years ago
This is example how we "extend" official nginx chart with additional resources
https://github.com/cloudposse/helmfiles/blob/master/helmfile.d/0320.nginx-ingress.yaml#L156
Daren
Darenalmost 8 years ago(edited)
With both your examples, you are ordering the helm releases and using wait: true?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
yea, concurrency 1
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
wait true
Igor Rodionov
Igor Rodionovalmost 8 years ago
and release order playes role
Daren
Darenalmost 8 years ago
cool, we have the same approach
Igor Rodionov
Igor Rodionovalmost 8 years ago
also I sugget to deploy it with 2 bash commands
Igor Rodionov
Igor Rodionovalmost 8 years ago
like
Igor Rodionov
Igor Rodionovalmost 8 years ago
helmfile --selector component=job
helmfile --selector component=harbor
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago(edited)
well, or have a helmfile.yaml which includes the helmfiles in the right order
Daren
Darenalmost 8 years ago
My ordering is less sensitive
Daren
Darenalmost 8 years ago(edited)
If the pre-job runs at the same time or after its fine. Harbor will eventually recover once it does run
Igor Rodionov
Igor Rodionovalmost 8 years ago
then helmfile fits perfect
Daren
Darenalmost 8 years ago
Agreed
Daren
Darenalmost 8 years ago
@Erik Osterman (Cloud Posse) check out harbor sometime, its working out very well
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
that's great to hear
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
i want to add more security/compliance features to our solution so it sounds like something good to have there.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
did you go enterprise?
Daren
Darenalmost 8 years ago
Its open source...
Daren
Darenalmost 8 years ago
Docker registry & Chart museum: https://goharbor.io/
Daren
Darenalmost 8 years ago
backed by CNCF
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
yea, my understanding was though to get access to a larger library of vulns, you needed to go ent
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
though I met those guys ~2 years ago at a meetup and a lot might have changed
Daren
Darenalmost 8 years ago
I was not aware of an enterprise version
Daren
Darenalmost 8 years ago
It uses clair (https://github.com/coreos/clair) for vuln.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
hah, my fault
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
all these nautical product names mixed me up
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
this is an open core + enterprise
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
don't know how it compares to harbor vs clair
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
isn't harbor providing redundant capabilities to twistlock which you're already using?
joshmyers
joshmyersalmost 8 years ago
https://github.com/future-architect/vuls is pretty nice for a lightweight vuln scanner
Daren
Darenalmost 8 years ago
Ive added a chart repository that requires auth + mfa to helm. However every helm operation involving the repo (install, repo update, push) trigger an MFA verification. Is there a way for helm to cache a session?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
helm-s3
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
or MFA with harbor?
Daren
Darenalmost 8 years ago
In this harbor
Daren
Darenalmost 8 years ago
Im wondering if its because harbor uses LDAP for auth
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
oh, yea, in that case no clue.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
@jdolitsky probably could answer it though
jdolitsky
jdolitskyalmost 8 years ago
@jdolitsky has joined the channel
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
@Shane what do you think about this? https://github.com/roboll/helmfile/issues/347
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
our use-case is we write a lot of helmfiles for our clients, but i'd like to reuse them while version pinning
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
Shane
Shanealmost 8 years ago
got ya, one issue I would see with it is helmfile is not really self contained.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
hrmm
Shane
Shanealmost 8 years ago
everything would have to be in the helmfile without any external file references.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
yes - it would be limited in that regard
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
but those external file references in our case are usually the non-portable settings
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
things like URLs, configuration files, keys, etc
Shane
Shanealmost 8 years ago
got ya
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
this is the most common use-case though for us
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
everytime we deploy an app for some staging environment we copy and paste this over
Shane
Shanealmost 8 years ago
I'll check it out I got to jump off the wife wants to eat.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
haha, no rush.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
basically i don't want to copy all these snippets everywhere which add a lot of technical debt.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
(heading out myself! ttyl)
Daren
Darenalmost 8 years ago
Anyone know how to remove a lifecycle hook from a resource and not get the helm state out of sync? This chart https://github.com/helm/charts/blob/master/stable/sumologic-fluentd/templates/secrets.yaml#L11 has a pre hook on the secret, which is not needed and causes issues. Removing the hook and performing a sync causes helm to error out thinking the secret does not exist.
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
hrm... can you create a stubsecret deployed as a standalone release using helmfile?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)almost 8 years ago
.... not sure if i full comprehend the problem