#helmfile - April 2021 | Slack Archive

78 messages

Questions and discussion around helmfile https://github.com/roboll/helmfile and https://github.com/cloudposse/helmfiles
Archive: https://archive.sweetops.com/helmfile/

Mar 2021

April 2021

May 2021

Balazs Vargaover 5 years ago

any idea why it does not work like values ?

skipping missing values file matching "../config/{{ .Release.Name }}/patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml"
skipping missing values file matching "../config/{{ .Release.Name }}/merge.yaml.gotmpl"
skipping missing values file matching "../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml"
skipping missing values file matching "../config/app/values.yaml"
Successfully generated the value file at ../config/app/values.yaml.gotmpl. produced:

My templates settings are :

templates:
  default: &default
    missingFileHandler: Debug
    values:
      - ../config/{{ .Release.Name }}/values.yaml
      - ../config/{{ .Release.Name }}/values.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}.yaml.gotmpl
    secrets:
      - ../config/{{ .Release.Name }}/secrets.yaml
      - ../config/{{ .Release.Name }}/secrets.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-secrets.yaml
    strategicMergePatches:
      - ../config/{{ .Release.Name }}/merge.yaml
      - ../config/{{ .Release.Name }}/merge.yaml.gotmpl
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-merge.yaml
    jsonPatches:
      - ../config/{{ .Release.Name }}/patches.yaml
      - ../config/{{ .Release.Name }}/{{ .Environment.Name }}-patches.yaml

PepperKickover 5 years ago(edited)

Hi everyone, this is probably me missing something simple but for some reason I am not able to use the new "waitForJobs" config.

I added it to the release section as follows

releases:
  - name: test
	...
    wait: true
    waitForJobs: true
    timeout: 60

helmfiles:
	...
	{{ if eq .Environment.Name "cluster" }}
	- path: environments/cluster/test.yaml
	{{ end }}

environments:
	cluster:

The helmfile apply command fails with

...
[1] in /home/helmfile/helm-installer/resources/helmfile.yaml: in .helmfiles[1]: in environments/cluster/test.yaml: failed to read test.yaml: reading document at index 1: yaml: unmarshal errors:
[1]   line 7: field waitForJobs not found in type state.ReleaseSpec

Helm and helmfile version

bash-5.0# helm version
version.BuildInfo{Version:"v3.5.0", GitCommit:"32c22239423b3b4ba6706d450bd044baffdcf9e6", GitTreeState:"clean", GoVersion:"go1.15.6"}
bash-5.0# helmfile version
helmfile version v0.138.7

mumoshuover 5 years ago

Does anyone rely on helmfile's current behavior that helmfile -l name=foo apply NOT failing when foo had some needs to other releases?

https://github.com/roboll/helmfile/pull/1772 can be a breaking change to you so please chime-in and leave your comments if you have opinions

Jim Connerover 5 years ago(edited)

Good evening everyone. I’m trying to learn helmfile and am struggling to figure out why I’m getting the following errors. My directory structure is:

.(helmfile.d)                                                                                                                                                                                                                                                                                                                                                        
├── generic 
|   ├── helmfile.yaml
│   ├── 01-secrets-management
│   │   ├── dex
│   │   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                            
│   │   │   └── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                                
│   │   ├── oauth2-proxy                                                                                                                                                                                                                                                                                                                                 
│   │   │   ├── arm64-values.yaml                                                                                                                                                                                                                                                                                                                        
│   │   │   ├── secrets                                                                                                                                                                                                                                                                                                                                  
│   │   │   ├── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   │   └── wait_for_endpoint.sh                                                                                                                                                                                                                                                                                                                     
│   │   ├── vault-operator                                                                                                                                                                                                                                                                                                                               
│   │   │   ├── helmfile.yaml                                                                                                                                                                                                                                                                                                                            
│   │   │   ├── secrets                                                                                                                                                                                                                                                                                                                                  
│   │   │   │   └── vault-cr-secret-dec.yaml                                                                                                                                                                                                                                                                                                             
│   │   │   └── values.yaml                                                                                                                                                                                                                                                                                                                              
│   │   └── vault-secrets-webhook                                                                                                                                                                                                                                                                                                                        
│   │       └── values.yaml
|   └── common
│       ├── config.yaml
│       ├── environments.yaml
│       ├── helmdefaults.yaml
│       └── repos.yaml
└── helmfile.yaml

The helmfile.yaml in . is:

---
helmfiles:
  - "*/*"

The helmfile.yaml in generic is:

helmfiles:
  - "*"

The helmfile.yaml in 01-secrets-management is:

bases:
  - ../common/environments.yaml
  - ../common/repos.yaml
  - ../common/helmdefaults.yaml

Jim Connerover 5 years ago

finally:

Jim Connerover 5 years ago(edited)

The helmfile.yaml in dex is:

bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

## ************************************
## Start of DEX installation
## ************************************
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      tier: "secrets-management"
      app: dex
    chart: repo/helm-charts
    version: {{ .Values.dex.version }}
    values:
      - values.yaml

I’m running helmfile -e default --log-level debug lint from the dex directory. I get the following output/error:

Jim Connerover 5 years ago

The first error means that it’s not finding/using the common bases. I’m not understanding why. The last error is seemingly cascading from the first.

Jim Connerover 5 years ago

perhaps I’ve misread/misunderstood the documentation on bases but re-reading it it seems that helmfile is trying to render helmfile.yaml BEFORE rendering the other layers. Is that correct?

Jim Connerover 5 years ago

so if that’s the case then that means I can’t use anything defined in environments because it’s not yet read/rendered. Correct?

Jim Connerover 5 years ago

seems kinda useless to me if that’s the case.

Jim Connerover 5 years ago

FTR, if this is how bases works, it seems that it’s not very useful in this case and would be more useful in top-level helmfile.yaml files.

mumoshuover 5 years ago

We can't resolve that chicken-and-egg problem automagically. I may be still missing something but at glance what you wanted seems like

mumoshuover 5 years ago

bases:
  - ../../common/environments.yaml
  - ../../common/repos.yaml
  - ../../common/helmdefaults.yaml

---

## ************************************
## Start of DEX installation
## ************************************
releases:
  - name: dex
    namespace: {{ .Values.dex.namespace }}
    createNamespace: true
    labels:
      tier: "secrets-management"
      app: dex
    chart: repo/helm-charts
    version: {{ .Values.dex.version }}
    values:
      - values.yaml

mumoshuover 5 years ago

Notice --- so that the first part is rendered as a template to produce a YAML structure that includes bases. Bases should be loaded and the env values are loaded before rendering the latter part as a template

Jim Connerover 5 years ago

ahh! Lemme give that a try real fast.

Jim Connerover 5 years ago

nice!

Jim Connerover 5 years ago

not outta the woods yet…but that got me past that roadblock

Jim Connerover 5 years ago

how much time you got @mumoshu? Is it late where you are? I have questions, thoughts, and possibly ideas.

Jim Connerover 5 years ago

and I’m still very new to helmfile so I’m struggling to get a good working set of helmfiles here.

Jim Connerover 5 years ago

The error has changed…but I think this is good as it seems that the bases files are getting read now. These are the new errors.

Jim Connerover 5 years ago

This looks like possibly over-lapping values between my config.yaml or environments.yaml files.

Jim Connerover 5 years ago

actually, this looks like it’s reading environments.yaml which in turn references config.yaml within the directory where environments.yaml resides. This might be a referential problem with respect to … something.

Jim Connerover 5 years ago

taking another look at https://github.com/roboll/helmfile#paths-overview

Jim Connerover 5 years ago

Relative paths referenced in the Helmfile manifest itself are relative to that manifest

does this mean in sub-helmfile.yaml files that the path references in those files are relative to them or the relative to the top-level helmfile.yaml?

Jim Connerover 5 years ago

I’m trying to use paths in sub-files as relative to the sub-files.

Jim Connerover 5 years ago

wow.

Jim Connerover 5 years ago

that’s the case.

Jim Connerover 5 years ago

@mumoshu that’s a super-confusing thing methinks.

Jim Connerover 5 years ago

it’s also not very portable imo.

Marianoover 5 years ago

hey guys, im trying to add helmfile to a docker image (dockerfile) and not sure if https://github.com/roboll/helmfile/releases/download/v0.138.7/helmfile_linux_amd64 is the actual binary?

Balazs Vargaover 5 years ago

can we set a "security" to avoid misdeploys? I mean if I accidentally deploy from a wrong branch to a cluster. ? a warning message or interactive promt... and not -i because we can forget it. in default config maybe?

Jim Connerover 5 years ago

I have the following directory structure where helmfile.d has the helmfile manifests. The helmfile directory tree has my environments and this is from where I want to invoke environmental helmfile runs:


├── helmfile
│   ├── envs
│   │   ├── dev
│   │   │   └── cluster-a
│   │   ├── preprod
│   │   └── production
│   └── shlib
└── helmfile.d
    └── generic
        ├── 01a-network-and-proxies
        │   ├── ambassador
        │   ├── external-dns
        │   └── ingress-nginx
        ├── 01b-secrets-management
        │   ├── certmanager
        │   ├── dex
        │   ├── oauth2-proxy
        │   │   └── secrets
        │   ├── vault-operator
        │   │   └── secrets
        │   └── vault-secrets-webhook
        └── common

In cluster-a there’s a helmfile.yaml and config.yaml:

❯ \cat helmfile.yaml 
environments:
  default:
    values:
      - config.yaml

helmfiles:
  - "../../../../helmfiles.d/*"

The concept is that any specific configuration values for cluster-a are specified in the config.yaml inside the cluster-a directory. However, when I test invoking helmfile from this path, I get the error:

envvals_loader: loaded config.yaml:map[values:map[keyname:foobar]]
no matches for path: ../../../../helmfiles.d/*
merged environment: &{default map[values:map[keyname:foobar]] map[]}
helm:XVlBz> v3.4.1+gc4e7485
0 release(s) found in helmfile.yaml

err: no releases found that matches specified selector() and environment(default), in any helmfile

I don’t understand why.

Jim Connerover 5 years ago(edited)

one thing to note is there’s a helmfile.yaml in each directory which specifies helmfiles property:

.(helmfile.d)
├── generic
│   ├── 01a-network-and-proxies
│   │   └── helmfile.yaml
│   ├── 01b-secrets-management
│   │   └── helmfile.yaml
│   └── helmfile.yaml
└── helmfile.yaml

Jim Connerover 5 years ago

in helmfile.d/helmfile.yaml

---
helmfiles:
  - "generic/*"

Jim Connerover 5 years ago

in generic/helmfile.yaml

helmfiles:
  - "*/*.yaml"

Jim Connerover 5 years ago

won’t the attribute in helmfiles: cause helmfile to “walk up the chain” so-to-speak?

Jim Connerover 5 years ago

sigh, figured it out.

Jim Connerover 5 years ago

for posterity sake, the helmfile.yaml in the cluster-a directory had a misspelled directory name: - "../../../../helmfiles.d/*" (note helmfiles.d vs helmfile.d)

Jim Connerover 5 years ago

@mumoshu https://github.com/roboll/helmfile/issues/1045#issuecomment-820870785 might be interesting to you?

Jim Connerover 5 years ago

sigh. I’m really struggling to understand certain points in this documentation. I keep getting blocked on aspects of how to properly use helmfile and I either can’t find documentation to meet my needs or the documentation I think will help me either doesn’t make sense or is not as verbose as necessary to understand the full aspect of what the documentation is trying to point out.

For instance, I’m blocked on certain aspects of this project (tickets and questions are submitted) and I’m moving on to other aspects of this project where I’m not blocked. One of those aspects is the ability to logically select charts. It doesn’t work the way I’m trying to implement so I’m reading the documentation, which doesn’t make sense to me. I don’t understand the context of inheritance with respect to sub-helmfiles and the inherited properties. That point is not well spelled out. What do you mean by inherited in context? Everything? Certain properties? The selectors? I don’t understand.

The use-case I’m trying to solve is to simply use a selector to run helmfiles only identified by that selector. Everything will run exactly the same as if I didn’t use any selector except ONLY the helmfiles specified by that selector (or a negated selector) would run. The details of how that works should not be something the end-user is concerned about if it’s clear-cut as what I was hoping would be the case. I’m simply not understanding, which I think is because the documentation is a little too sparse, but there is a lot of documentation. I can’t put my finger on the problem. It might be me, which I can concede. It shouldn’t be this complicated to understand what software can do and I keep running into roadblocks here. I want to use helmfile because I believe it does what we need it to do. Some assistance would be greatly appreciated.

jedineeperover 5 years ago

Do I have to explicitly define a reference to each of the values I include under a named environment block into a release or is there a tidier way to bulk include the contents of the environment block ontop of the release values?

Jim Connerover 5 years ago

@mumoshu we can continue the thread here

Alex Gencoover 5 years ago

Hello! I'm new to helmfile and need some guidance on a use case. I'm adding a nested helmfile inside a monorepo-like helmfile repository. The parent repo so far doesn't use environments since all services are global, but my child helmfile benefits greatly from them, because it defines production and staging namespaces and deployments. My helmfile looks basically like this:

environments:                                
  staging:                                   
    values:                                  
      - staging.yaml                         
  production:                                
    values:                                  
      - production.yaml                      
releases:                                    
  - name: app
    chart: foo/app
    namespace: app-{{.Environment.Name}}
    values:
    - values.yaml.gotmpl

This works great as long as I specify -e staging or -e production during helmfile apply. But since the parent repo has no concept of environments, it breaks the deploy process.

I guess my question is, what's the best way to "inline" environments into the releases list? i.e. I would like something like this:

releases:                                    
  - name: app-staging
    chart: foo/app
    namespace: app-staging
    values:
      - staging.yaml
      - values.yaml.gotmpl
  - name: app-production
    chart: foo/app
    namespace: app-production
    values:
      - production.yaml
      - values.yaml.gotmpl

However, it's attempting to "merge" the values files, rather than use the first to fill in the template of the second. Is there a better way to do this, rather than just duplicating the mostly-identical values files?

PePe Amengualover 5 years ago

Hi, I'm doing hemlfile sync to change from a internet-facing alb to a internal alb and I tried destroy, sync and diff but even though there is changes the alb does not get get created, anyone had and issue like this before?

Jim Connerover 5 years ago

@PePe Amengual it seems like it would be a better use-case for terraform ¯\(ツ)/¯ — Do you have some helmchart you’re using to deploy your ALB?

Jim Connerover 5 years ago

@Alex Genco I’d try to help you but I’m in the same boat as you man. Still learning.

Eugene Korekinover 5 years ago

is there any way to ignore helm errors for a specific release and continue?

Andrew Nazarovover 5 years ago

Victor Farcic spread the word about Helmfile on his channel: https://www.youtube.com/watch?v=qIJt8Iq8Zb0

muhahaover 5 years ago(edited)

guys, is there any replacement for deprecated inclubator/raw chart ?

Leia Renéeover 5 years ago(edited)

Hi guys ,
Do you have a sample repository which installs kubernetes cluster auto scaler that works with TF 15.0 properly
I was using cookie labs which broken after upgrade.
Thanks
Leia
https://www.linkedin.com/in/leia-renee/

Jim Connerover 5 years ago

@mumoshu I hope that helps more (the example I just gave in the issue. I’m heading to bed now…
Note that the example manifests from upstream (vault) show no single tics around the property. I added those for testing. With or without, it seems to be valid yaml. You can apply it successfully with kubectl but helmfile doesn’t process it right. Thanks for your help.

Jim Connerover 5 years ago

gnight 👋🏼

mumoshuover 5 years ago

I've curated a list of important feature requests and planned changes https://github.com/roboll/helmfile/issues/1806
Please feel free to review and add your voices if you find anything interesting to you 🙂

esover 5 years ago

hi. is this a bug?

if I run helmfile lint against helmfile.yaml that contains

helmfiles:
  - environments/test1/helmfile.yaml
  - environments/test2/helmfile3.yaml

it fails with Error: repo not found it’s referring to a release located in helmfile3.yaml. if I remove the first line (environments/test1/helmfile.yaml) it works fine. btw, I am using helmBinary option to specify helm version in each of those helmfiles

esover 5 years ago

if I run helmfile -f environments/test2/helmfile3.yaml repos it works. shouldn’t it automatically fetch charts?

mumoshuover 5 years ago

@es Does it work when you run helmfile -f environments/test1/helmfile.yaml repos, too?

mumoshuover 5 years ago

Anyway, it would be great if you could provide provide us a more complete example for reproduction. At this point it's too hard to say if it's either a bug or not

Jim Connerover 5 years ago(edited)

@mumoshu I think I have a fundamental misunderstanding of how to use selectors for sub-helmfiles. You already have a basic skeleton of my repo/condition although I’ve made many changes to it so I’m gonna have to articulate my ask.

I’m using a layered approach to my helmfile project as you know. If you look in the generic directory there is a 01a-… and 01b… directory for a tier of tools in each directory so I want to create a selector that would cause helmfile to 1) run everything in the respective 01a-… and 01b-… directories and 2) I want the flexibility to be able to just run one helmfile within a subdirectory of say, 01b-… (essentially just installing a single chart). It seems to me that the documentation says that the selectors are defined in the helmfiles directive of the parent directories to those sub-directories.

given: <project root>/helmfile.d/generic/01a-tier1 and <project root>/helmfile.d/generic/01b-tier2
if I want to be in helmfile.d to run helmfile and I want to install just charts in 01b-tier2 then the helmfile.yaml in helmfile.d would have to define the selectors and point to the helmfile.yaml in generic defining the selectors with a path: pointing to helmfile.yaml in 01b-tier2 and so forth, correct?

Jim Connerover 5 years ago

My helmfile.yaml in helmfile.d looks like:

# helmfile.yaml in helmfile.d
---
helmfiles:
  - "generic/*"
  - path: "generic/common/*"

My helmfile.yaml in generic is:

# helmfile.yaml in generic
---
helmfiles:
  - "*/*.yaml"
  - path: "common/*"

  - path: "01a-network-and-proxies/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=network-and-proxies"

  - path: "01b-secrets-management/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"

Jim Connerover 5 years ago

and finally, my helmfile.yaml in 01b-… for instance is:

# helmfile.yaml in security-management
---
helmfiles:
  - "*/helmfile.yaml"
  - path: "../common/*"

  - path: "certmanager/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=certmanager"

  - path: "vault-operator/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=vault-operator"

  - path: "vault-secrets-webhook/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=vault-secrets-webhook"

  - path: "dex/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=dex"

  - path: "oauth2-proxy/helmfile.yaml"
    values:
      - {{ .Values | toYaml | nindent 8 }}
    selectors:
      - "tier=secrets-managment"
      - "app=oauth2-proxy"

Jim Connerover 5 years ago(edited)

I’m inclined to think that that is incorrect, though, because when I run sync --selectors tier=secrets-managment,app=dex (or delete) for instance, it runs everything:

Jim Connerover 5 years ago

mumoshuover 5 years ago

Why are you giving selectors: for every sub-helmfile? What if you just omitted that and let it install all when helmfile sync and let it install only part of releases with helmfile -l foo=bar sync?

Jim Connerover 5 years ago

I guess because when I tested that, it didn’t work as I’d hoped…

Jim Connerover 5 years ago

maybe I did something wrong.

Jim Connerover 5 years ago

I was just looking at https://github.com/roboll/helmfile/blob/master/examples/README.md and I saw that helmfile is able to use the release labels…

Jim Connerover 5 years ago

so I’ll check that out again and see if I can find out why that doesn’t work.

Jim Connerover 5 years ago

nevertheless, I would like to be able to assign each full directory to a tier

Jim Connerover 5 years ago

sorry for the slow response. I was at the store.

Jim Connerover 5 years ago

very cool. That seems to be working. Not sure what I was doing wrong before…

Jim Connerover 5 years ago

noice!

Jim Connerover 5 years ago

I have to go to the store again to get something for my wife but if you can answer this while I’m gone…
I thought I read in the docs once that there was a way to set a dependency up (like terraforms depends on) but of course, differently implemented. Did I misread that?
For instance, our implementation of dex requires a running vault. I’m sure there are ways to set up the environment that such a thing could be jiggered but is there a native way for helmfile to do this type of functionality?

I’ll be back in about an hour

Jim Connerover 5 years ago

also, quick question about a thought on a feature request which may already exist. It would be super awesome if I could run an argument that would just show me the charts which would be affected by an operation on requested selector(s).

Jim Connerover 5 years ago

bbiab

Jim Connerover 5 years ago

cool. nm. don’t have to go now.

Jim Connerover 5 years ago

I thought I read in the docs once that there was a way to set a dependency up (like terraforms depends on) but of course, differently implemented. Did I misread that?

For instance, our implementation of dex requires a running vault. I’m sure there are ways to set up the environment that such a thing could be jiggered but is there a native way for helmfile to do this type of functionality?

Jim Connerover 5 years ago(edited)

@mumoshu if you’re available: how on earth would one define globally used values for all releases in an environment. This has been my absolute biggest blocker. Even better is that the global scoped values.yaml could be also templatized. I cannot figure out how to achieve this goal. Nothing I’ve tried works.

March 2021 Browse by date May 2021