#kubernetes - November 2018 | Slack Archive

250 messages

Archive: https://archive.sweetops.com/kubernetes/

Oct 2018

November 2018

Dec 2018

Rohit Vermaalmost 8 years ago

hi all, what are you opinion on different networking option in kubernetes on aws. Which is more preferred and felt robust. We did tried with aws-vpc-cni but felt that its not stable enough even with 1.1.0 for kuberntes 1.10.6. This becomes more unstable when all your worker nodes are unstable and started giving exception as sandox ip changed etc..

Rohit Vermaalmost 8 years ago

we then switched to calico, but somehow its observed that its impacting the way pods terminate. If we delete a deployment, pods remain in terminating state for 5+ minutes.

Andriy Knysh (Cloud Posse)almost 8 years ago

@Rohit Verma we saw something like that with some of k8s pods, in particular kiam- when deleted, the pods take many minutes to terminate

Andriy Knysh (Cloud Posse)almost 8 years ago

so maybe it’s an issue with some deployments, not the network itself?

Rohit Vermaalmost 8 years ago

But the pods I am referring here is generic like nginx or spring boot app

Rohit Vermaalmost 8 years ago

Anyways more concerned about a general opinion on different kubernetes networks

Erik Osterman (Cloud Posse)almost 8 years ago

We haven’t had the opportunity to explore/optimize the network layer in k8s

Erik Osterman (Cloud Posse)almost 8 years ago

Also are you familiar with the dumb-init “fix” ?

Erik Osterman (Cloud Posse)almost 8 years ago

This is to address the same symptoms

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/Yelp/dumb-init/blob/master/README.md

onzyonealmost 8 years ago

@Andriy Knysh (Cloud Posse) hello again, do you have any doco's and best practices for promoting kube, within nonp? ... ie ... dev to staging?

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

Nonp?

onzyonealmost 8 years ago

ya we run a two accounts ... were one is prod and one is none-prod

onzyonealmost 8 years ago

and all our none-prod stuff happens in nonp

Erik Osterman (Cloud Posse)almost 8 years ago

Aha gotcha

Erik Osterman (Cloud Posse)almost 8 years ago

We don’t have the promotion process documented but I can share how it looks (we use Codefresh)

Erik Osterman (Cloud Posse)almost 8 years ago

I am currently on my phone so will share a little later

onzyonealmost 8 years ago

np sounds good

Teealmost 8 years ago

using kops or terraform for creating kubernetes Production. What is better and cons ??

Andriy Knysh (Cloud Posse)almost 8 years ago

@Tee we use terraform to create kops resources, e.g.

Andriy Knysh (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/terraform-root-modules/blob/master/aws/kops/main.tf

Andriy Knysh (Cloud Posse)almost 8 years ago

https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform

Andriy Knysh (Cloud Posse)almost 8 years ago

and then use kops to provision k8s clusters

Erik Osterman (Cloud Posse)almost 8 years ago

there was some discussion earlier in #terraform I think related to EKS

Andriy Knysh (Cloud Posse)almost 8 years ago

we also have TF modules for EKS

Erik Osterman (Cloud Posse)almost 8 years ago

@Tee are you thinking GCP or AWS?

Teealmost 8 years ago

AWS

Erik Osterman (Cloud Posse)almost 8 years ago

so on AWS, my opinion is that it's more work than than necessary to manage EKS with terraform. the challenge comes down to upgrading. there's some discussions on strategies for that.

Andriy Knysh (Cloud Posse)almost 8 years ago

https://sweetops.slack.com/archives/CB6GHNLG0/p1541085179253600

Erik Osterman (Cloud Posse)almost 8 years ago

with kops, the ability to do rolling-updates is built in; it's a purpose built tool like kops will do a better job at managing lifecycles.

Erik Osterman (Cloud Posse)almost 8 years ago

if fargate announces EKS support at the end of the month, I might change my stance 😃

Teealmost 8 years ago

But the EKS and FARGATE gets pretty expensive

Teealmost 8 years ago

as far i think

Erik Osterman (Cloud Posse)almost 8 years ago

humans aren't cheap either 😃

Teealmost 8 years ago

Right 😆

Teealmost 8 years ago(edited)

So what do you suggest for longterm. Not considering the cost. With less bottlenecks and nightmares

Andriy Knysh (Cloud Posse)almost 8 years ago

not an easy question 🙂

Teealmost 8 years ago

I mean in terms of stability

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

kops is well established and works well, and does lifecycle management

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

EKS is new and lacks a lot of features, but it will stay and they will improve it

Andriy Knysh (Cloud Posse)almost 8 years ago

Fargate will improve and cost will be reduced

Erik Osterman (Cloud Posse)almost 8 years ago

(we're not using EKS in production yet, so our story will be biased towards kops)

Teealmost 8 years ago

Oh ok. Thanks @Erik Osterman (Cloud Posse) & @Andriy Knysh (Cloud Posse) for your suggestions.

Andriy Knysh (Cloud Posse)almost 8 years ago

yea, the point is that with the current state of EKS, you need to do and provision even more resources than using kops

Andriy Knysh (Cloud Posse)almost 8 years ago

and it does not support many features

Andriy Knysh (Cloud Posse)almost 8 years ago

Fargate could improve it, but as many mentioned it’s costly (and it does not exists yet)

Teealmost 8 years ago

That makes sense

Matthewalmost 8 years ago

I am currently moving all of our infrastructure off Mesosphere DC/OS onto EKS and EKS has been phenomenal in my opinion - just lots of support from many different aspects such as AWS and the Kubernetes community

Matthewalmost 8 years ago

as well as great folks like Cloud Posse 🙂

Andriy Knysh (Cloud Posse)almost 8 years ago

yea thanks @Matthew 🙂

Andriy Knysh (Cloud Posse)almost 8 years ago

the point is that with EKS, if for example you need to perform a rolling update, it’s not supported out of the gate

Andriy Knysh (Cloud Posse)almost 8 years ago

so a lot of friction with many things

Andriy Knysh (Cloud Posse)almost 8 years ago

with kops it just works

Andriy Knysh (Cloud Posse)almost 8 years ago

but sure for longterm EKS/Fargate would be better

Matthewalmost 8 years ago

Yeah i've talked with EKS specialist from AWS and they currently suggest a blue/green strategy for upgrading which can be tedious and at times break backwards compatibility

btaialmost 8 years ago

how do you export a single context of your kubeconfig?

btaialmost 8 years ago

say my local kubeconfig has a dev qa prod context

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

@btai we don’t have multiple contexts. We use containers + ENV vars pattern (implemented in geodesic + repo per env + Dockerfile(s)). So in each container (prod, staging, dev, etc), when we run it, we have all ENV vars defined for that particular env (ENV vars come from Dockerfiles or from SSM if they are secrets). That includes everything for Terraform, kops, k8s, etc.

Andriy Knysh (Cloud Posse)almost 8 years ago

So when we do for example kops export kubecfg, the environment knows what context we want

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

and we can run those geodesic containers locally and also in CI/CD pipelines (for which we use Codefresh since it can run each pipeline step as a Docker container)

btaialmost 8 years ago

nice thanks

Ryan Smithalmost 8 years ago

Have you guys used Codefresh enterprise? I know you're all big into codefresh here. 🙂 Just curious of any pitfalls or bits of advice you guys have

Ryan Smithalmost 8 years ago

(enterprise to run on-prem)

Erik Osterman (Cloud Posse)almost 8 years ago

So Codefresh enterprise has 3 variations: full SaaS, hybrid and on-prem

Erik Osterman (Cloud Posse)almost 8 years ago

we’ve been working exclusively with the enterprise SaaS

Ryan Smithalmost 8 years ago

Ooo

Erik Osterman (Cloud Posse)almost 8 years ago

what’s the primary driver for going on-prem?

Ryan Smithalmost 8 years ago

compliance requiring no dependence on external SaaS providers

Ryan Smithalmost 8 years ago

😕

Erik Osterman (Cloud Posse)almost 8 years ago

which compliance certification?

Ryan Smithalmost 8 years ago

oh wow, you haven't even taken me out on a date yet to be asking such risquè questions. 😊

Ryan Smithalmost 8 years ago

lol jk, I think fedramp

Erik Osterman (Cloud Posse)almost 8 years ago

lol

Erik Osterman (Cloud Posse)almost 8 years ago

ok - that’s a whole ’nother cup of tea

Erik Osterman (Cloud Posse)almost 8 years ago

not familiar with

Erik Osterman (Cloud Posse)almost 8 years ago

but sounds like you’d need full on-prem.

Erik Osterman (Cloud Posse)almost 8 years ago

so i probably wouldn’t enlighten you more than you already know

Ryan Smithalmost 8 years ago

No worries. We're new to codefresh--so just probing for any gotcha's really

Erik Osterman (Cloud Posse)almost 8 years ago

@dustinvb can definitely elaborate

Erik Osterman (Cloud Posse)almost 8 years ago

are you using the helm based install?

Ryan Smithalmost 8 years ago

I think at the moment, yes

Ryan Smithalmost 8 years ago

debating about the release of terraform 0.12 and using all the templating stuff

Ryan Smithalmost 8 years ago

rather than 2 templating engines.. tiller.. and all that jazz

Erik Osterman (Cloud Posse)almost 8 years ago

so i get where you’re coming from - but from what i’ve gleaned the current helm provider is too basic to handle all kinds of helm charts. maybe with 0.12 it’s better off

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

you’ve seen our helmfiles repo? basically you can’t do half of what we do with helmfile using that provider

Ryan Smithalmost 8 years ago

https://marketplace.fedramp.gov/#/product/aiware-government?sort=productName

Just as a cover my ass that I'm not saying providing any confidential information.. it's publicly available that we're on fedramp ^^^ lol

Ryan Smithalmost 8 years ago

Oh, my bad. I didn't mean the helm provider.. I meant generating the k8s.yml files on the fly based on the infra-state.. no helm installation anywhere

Ryan Smithalmost 8 years ago

just a thought at the moment--not necessarily going that direction for sure

Erik Osterman (Cloud Posse)almost 8 years ago

but yea, you could basically create terraform modules in place of helm charts

Erik Osterman (Cloud Posse)almost 8 years ago

… if terraform templating is sufficient

Ryan Smithalmost 8 years ago

hehe, yeah, big "if"

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

it’s been my experience, the “simple” case always works well regardless of the technology

Ryan Smithalmost 8 years ago

how have you guys been liking helm? any complaints with the tiller stuff, or you guys are experienced enough with it all--nothing really bugs you?

Erik Osterman (Cloud Posse)almost 8 years ago

i mean, it sucks about the tiller and all

Erik Osterman (Cloud Posse)almost 8 years ago

but i look at helm more like an interface

Erik Osterman (Cloud Posse)almost 8 years ago

and the interface won’t change dramatically, but the underlying implementation is getting a big overhaul as you’re probably aware

Erik Osterman (Cloud Posse)almost 8 years ago

as part of that tiller is going away and the template engine going pluggable

Ryan Smithalmost 8 years ago

"tillerless helm" is the buzz

Ryan Smithalmost 8 years ago

yea

Erik Osterman (Cloud Posse)almost 8 years ago

as a way to manage a complex apps it’s great

Erik Osterman (Cloud Posse)almost 8 years ago

and app dependencies

Erik Osterman (Cloud Posse)almost 8 years ago

i say (and with some humility) that those before us have invested a lot of time in what it takes to manage software releases

Erik Osterman (Cloud Posse)almost 8 years ago

deb, rpm, apk, etc.

Erik Osterman (Cloud Posse)almost 8 years ago

we tried to avoid that with just a Makefile; it worked well until it didn’t. in the end, we needed all that a package manager provides and conceded to package .apk alpine packages

Erik Osterman (Cloud Posse)almost 8 years ago

my point is that just templatizing raw kubernetes resources and applying them seems easy enough and i’m sure you can get away with it for a long time

Erik Osterman (Cloud Posse)almost 8 years ago

but then you realize you want to have dependencies, triggers on deployment or uninstall, and rollbacks, etc. then you’re on your own.

Erik Osterman (Cloud Posse)almost 8 years ago

the more homegrown/spun, the more the solution diverges from the trajectory the community is taking

Erik Osterman (Cloud Posse)almost 8 years ago

because the community is solving problems around a standardized toolset

Ryan Smithalmost 8 years ago

all true

Ryan Smithalmost 8 years ago

so i'm curious.. you bring up rollbacks

Ryan Smithalmost 8 years ago

codefresh/spinnaker's solutions didn't offer enough in that aspect?

Erik Osterman (Cloud Posse)almost 8 years ago

codefresh relies on the fact that helm does rollbacks automatically

Ryan Smithalmost 8 years ago

Erik Osterman (Cloud Posse)almost 8 years ago

and even bakes that into the UI with one-click rollbacks

Erik Osterman (Cloud Posse)almost 8 years ago

they also have some even more cool stuff in the works - but you’ll have to ask them to see it 😃

Ryan Smithalmost 8 years ago

For sure

Ryan Smithalmost 8 years ago

we have meetings setup with them

Ryan Smithalmost 8 years ago

We'll probe 😉

Erik Osterman (Cloud Posse)almost 8 years ago

very cool! hit me afterwards and let me know how it goes

Ryan Smithalmost 8 years ago

does all this reveal a well needed niche (product offering) in the CI/CD process for k8s?

Ryan Smithalmost 8 years ago

since there always ends up being handrolled stuff?

Erik Osterman (Cloud Posse)almost 8 years ago

haha, not sure - there are more CI/CD platforms today than ever

Ryan Smithalmost 8 years ago

https://github.com/gaia-pipeline/gaia I like their philosophy at that in particular

Ryan Smithalmost 8 years ago

yea

Erik Osterman (Cloud Posse)almost 8 years ago

i can't keep them straight anymore

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

spinnaker is now coming out with an enterprise offering too

Ryan Smithalmost 8 years ago

haha nice. Well, after the bloodbath, hopefully the best solution reigns supreme

Ryan Smithalmost 8 years ago

halyard was surprising when I first played with it

Erik Osterman (Cloud Posse)almost 8 years ago

and then github actions

Ryan Smithalmost 8 years ago

then I looked at the helm chart for spinnaker.. and it was just a bunch of hal commands

Ryan Smithalmost 8 years ago

yea

Erik Osterman (Cloud Posse)almost 8 years ago

but i agree that there's still big room for improvement

Erik Osterman (Cloud Posse)almost 8 years ago

the fact there is so much handrolling and independent tooling

Erik Osterman (Cloud Posse)almost 8 years ago

I think codefresh is well poised to do that as it relates to cicd+kubernetes+helm

Ryan Smithalmost 8 years ago

does your gut think helm isn't going anywhere?

Erik Osterman (Cloud Posse)almost 8 years ago

until I see an alternative that has anywhere near the critical mass of helm, yes - i think it’s here for the foreseeable future

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

for example, there’s ksonnet (based on jsonnet) which looks interesting

Erik Osterman (Cloud Posse)almost 8 years ago

but i think some variation of that could be used as a pluggable engine for helm

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

also, i don’t want to see proliferation of more packaging systems right now - it’s too early

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/rimusz/helm-tiller

Erik Osterman (Cloud Posse)almost 8 years ago

have you seen this plugin?

Erik Osterman (Cloud Posse)almost 8 years ago

this is pretty smart.

Ryan Smithalmost 8 years ago

I thiiiink I've seen this one.. if not it was something similar

Erik Osterman (Cloud Posse)almost 8 years ago

basically, it’s a drop in replacement. it still stores all configs in the cluster (per namespace if you want)

Erik Osterman (Cloud Posse)almost 8 years ago

you run a temporary tiller locally

Erik Osterman (Cloud Posse)almost 8 years ago

this can be run as part of CI

Ryan Smithalmost 8 years ago

interesting.. hmm.. nice actually!

Erik Osterman (Cloud Posse)almost 8 years ago

(though would break the codefresh helm UI, since it would need to talk to the tiller and there would be none running)

Ryan Smithalmost 8 years ago

ah, right

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/technosophos/helm-ksonnet/blob/master/README.md

Erik Osterman (Cloud Posse)almost 8 years ago

Dig it

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/helm/helm/issues/2577#issuecomment-339238305

Erik Osterman (Cloud Posse)almost 8 years ago

Guess my hopes of seeing ksonnet as a template engine in helm were misguided

Erik Osterman (Cloud Posse)almost 8 years ago

I know Lua is coming. I’d heard such great things about jsonnet, that I assumed it would be well suited. But Lua I guess is a better understood embeddable language

Erik Osterman (Cloud Posse)almost 8 years ago

Last I had to write Lua was 14 years ago when dealing with Nginx

Ryan Smithalmost 8 years ago

For the codefresh peeps out there.. does it matter what/how the ingress controller looks when using codefresh for deployments?

Ryan Smithalmost 8 years ago(edited)

Was reading through: https://docs.traefik.io/user-guide/kubernetes/#traffic-splitting and it came to mind

Erik Osterman (Cloud Posse)almost 8 years ago

Nope, we use for example the CloudFlare Acesss/Argo ingress and nginx-ingress controller in the same cluster

Ryan Rykealmost 8 years ago

have you seen this https://www.youtube.com/watch?v=kOa_llowQ1c

Ryan Rykealmost 8 years ago

i hate kelsey hightower in the best way possible

Andriy Knysh (Cloud Posse)almost 8 years ago

this is interesting https://aws.amazon.com/blogs/opensource/continuous-delivery-eks-jenkins-x/

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

i did not think it could do so much, it creates pipelines for infrastructure itself (prod and staging), and pipelines for the app, and even spawns a separate testing/staging env in k8s for each PR, and comments on GitHub on PRs (like atlantis), and creates GitHub repos with Helm charts for the infrastructure (prod and staging)

Andriy Knysh (Cloud Posse)almost 8 years ago

https://github.com/jenkins-x/sso-operator (@Erik Osterman (Cloud Posse) already posted it before)

Andriy Knysh (Cloud Posse)almost 8 years ago

one thing it can’t do is to upgrade the k8s cluster b/c it itself sits in the same cluster 🙂

Erik Osterman (Cloud Posse)almost 8 years ago

wow

Ramesh Kumaralmost 8 years ago

@here Any recommendations for learning distributed systems from basics to advance?

Ramesh Kumaralmost 8 years ago

noticed, lot of people knows the tools but not the concepts…

Andriy Knysh (Cloud Posse)almost 8 years ago(edited)

@Ramesh Kumar I found this site very interesting and with lots of resources about distributed systems, and real-life examples from many companies

Andriy Knysh (Cloud Posse)almost 8 years ago

http://highscalability.com/

Andriy Knysh (Cloud Posse)almost 8 years ago

http://highscalability.com/all-time-favorites/

Andriy Knysh (Cloud Posse)almost 8 years ago

http://highscalability.com/blog/category/example

Andriy Knysh (Cloud Posse)almost 8 years ago

http://highscalability.squarespace.com/blog/category/strategy

Ramesh Kumaralmost 8 years ago

@Andriy Knysh (Cloud Posse) thanks..

Erik Osterman (Cloud Posse)almost 8 years ago

https://news.ycombinator.com/item?id=18428497

Erik Osterman (Cloud Posse)almost 8 years ago

“Google Kubernetes Engine's third consecutive day of service disruption”

btaialmost 8 years ago

anyone use the official python kube library?

btaialmost 8 years ago

can you load the config from a dict?

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

~~why not use config profiles instead?~~

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

~~e.g. AWS_DEFAULT_PROFILE=cp-prod-admin~~

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

~~the underlying aws SDK should then handle everything automatically~~

btaialmost 8 years ago(edited)

the kube config?

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

heh, my bad @btai

btaialmost 8 years ago

😛

btaialmost 8 years ago

how would i run kubectl within a container running from a job?

Erik Osterman (Cloud Posse)almost 8 years ago

here’s an example doing it from a deployment: https://github.com/onfido/k8s-rabbit-pod-autoscaler

Erik Osterman (Cloud Posse)almost 8 years ago

doing it from a job wouldn’t be any different

Erik Osterman (Cloud Posse)almost 8 years ago

just need the proper role bindings

Erik Osterman (Cloud Posse)almost 8 years ago

in this case, kubectl is gettin called from in the autoscale.sh

btaialmost 8 years ago(edited)

so if i have the wrong role bindings

btaialmost 8 years ago

would i be getting this error:

btaialmost 8 years ago

The connection to the server localhost:8080 was refused - did you specify the right host or port?

Erik Osterman (Cloud Posse)almost 8 years ago

all i know is when we implemented it for redis using the strategy above (for rabbit), we didn’t need to specify any hosts

Erik Osterman (Cloud Posse)almost 8 years ago

it just autodiscovers it

btaialmost 8 years ago

the pod autodiscovers

btaialmost 8 years ago

ok thats what i was hoping for

Erik Osterman (Cloud Posse)almost 8 years ago

it also provides a kube context

btaialmost 8 years ago

so the pod itself didnt have any kubeconfig or kube api secrets

Erik Osterman (Cloud Posse)almost 8 years ago

yea, it didn’t have anythign like that

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/vanvalenlab/kiosk-autoscaler

btaialmost 8 years ago

yeah i have a job basically doing the same thing

btaialmost 8 years ago

executing a shell script that makes a kubectl call

btaialmost 8 years ago

but i get the above error

Erik Osterman (Cloud Posse)almost 8 years ago

kops cluster?

btaialmost 8 years ago

aks

Erik Osterman (Cloud Posse)almost 8 years ago

we tested it on gke and kops

btaialmost 8 years ago

yeah it works in kops

btaialmost 8 years ago

that job

Erik Osterman (Cloud Posse)almost 8 years ago

oh interesting!

Erik Osterman (Cloud Posse)almost 8 years ago

you have rbac enabled in kops?

btaialmost 8 years ago

although the kops

btaialmost 8 years ago

doesnt have rbac enabled

Erik Osterman (Cloud Posse)almost 8 years ago

😃

Erik Osterman (Cloud Posse)almost 8 years ago

haha

btaialmost 8 years ago

yeah

btaialmost 8 years ago

so do i create a clusterrolebinding for the job?

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/onfido/k8s-rabbit-pod-autoscaler/blob/master/deploy.yml#L1-L29

btaialmost 8 years ago

thanks

btaialmost 8 years ago

sorry, still new to k8s

btaialmost 8 years ago

hypothetically if i create a cluster role binding with the namespace and name that matches the job, that should work?

Erik Osterman (Cloud Posse)almost 8 years ago

More or less

Erik Osterman (Cloud Posse)almost 8 years ago

I don’t know the specific matching selectors that are available

Erik Osterman (Cloud Posse)almost 8 years ago

This is near. Copy secrets from a centralized system of record. https://github.com/mittwald/kubernetes-replicator/

Erik Osterman (Cloud Posse)almost 8 years ago

How are you guys handling busy helm deployments where the tiller is busy attending to other deployments...

                                                                                                                             
Error: could not find a ready tiller pod

Erik Osterman (Cloud Posse)almost 8 years ago

@Max Moon @dustinvb

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/helm/helm/pull/3464

dustinvbalmost 8 years ago

I haven't ran into this scaling issue yet.

Erik Osterman (Cloud Posse)almost 8 years ago

—replicas option looks nice

Erik Osterman (Cloud Posse)almost 8 years ago

@Michal Matyjek @Daren have you run into this?

Max Moonalmost 8 years ago

I have not run into this yet either

Darenalmost 8 years ago

I have not

Michal Matyjekalmost 8 years ago

not yet

Michal Matyjekalmost 8 years ago

how many deployments are we talking about?

Erik Osterman (Cloud Posse)almost 8 years ago

just concurrency

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

so we're running helm on every PR synchronization for unlimited staging environments

Erik Osterman (Cloud Posse)almost 8 years ago

so we're getting it

Erik Osterman (Cloud Posse)almost 8 years ago

e.g. 2 developers push at around the same time

sarkisalmost 8 years ago

hey all - curious what the verdict is on kiam vs kube2iam... it seems like kiam was created to address some issues with kube2iam - is kiam the way to go these days?

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

@sarkis yea, kube2iam is dead and should not be used. It's a massive liability to even deploy in an AWS account. If you run more than N hosts (N ~10), you'll DoS AWS APIs and they rate limit you.

Erik Osterman (Cloud Posse)almost 8 years ago

kiam addresses this by having a client/server model. clients run on all nodes (agents), and talk to the server. the server is responsible for fetching the credentials which reduces rate of requests

Erik Osterman (Cloud Posse)almost 8 years ago(edited)

it also caches

Erik Osterman (Cloud Posse)almost 8 years ago

I think there's been some frustration related to the rate of development on Kiam, but the worse bugs are fixed.

Erik Osterman (Cloud Posse)almost 8 years ago

Also, I don't know of any alternatives to kiam and kube2iam for AWS

sarkisalmost 8 years ago

thanks @Erik Osterman (Cloud Posse)!

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/futuresimple/helm-secrets

Erik Osterman (Cloud Posse)almost 8 years ago

https://kubernetes.io/docs/tasks/inject-data-application/podpreset/