#kubernetes - February 2019 | Slack Archive

kubernetes

241 messages

Archive: https://archive.sweetops.com/kubernetes/

Jan 2019

February 2019

Mar 2019

Erik Osterman (Cloud Posse)almost 8 years ago

https://github.com/iJanki/kubecron

Erik Osterman (Cloud Posse)almost 8 years ago

@Daren ☝️

nutellinoitalmost 8 years ago

Hi everyone, which is the best way to manage kubernetes deployments using terraform? We are using atlantis to CI/CD infrastructure

nutellinoitalmost 8 years ago

There is the terraform kubernetes provider, but i don’t know if is good for production use

Erik Osterman (Cloud Posse)almost 8 years ago

Personal opinion is that terraform is not a tool well suited for deployments on top of Kubernetes because it is only really good at creating and destroying resources. But updating resources less so.

btaiover 7 years ago

do you guys blue/green your k8s clusters when you want to upgrade or do you utilize rolling updates?

Andriy Knysh (Cloud Posse)over 7 years ago

with kops we usually do rolling updates https://docs.cloudposse.com/geodesic/kops/upgrade-cluster/

Andriy Knysh (Cloud Posse)over 7 years ago

https://docs.cloudposse.com/kubernetes-optimization/scale-cluster-horizontally/

Andriy Knysh (Cloud Posse)over 7 years ago

https://docs.cloudposse.com/kubernetes-optimization/scale-cluster-vertically/

btaiover 7 years ago

slow isnt it?

Andriy Knysh (Cloud Posse)over 7 years ago

yea, takes some time 🙂

btaiover 7 years ago

this is more of a terraform question, but if i had my k8s cluster deployed in its own VPC and I had the database in a seperate VPC. (they are provisioned seperately because I blue/green my k8s clusters when I want to upgrade) If I were to VPC peer, is it possible to not have to upgrade the security group of the database?

btaiover 7 years ago

basically allow full access to the db if there is a vpc peering connection?

Andriy Knysh (Cloud Posse)over 7 years ago

when you upgrade the cluster, is it still the same VPC?

btaiover 7 years ago

nope

btaiover 7 years ago

new k8s cluster, new vpc

Andriy Knysh (Cloud Posse)over 7 years ago

can you make two of them in advance and just add the two SGs to the database’s SG?

btaiover 7 years ago

yes

btaiover 7 years ago

i can do that

btaiover 7 years ago(edited)

that would require an extra step but i think thats the best approach

btaiover 7 years ago(edited)

1. spin up new k8s cluster/VPC
2. update database terraform with new SG
3. cutover
4. spin down old k8s cluster
5. update database terraform remove old SG

btaiover 7 years ago

actually @Andriy Knysh (Cloud Posse), if i provide the db security group to my cluster terraform I could use this

resource "aws_security_group_rule" "allow_all" {
  type            = "ingress"
  from_port       = 0
  to_port         = 65535
  protocol        = "tcp"
  cidr_blocks     = ["0.0.0.0/0"]
  prefix_list_ids = ["pl-12c4e678"]

  security_group_id = "sg-123456"
}

btaiover 7 years ago

🧐 that would automatically do step 2 & 5 for me during cluster spin up and spin down

Andriy Knysh (Cloud Posse)over 7 years ago

hmm… what about ingress rules for the db SG? 🙂 (you need to update them as well)

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

when you create a new VPC and VPC peering, you can update the db SG with new ingress rules (unless you always have just the two VPCs and they never change, in which case you can add the SGs to the db ingress just once)

Andriy Knysh (Cloud Posse)over 7 years ago(edited)

or, if you create the two VPCs with the same CIDRs and they never change, you can add the CIDRs to the db SG (after peering, the db will see those CIDRs)

btaiover 7 years ago

I cant create two vpcs with the same cidr because its in the same account

btaiover 7 years ago

that aws_security_group_rule will update the db SG with the new vpc_id to allow ingress

Andriy Knysh (Cloud Posse)over 7 years ago

by the same I meant they could be different for the two VPCs, but they never change so you know the CIDRs in advance

btaiover 7 years ago

ah yeah

btaiover 7 years ago

that could work, but risk the chances someone spins up a different service using the same unused CIDR

btaiover 7 years ago

(theres only 2 of us at my company that works on this stuff so very unlikely)

Andriy Knysh (Cloud Posse)over 7 years ago

yes

Andriy Knysh (Cloud Posse)over 7 years ago

so it’s better to just update the db SG with the new rule after you spin a new VPC

btaiover 7 years ago

yep

Erik Osterman (Cloud Posse)over 7 years ago

https://engineering.tumblr.com/post/182013497734/open-sourcing-our-kubernetes-tools

btaiover 7 years ago

how are you guys monitoring your kubernetes nodes?

Erik Osterman (Cloud Posse)over 7 years ago

Prometheus & grafana

joshmyersover 7 years ago

https://sysdig.com/blog/enable-kubernetes-pod-security-policy/ Nice

Erik Osterman (Cloud Posse)over 7 years ago

https://containership.github.io/konstellate-editor/

Erik Osterman (Cloud Posse)over 7 years ago

maybe a good learning tool

btaiover 7 years ago

@Erik Osterman (Cloud Posse) are you guys catching nodes that are going to have issues ahead of time?

btaiover 7 years ago

i had a k8s node yesterday that spiked to 100% CPU randomly that had to be cordon & drained

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/kubernetes/node-problem-detector/blob/master/README.md

Erik Osterman (Cloud Posse)over 7 years ago

@btai this look good?

btaiover 7 years ago

interesting

btaiover 7 years ago

i will try it out

btaiover 7 years ago

the daemon.log was showing some interesting stuff

btaiover 7 years ago

on that node that started having issues

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/kubernetes/node-problem-detector/blob/master/config/custom-plugin-monitor.json

Erik Osterman (Cloud Posse)over 7 years ago

If you can generate a check, you can do a custom plugin like this

btaiover 7 years ago

whats a custom plugin?

Erik Osterman (Cloud Posse)over 7 years ago

See example

Erik Osterman (Cloud Posse)over 7 years ago

Basically as simple as writing a a script that exits non zero

btaiover 7 years ago

ah i see

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/danisla/terraform-operator

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/danisla/terraform-operator/blob/master/examples/basic/README.md

Erik Osterman (Cloud Posse)over 7 years ago

http://www.leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml.html

nutellinoitover 7 years ago

Hi everyone, there is a project that manage EKS workers scale in using lifecycle hooks and lambda?

Erik Osterman (Cloud Posse)over 7 years ago

That is what the cluster autoscaler is used for

Erik Osterman (Cloud Posse)over 7 years ago

In other words, using a lambda to scale the cluster node pools could work, but it’s not the prescribed way in Kubernetes

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/kubernetes/autoscaler

nutellinoitover 7 years ago

Thank you Erik

nutellinoitover 7 years ago

but i need only to manage the scale in, when a node is removed by asg

nutellinoitover 7 years ago

i’m writing a new lambda that does kubectl drain on the node via SNS topic

joshmyersover 7 years ago

Doesn’t the autoscaler do scale in too?

nutellinoitover 7 years ago

i’m using plain asg with eks

joshmyersover 7 years ago

plain asg’s as opposed to?

joshmyersover 7 years ago

https://medium.com/@alejandro.millan.frias/cluster-autoscaler-in-amazon-eks-d9f787176519 looks like what I’d expect

dryackover 7 years ago

@dryack has joined the channel

Rohit Vermaover 7 years ago

hi all, wondering how can we retain the NATIP when recreating a cluster using kops.

Rohit Vermaover 7 years ago

there is an open issue https://github.com/kubernetes/kops/issues/3182 but couldn't find a better solution

Rohit Vermaover 7 years ago

all solutions are more about deleting the cluster manually

Erik Osterman (Cloud Posse)over 7 years ago

@Rohit Verma haven't had to do that

Erik Osterman (Cloud Posse)over 7 years ago

though I have had to do other things related to networking in kops and it's always led to that I destroy/recreate =(

Erik Osterman (Cloud Posse)over 7 years ago

----

Erik Osterman (Cloud Posse)over 7 years ago

@ryangolfs

Have you ran <https://github.com/mumoshu/aws-secret-operator>

Because for the life of me I can't get it to create secrets

<https://github.com/mumoshu/aws-secret-operator/issues/1>

Is my issuse as well .. just curious if you ran into this

Erik Osterman (Cloud Posse)over 7 years ago

@mumoshu ☝️

ryangolfsover 7 years ago

@ryangolfs has joined the channel

btaiover 7 years ago

have you guys used envoy?

btaiover 7 years ago

thoughts on it?

Erik Osterman (Cloud Posse)over 7 years ago

we have a basic example......

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/example-app

Erik Osterman (Cloud Posse)over 7 years ago

with istio (envoy sidecar injection)

Erik Osterman (Cloud Posse)over 7 years ago

TL;DR: was impressed how it works and want to do more with it

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/cloudposse/example-app/blob/master/deploy/releases/istio.yaml

btaiover 7 years ago

i dont really need service mesh/service discovery

btaiover 7 years ago

is it worth it just for proxying/traffic mgmt

Erik Osterman (Cloud Posse)over 7 years ago

yea

Erik Osterman (Cloud Posse)over 7 years ago

traffic mgmt / shapping is what i like

Erik Osterman (Cloud Posse)over 7 years ago

circuit breakers, rate limiting, auth, etc

btaiover 7 years ago

whats shapping?

Erik Osterman (Cloud Posse)over 7 years ago

how the traffic flows across deployments (canary releases)

btaiover 7 years ago

ahh

btaiover 7 years ago

sorry im not super familiar with istio, is it recommended to run envoy w/istio?

Dave Yuover 7 years ago

i haven’t used it yet, but i like the promise of standardized request logging also

btaiover 7 years ago

can i just run envoy as my proxy layer?

Erik Osterman (Cloud Posse)over 7 years ago

so istio is a way to manage envoy sidecars

Erik Osterman (Cloud Posse)over 7 years ago

linkerd does the same thing

Erik Osterman (Cloud Posse)over 7 years ago

and there are other ways too

btaiover 7 years ago(edited)

ah so i deploy istio and it deploys envoy sidecars for me in my pods

Erik Osterman (Cloud Posse)over 7 years ago

yup

btaiover 7 years ago

so i currently use traefik as my reverse proxy

btaiover 7 years ago

deployed as daemon set (pod on each node)

btaiover 7 years ago

is envoy considered an optimization?

Erik Osterman (Cloud Posse)over 7 years ago

basically isitio helps you deploy envoy on k8s

Erik Osterman (Cloud Posse)over 7 years ago

i like traefik too, but we haven't used it in the same context

Erik Osterman (Cloud Posse)over 7 years ago

not sure if the feature set overlaps

btaiover 7 years ago

have you guys used istio with EKS?

btaiover 7 years ago

https://istio.io/docs/setup/kubernetes/helm-install/

btaiover 7 years ago

not sure if its outdated, but if you look under prereqs it doesn’t mention EKS

Erik Osterman (Cloud Posse)over 7 years ago

@johncblandii might also have done some research into that

btaiover 7 years ago

https://aws.amazon.com/blogs/opensource/getting-started-istio-eks/

btaiover 7 years ago

sweet

Erik Osterman (Cloud Posse)over 7 years ago

ohhh

Erik Osterman (Cloud Posse)over 7 years ago

i misread EKS (!= ECS)

btaiover 7 years ago

yeah no, eks 🙂

btaiover 7 years ago

after using k8s, no point in using ecs

Erik Osterman (Cloud Posse)over 7 years ago

lol

Erik Osterman (Cloud Posse)over 7 years ago

yes

johncblandiiover 7 years ago

I didn’t actually use Istio. I started to mess with it but hadn’t. We are using EKS and ECS (Fargate), though.

Ramesh Kumarover 7 years ago

Does anyone faced CoreDNS pods are getting stuck at "ContainerCreating" issue?

Erik Osterman (Cloud Posse)over 7 years ago

What do you see when you describe pod?

Ramesh Kumarover 7 years ago

kubelet, ip-10-225-0-236.ec2.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "2c2fa70a9231264ea9e67bd058126b67fee7409691c74165590a75bfecf29d1f" network for pod "coredns-7bcbfc4774-kxqmd": NetworkPlugin cni failed to set up pod "coredns-7bcbfc4774-kxqmd_kube-system" network: add cmd: failed to assign an IP address to container

Ramesh Kumarover 7 years ago

something like that

Ramesh Kumarover 7 years ago

cni plugin version is 1.2.1

Ramesh Kumarover 7 years ago

i have checked, its not related to EC2 instance or networking or IP addresses in subnet

Erik Osterman (Cloud Posse)over 7 years ago

Haven’t had that, but that error looks to be a pretty good hint

Ramesh Kumarover 7 years ago

my subnet has lot of free IP 's and instance has only 3 ENI's used and it can attach up to 10

sarkisover 7 years ago(edited)

what instance sizes are your master/worker nodes @Ramesh Kumar

sarkisover 7 years ago

i was reading abut some issues with t3, m5, c5 or basically the new hypervisor (nitro) instances having this problem

Ramesh Kumarover 7 years ago

i am using r5 instances, @sarkis and checked they are supported

btaiover 7 years ago

@sarkis can you link where you were reading that?

sarkisover 7 years ago

https://github.com/aws/amazon-vpc-cni-k8s/issues/59

sarkisover 7 years ago(edited)

multiple reports of t3, m5, r5 ^ which are all the new nitro instances

erikover 7 years ago

set the channel description: Archive: https://archive.sweetops.com/kubernetes/

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/open-policy-agent/gatekeeper

btaiover 7 years ago

What container registry do u guys use

btaiover 7 years ago

would you guys say if we were to use Istio for traffic management, we could just stay with classic AWS ELBs?

Erik Osterman (Cloud Posse)over 7 years ago

Yes

Erik Osterman (Cloud Posse)over 7 years ago

I'm still not jazzed on ALBs + k8s

Erik Osterman (Cloud Posse)over 7 years ago

current implementation creates one ALB per Ingress

Erik Osterman (Cloud Posse)over 7 years ago

also, enabling NLBs on classic ELBs is trivial

Erik Osterman (Cloud Posse)over 7 years ago


  annotations:
    # by default the type is elb (classic load balancer).
    <http://service.beta.kubernetes.io/aws-load-balancer-type|service.beta.kubernetes.io/aws-load-balancer-type>: nlb

Erik Osterman (Cloud Posse)over 7 years ago

the downside with ELB classic is you lose the client IP

Erik Osterman (Cloud Posse)over 7 years ago

this can be hacked with Proxy Protocol

Erik Osterman (Cloud Posse)over 7 years ago

but nginx-ingress doesn't report the target port with Proxy Protocol correctly, so you don't know if the user is using TLS or not

sarkisover 7 years ago

do ALBs still take forever to create? 😞

Erik Osterman (Cloud Posse)over 7 years ago

Yea they slow the create too

Maarten van der Hoefover 7 years ago

👋 Anyone using Vault instead of Kiam, I'm new to k8s, and wondering what advantages&drawbacks are over using vault like this.

joshmyersover 7 years ago

For AWS authentication? You have to manage Vault for a start 🙂

joshmyersover 7 years ago

Vault could allow more flexibility than Kiam

Maarten van der Hoefover 7 years ago

Figured the kiam server needs to be managed as well, was hoping for it to be more elegant like the ecs-agent in that respect.

joshmyersover 7 years ago

Yeah, you need to manage that too, agents and server

joshmyersover 7 years ago

Has proved interesting in the past but I think mostly OK now

joshmyersover 7 years ago

Vault does a lot more than Kiam though

joshmyersover 7 years ago

How much do you want those other features?

Maarten van der Hoefover 7 years ago(edited)

I think Vault was chosen for the application secrets, so the logical step here would be adding the iam sessions

joshmyersover 7 years ago

kiam is strictly around AWS services

joshmyersover 7 years ago

If already using Vault, I’d stick with it over Kiam for IAM stuff

joshmyersover 7 years ago

if not, kiam maybe a lower hanging fruit

Maarten van der Hoefover 7 years ago

thanks Josh!

joshmyersover 7 years ago

IMO anyway, others will have other views 😄

Maarten van der Hoefover 7 years ago

for sure, no worries 🙂

Andriy Knysh (Cloud Posse)over 7 years ago

vault is not easy to setup https://aws-quickstart.s3.amazonaws.com/quickstart-hashicorp-vault/doc/hashicorp-vault-on-the-aws-cloud.pdf

Maarten van der Hoefover 7 years ago

( Still liking ECS even more, knowing all this 😄 )

joshmyersover 7 years ago

Nope ^^ , but if you are already running it and have gone through that pain…

joshmyersover 7 years ago

If you are AWS, SSM and Kiam may get you what you want easier

Maarten van der Hoefover 7 years ago

but I guess what vault can also do, is probably combining GCP with AWS, for the ones thinking about that ..

joshmyersover 7 years ago

Sure….

joshmyersover 7 years ago

but I don’t know of many folks actually doing that 🧌

joshmyersover 7 years ago

Multi provider is hard.

joshmyersover 7 years ago

Vendor lock in is a thing

joshmyersover 7 years ago

It’s all a tradeoff

joshmyersover 7 years ago

I also don’t really care about being locked into AWS 😄

Maarten van der Hoefover 7 years ago

me neither, they keep adding new stuff, and it works.

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/banzaicloud/bank-vaults

Erik Osterman (Cloud Posse)over 7 years ago

saw that the other day

Erik Osterman (Cloud Posse)over 7 years ago

looks interesting and is related

joshmyersover 7 years ago

Ah nice

joshmyersover 7 years ago

I’ve used https://github.com/UKHomeOffice/vault-sidekick before

joshmyersover 7 years ago

bank-vaults looks fuller featured

joshmyersover 7 years ago

Certainly more complex than Kiam to manage 🙂

Erik Osterman (Cloud Posse)over 7 years ago

haha yea

Erik Osterman (Cloud Posse)over 7 years ago

seriously

Erik Osterman (Cloud Posse)over 7 years ago

what I'd like to see (and there probably exists), is something that implements the AWS IAM metadata proxy pattern of kube2iam, kiam but uses vault as the mediator

Erik Osterman (Cloud Posse)over 7 years ago

then uses the <http://iam.amazonaws.com/role|iam.amazonaws.com/role> annotation just like kube2iam and kiam

Erik Osterman (Cloud Posse)over 7 years ago

that way the interface is interchangable

joshmyersover 7 years ago

Annotations is a super nice way to drive those things in k8s

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/gardener/machine-controller-manager

Erik Osterman (Cloud Posse)over 7 years ago

🤯

Erik Osterman (Cloud Posse)over 7 years ago

https://gardener.cloud/

Erik Osterman (Cloud Posse)over 7 years ago

looks sweet

Erik Osterman (Cloud Posse)over 7 years ago

apparently 100% open source

btaiover 7 years ago(edited)

do you guys have an example using alb-ingress-controller with istio?

Erik Osterman (Cloud Posse)over 7 years ago

not together

mpogrebnyakover 7 years ago

hello,
does anyone know, how can i limit inbound traffic using AWS EKS nodes?

joshmyersover 7 years ago

Limit inbound according to?

joshmyersover 7 years ago

Close your security groups 😂

btaiover 7 years ago

for helm, do you guys do multiple helm installs for dependent helm packages or do you nest them in your helm package for the application being deployed?

btaiover 7 years ago

im trying to use this helm package: https://github.com/helm/charts/tree/master/incubator/aws-alb-ingress-controller

btaiover 7 years ago

and im curious how I should use it because it sets the namespace to be the namespace of the helm release but what if I don’t necessarily want to do that? Should I just modify the helm package files after I fetch them or is it bad practice

Erik Osterman (Cloud Posse)over 7 years ago

are you passing --namespace?

btaiover 7 years ago

i wanted to avoid passing --namespace

nutellinoitover 7 years ago(edited)

@btai You can set a value for namespace in values.yaml eg “custom_namespace” and then you reference it the templates {{ .Values.custom_namespace }}

frednotetover 7 years ago

Hi everyone ! Does somebody know the simplest way to enable hpa's on a fresh new kops cluster ? metrics-server cannot connect (401 forbidden) and I can't find the solution to retrieve metrics… maybe another solution ?

James D. Bohrmanover 7 years ago(edited)

Has anyone seen this yet? I haven't played with it, but it looks really cool.


Write a Tiltfile script that describes how your services fit together. Share it with your team so that any engineer can hack on any server. See a complete view of your system, from building to deploying to logging to crashing.

https://tilt.dev/

James D. Bohrmanover 7 years ago

Anyone using Jaeger with K8's here?

Visheyraover 7 years ago(edited)

@James D. Bohrman i'm using jaeger with k8s

Erik Osterman (Cloud Posse)over 7 years ago

Has anyone looked into using AWS App Mesh (managed Envoy control plane ~ istio) with non-EKS kubernetes clusters? (e.g. #kops)

Erik Osterman (Cloud Posse)over 7 years ago

Is designed to pluggable and will support bringing your own Envoy images and Istio Mixer in the future.

Erik Osterman (Cloud Posse)over 7 years ago

Today, AWS App Mesh is available to use in preview

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/solo-io/supergloo

Erik Osterman (Cloud Posse)over 7 years ago

@mumoshu have you seen this? ☝️

Erik Osterman (Cloud Posse)over 7 years ago

https://medium.com/solo-io/https-medium-com-solo-io-supergloo-ff2aae1fb96f

btaiover 7 years ago

what do you guys use for SSL certs?

Visheyraover 7 years ago

@btai which cert ? the one facing our apps ? or the one needed by kube to works ? (like api server, kubelet, ...)

btaiover 7 years ago

facing your apps

endofcakeover 7 years ago

Anyone using Loki? https://github.com/grafana/loki

Visheyraover 7 years ago

@endofcake I know that @zadkiel gave a try on this

Visheyraover 7 years ago

@btai https://github.com/jetstack/cert-manager

Visheyraover 7 years ago

this is what you need 😉

btaiover 7 years ago

nice im looking into that right now

btaiover 7 years ago

whats the best way to generate some certs manually in the meantime?

Visheyraover 7 years ago

openssl man

btaiover 7 years ago

can i generate some with letsencrypt ?

Visheyraover 7 years ago

or this https://github.com/cloudflare/cfssl

Visheyraover 7 years ago

yes you can

Visheyraover 7 years ago(edited)

but man, certmanager is a maximum 1 hour setup for basic certificate generation

btaiover 7 years ago

yeah?

Visheyraover 7 years ago

yes !

Visheyraover 7 years ago

there is an helm chart for that also in the github I linked to you

Visheyraover 7 years ago

let me take a look I have some documentation for this in local

erikover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

anyone going?

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/linki/cloudformation-operator

endofcakeover 7 years ago

https://banzaicloud.com/blog/istio-operator/

Erik Osterman (Cloud Posse)over 7 years ago

Anyone using AWS Service Mesh?

Erik Osterman (Cloud Posse)over 7 years ago

I love Istio, but it's k8s centric; we have a upcoming use-case to create a mesh across ECS and k8s

btaiover 7 years ago

one day istio will be independent of k8s 😄

January 2019 Browse by date March 2019