#kubernetes - June 2019 | Slack Archive

kubernetes

157 messages

Archive: https://archive.sweetops.com/kubernetes/

May 2019

June 2019

Jul 2019

Igor Rodionovover 7 years ago

Intresting tool that checks K8s best practices https://github.com/reactiveops/polaris

btaiover 7 years ago

what are you guy’s strategy for memory requests? for example looking at my historical data, my api pods use about 700Mi memory on average. I believe it’s better to set that memory request down to around that number, which will allow for more excess memory in the pool. I have it currently overallocated (1000Mi per api pod) and it adds up how much memory is being reserved but unusable by others that may need it.

Erik Osterman (Cloud Posse)over 7 years ago

Other considerations to take into account is (a) how much memory volatility there is... perhaps 30% variance is a bit high (b) disruptions - how bad is it if the service is evicted to another node?

Erik Osterman (Cloud Posse)over 7 years ago

I would suspect the more pods of a given service you run, the more insulated you are from disruptions of pod evictions

Erik Osterman (Cloud Posse)over 7 years ago

which means you can get by with a a 5-10% limit. make sure you monitor pod restarts.

Erik Osterman (Cloud Posse)over 7 years ago

so long as that number stays at or near 0, you're good.

sarkisover 7 years ago

how are you all connecting kubectl into the k8s cluster these days?

Erik Osterman (Cloud Posse)over 7 years ago

via teleport

Erik Osterman (Cloud Posse)over 7 years ago

teleport supports both ssh and kubectl

Erik Osterman (Cloud Posse)over 7 years ago

SAML authentication

Erik Osterman (Cloud Posse)over 7 years ago

what they call proxy is ~ a bastion, for a centralized entry point

Erik Osterman (Cloud Posse)over 7 years ago

https://gravitational.com/teleport

sarkisover 7 years ago

Interesting ty

btaiover 7 years ago

thanks @Erik Osterman (Cloud Posse) i think i can get away with closer to 5-10%. dont have that much memory volatility looking at my metrics

Alex Coover 7 years ago

Alex Co, [Jun 6, 2019 at 1:28:43 PM]:
i’m having an issue while looping the helm template

env:
{{- range .Values.app.configParams }}
- name: {{ . | title }}
valueFrom:
secretKeyRef:
name: “{{ .Values.app.secretName }}”
key: {{ . | title }}
{{- end }}

this is my code in the template to generate the environment var from the values.yaml

but when i ran the helm lint, it complaints like this

executing “uiza-api-v4/templates/deployment.yaml” at <.Values.app.secretName>: can’t evaluate field Values in type interface {}

i guess that helm template does not allow me to put the secretName value inside a loop

is there anyway to solve this ?

Alex Coover 7 years ago

nvm, it’s because i did not declare .Values.app.secretName as global variable

nutellinoitover 7 years ago

I encountered an issue with eks ebs volume provisioning, with small worker groups (less than 3) the pv was created before the pod and in the wrong AZ.

nutellinoitover 7 years ago

is settting volumeBindingMode: WaitForFirstConsumer enough on v1.12 to fix this problem?

Pablo Costaover 7 years ago(edited)

Yes @nutellinoit It works. But I would also suggest to set an affinity policy for one AZ only, to ensure in case of pod restart or eviction, the pod be scheduled on the same AZ of the PVC

Nikola Velkovskiover 7 years ago

Hi People , do you know of a best/sane way to install k8s on AWS. I see that there are multiple ways to do it. I am eyeing kops because terraform duh but before creating the cluster there's still a lot of preparation to do like:
- creating vpc
- kops state bucket
- route53 record
And than all of it has to be passed on to kops as a cli command.
This is all fine but to me it looks like a bit too much.
Is there any other way of doing it ?

aaratnover 7 years ago

I use EKS

aaratnover 7 years ago

with terraform

Nikola Velkovskiover 7 years ago

ok that's a way

Nikola Velkovskiover 7 years ago

and how do you handle upgrades, i read somewhere that it's a bit tricky with EKS

aaratnover 7 years ago

You mean master version upgrade ?

Nikola Velkovskiover 7 years ago(edited)

like k8s 1.2 -> 1.3 upgrade

aaratnover 7 years ago

Yeah, well its a pretty new cluster

aaratnover 7 years ago

right now my cluster is running on version 1.2

aaratnover 7 years ago

what are the challenges that you have heard of ?

Nikola Velkovskiover 7 years ago

I don't remember the details but I think Erik mentioned something about the upgrade in EKS is not as easy

Nikola Velkovskiover 7 years ago

I might be wrong though

aaratnover 7 years ago

https://aws.amazon.com/blogs/compute/making-cluster-updates-easy-with-amazon-eks/

aaratnover 7 years ago

aws blog says its easy 😕

Nikola Velkovskiover 7 years ago

:)))

Nikola Velkovskiover 7 years ago

fair enough

aaratnover 7 years ago

offcourse we have multiple environments

aaratnover 7 years ago

so we can upgrade the lower environment and check if it works

aaratnover 7 years ago

and proceed with upgrade

Nikola Velkovskiover 7 years ago

niice

Tim Maloneover 7 years ago

they made it much easier recently - you can do it via the AWS console, just change the version

Tim Maloneover 7 years ago

then upgrade your worker nodes afterwards

Tim Maloneover 7 years ago

(but yes you’ll want to do it in non-prod first just in case)

aaratnover 7 years ago(edited)

Terraform has a parameter for version

Nikola Velkovskiover 7 years ago

oh that sounds promising

aaratnover 7 years ago


version – (Optional) Desired Kubernetes master version. If you do not specify a value, the latest available version at resource creation is used and no upgrades will occur except those automatically triggered by EKS. The value must be configured and increased to upgrade the version when desired. Downgrades are not supported by EKS.

aaratnover 7 years ago

https://www.terraform.io/docs/providers/aws/r/eks_cluster.html

Nikola Velkovskiover 7 years ago

and what about installing it, EKS gives you the master nodes only, what about getting the other nodes in EC2, is it just a matter of using cloud-init ?

aaratnover 7 years ago

We do with with auto-scaling-group

aaratnover 7 years ago

You can follow this

aaratnover 7 years ago

https://learn.hashicorp.com/terraform/aws/eks-intro

Nikola Velkovskiover 7 years ago

Thanks a lot people

Nikola Velkovskiover 7 years ago

I will try it out 🙂

nutellinoitover 7 years ago

i tried an upgrade with eks and terraform

nutellinoitover 7 years ago

from 1.11 to 1.12

nutellinoitover 7 years ago

is pretty smooth

nutellinoitover 7 years ago

control plane upgrades without downtime

nutellinoitover 7 years ago

to upgrade the workers the only thing to do is to update amis

nutellinoitover 7 years ago

and replace workers

nutellinoitover 7 years ago

and follow the directions on aws documentation to patch system deployment with new container versions

nutellinoitover 7 years ago

https://docs.aws.amazon.com/en_us/eks/latest/userguide/update-cluster.html

Nikola Velkovskiover 7 years ago

ok, so I guess there's a posibility to automate replacing the instances in the AS somehow

Nikola Velkovskiover 7 years ago

I will look into it

Nikola Velkovskiover 7 years ago

Thanks for your support people!

Nikola Velkovskiover 7 years ago

♥️

nutellinoitover 7 years ago

You can simply terminate one old instance at time and wait for autoscaling group to launch replacements

Nikola Velkovskiover 7 years ago

arhg ye gute ole click-ops 😄

Nikola Velkovskiover 7 years ago

we've developed a lambda with step functions that does the instance replacement, step functions serving as a waiter

Nikola Velkovskiover 7 years ago

so it's fire and forget

Nikola Velkovskiover 7 years ago

takes a while but it's atomic

btaiover 7 years ago(edited)

@Nikola Velkovski I’ve found the k8s upgrades to be a bit slow. it increases in time (by like 5~7 minutes per worker node) so upgrades can take a long time. For me, I wouldn’t be comfortable letting the upgrade for a production cluster run unattended (i.e. overnight while im sleeping) and naturally your production cluster probably has the most worker nodes. What I’ve found works for me pretty well is just using terraform to spin up a new cluster, deploy to the new cluster, and doing the cutover at the DNS level. food for thought

Nikola Velkovskiover 7 years ago

that sounds a lot like aws Elasticsearch @btai terraform apply usually times out when upgrading the ES cluster

Nikola Velkovskiover 7 years ago

thanks!

rjover 7 years ago(edited)

@Nikola Velkovski give a try with rancher. It is the most easiest way to spin up k8s on multiple clouds as per our experience with the tool. https://rancher.com/

Nikola Velkovskiover 7 years ago

@rj I saw that as well but I didn't know if it was any good.

Sandeep Kumarover 7 years ago

Hey Guys Does anyone configured SMTP as a grafana config-map for kubernetes?

Sandeep Kumarover 7 years ago

apiVersion: v1
kind: ConfigMap
metadata:
labels:
app: grafana
name: grafana-smtp-config-map
namespace: monitoring
data:
grafana.ini: |
enabled =true
host=<host>
user=<user>
password=<password>
skip_verify= false
from_address=<email>
from_name=Grafana
welcome_email_on_sign_up=false

Sandeep Kumarover 7 years ago

Ex: something like this

Sandeep Kumarover 7 years ago

and adding this config map in kubernetes grafana deployment
- configMap:
defaultMode: 420
name: grafana-smtp-config-map
name: grafana-smtp-config-map

Sandeep Kumarover 7 years ago

i am trying using above methods to add smtp to grafana.ini

Sandeep Kumarover 7 years ago

but i am unable to add smtp to grafana.ini, is there any documentation/suggestions which can help me here?

timduhenchanterover 7 years ago(edited)

Does anyone have any experience scaling with custom metrics from Datadog across namespaces (or the external metrics API in general)?

Alex Coover 7 years ago

hi, anyone here is using Gloo Gateway on K8s ?

Alex Coover 7 years ago

i’m having a problem that the virtual service stopped accepting traffic after awhile, and status on the ELB to gloo gateway proxy show that it ’s OutOfService

Alex Coover 7 years ago

wonder if anyone here got the same problem

Maarten van der Hoefover 7 years ago

Hi all! Has someone faced this error before?

kernel:[22989972.720097] unregister_netdevice: waiting for eth0 to become free. Usage count = 1

Erik Osterman (Cloud Posse)over 7 years ago(edited)

https://github.com/hjacobs/kubernetes-failure-stories

Erik Osterman (Cloud Posse)over 7 years ago

Public #office-hours starting now! Join us on Zoom if you have any questions. https://zoom.us/j/684901853

btaiover 7 years ago

what ingress controller are you guys using? it seems like alb-ingress-controller isnt quite robust enough for me. things that i feel like its missing:
1. new ingress object = new ALB so there would be a one-to-one mapping of ALBs to services for me (multi-tenant cluster)
2. provisioned resources don’t get cleaned up, at this point i feel like i might want to terraform the load balancer resources i need with the cluster

Maarten van der Hoefover 7 years ago

Im using Ambassador.. a lot of features regarding routing of traffic based on any kind headers, regex matching, Jaeger tracing. Name it :)

btaiover 7 years ago

@Maarten van der Hoef does ambassador spin up cloud resources for u? (load balancers, security groups, etc)

btaiover 7 years ago

i realized that might not be a feature i want as of now in k8s. since terraform is better at managing cloud resource state

davidvasandaniover 7 years ago

Can someone point me to best practices for setting up Traefik/Nginx-Proxy/etc as an ingress for Kubernetes running on 80? Everything is running but ClusterIP is internal and NodePort doesn’t allow ports below 30000. What am I missing?

davidvasandaniover 7 years ago

I was really excited about this post https://medium.com/localz-engineering/kubernetes-traefik-locally-with-a-wildcard-certificate-e15219e5255d

davidvasandaniover 7 years ago

but he’s using a LoadBalancer w/ Docker for Mac Kubernetes which doesn’t make sense.

Hugo Lestaover 7 years ago

Hello @davidvasandani thanks for the article that you've written. Could you please tell me the main capabilities that trafeik have as ingress-controller? Do you have any article with this capabilities?

davidvasandaniover 7 years ago(edited)

👋🏻 Hi @Hugo Lesta. Not my article but Traefik has many capabilities. https://docs.traefik.io/configuration/backends/kubernetes/

Hugo Lestaover 7 years ago(edited)

This previous article you sent me seems worthy for me, I'll try to improve my knowledhe about traefik over k8s. 👍️

davidvasandaniover 7 years ago(edited)

Its good and helped me out but its incomplete. The author mentions using LoadBalancer locally but doesn’t describe how. With a lot of additional work I’ve gotten it working with MetalLB locally. This was a very useful article: https://medium.com/@JockDaRock/kubernetes-metal-lb-for-docker-for-mac-windows-in-10-minutes-23e22f54d1c8

btaiover 7 years ago

anyone using this on their clusters? https://github.com/buzzfeed/sso

Brandon Wilsonover 7 years ago

Looks interesting

Erik Osterman (Cloud Posse)over 7 years ago

Doesn't support websockets, so it was a deal breaker for us

Erik Osterman (Cloud Posse)over 7 years ago

things like the k8s dashboard or grafana require that

Erik Osterman (Cloud Posse)over 7 years ago

bite the bullet. just deploy KeyCloak with Gatekeepers

btaiover 7 years ago

havent heard of keycloak/gatekeeper

Erik Osterman (Cloud Posse)over 7 years ago

I can give you a demo

Erik Osterman (Cloud Posse)over 7 years ago

it's open source, by redhat

Erik Osterman (Cloud Posse)over 7 years ago

we have the helmfiles for it too

btaiover 7 years ago

does it integrate w/google saml?

Erik Osterman (Cloud Posse)over 7 years ago

yup, that's the beauty with keycloak

Erik Osterman (Cloud Posse)over 7 years ago

it basically supports every saml provider

btaiover 7 years ago

yeah it looks like

Erik Osterman (Cloud Posse)over 7 years ago

and we use it with gsuite

Erik Osterman (Cloud Posse)over 7 years ago

not only that, you an use it with https://github.com/mulesoft-labs/aws-keycloak

Erik Osterman (Cloud Posse)over 7 years ago

with aws

Erik Osterman (Cloud Posse)over 7 years ago

it can become the central auth service for everything

btaiover 7 years ago

nice

Erik Osterman (Cloud Posse)over 7 years ago

we use it with kubernetes, teleport, atlantis, grafana, etc

btaiover 7 years ago

yeah super nice, fully integrated

Erik Osterman (Cloud Posse)over 7 years ago

you can even integrate it with multiple auth providers at the same time

btaiover 7 years ago

do helmfiles have a remote “helm chart” that is used as the base?

Erik Osterman (Cloud Posse)over 7 years ago(edited)

not sure, better to check in #helmfile

btaiover 7 years ago

ah yeah it does

btaiover 7 years ago

https://github.com/cloudposse/helmfiles/blob/master/releases/keycloak-gatekeeper.yaml#L4

Erik Osterman (Cloud Posse)over 7 years ago

oh, "base" is a loaded term now

Erik Osterman (Cloud Posse)over 7 years ago

since helmfile has the concept of bases

Erik Osterman (Cloud Posse)over 7 years ago(edited)

this is not a base in that sense

Erik Osterman (Cloud Posse)over 7 years ago

😃

btaiover 7 years ago

like uh

btaiover 7 years ago

base docker image base

btaiover 7 years ago

😄

Erik Osterman (Cloud Posse)over 7 years ago

aha

Erik Osterman (Cloud Posse)over 7 years ago

yes, we use the community image

Erik Osterman (Cloud Posse)over 7 years ago

but guess it would be more secure to run our own

Erik Osterman (Cloud Posse)over 7 years ago

given the role this plays

btaiover 7 years ago

i guess im not being clear enough

btaiover 7 years ago

i was curious if the helm chart is just abstracted out

btaiover 7 years ago

for a helmfile

Erik Osterman (Cloud Posse)over 7 years ago

for the gatekeeper, we're doing something a bit unusual/clever

Erik Osterman (Cloud Posse)over 7 years ago

we are defining an environment

Erik Osterman (Cloud Posse)over 7 years ago

then using that to generate a release for each service in the environment

Erik Osterman (Cloud Posse)over 7 years ago

the alternative is to use sidecars or automatic sidecar injetion

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/stakater/ProxyInjector

btaiover 7 years ago

Erik Osterman (Cloud Posse)over 7 years ago

Here's what our environments file looks like

Erik Osterman (Cloud Posse)over 7 years ago(edited)


services:
  - name: dashboard
    portalName: "Kubernetes Dashboard - Staging"
    host: <http://dashboard.xx-xxxxx-2.staging.xxxxxx.io|dashboard.xx-xxxxx-2.staging.xxxxxx.io>
    useTLS: true
    skipUpstreamTlsVerify: true
    upstream: <https://kubernetes-dashboard.kube-system.svc.cluster.local>
    rules:
      - "uri=/*|roles=kube-admin,dashboard|require-any-role=true"
    debug: false
    replicas: 1
  - name: forecastle
    host: <http://portal.xx-xxxx-2.xxxx.xxxx.io|portal.xx-xxxx-2.xxxx.xxxx.io>
    useTLS: true
    upstream: <http://forecastle.kube-system.svc.cluster.local>
    rules:
      - "uri=/*|roles=kube-admin,user,portal|require-any-role=true"
...

btaiover 7 years ago

i see

Nikola Velkovskiover 7 years ago(edited)

Have you people seen this ?
https://github.com/hjacobs/kubernetes-failure-stories

Nikola Velkovskiover 7 years ago

I got to the spotify video, which is kinda cool, they admit their rookie mistakes around terraform 🙂

Ribhararnus Pracutiarover 7 years ago

Hi guys, how to connect vpn from inside pods? any recommendation out there? So basically, I have to connect client data on premise, only using 1 ip

cabrinhaover 7 years ago

is anyone here using aws-okta with EKS? I'm having trouble granting additional roles access to the cluster.

Brandon Wilsonover 7 years ago

Hey @cabrinha I'd be interested in hearing anything you ran into with aws-okta and EKS. I'm going to be starting down that road this week.

Ayo Bamiover 7 years ago

HI guys, Please could someone help me 🙂. I just created an EKS cluster and its unable to apply some changes to the cluster. I keep getting that error log.. I am using the same user I used to create the cluster. I am also using auth account. So the Users are not exactly in that account they assume role. Am not sure what am missing here being trying this for days now.. Thanks

Erik Osterman (Cloud Posse)over 7 years ago

How did you bring up the cluster? Are you using terraform?

Erik Osterman (Cloud Posse)over 7 years ago

https://github.com/kubernetes/community/tree/master/icons

erikover 7 years ago

Abhishek Shawover 7 years ago

I am stuck while installing Kubeadm in AWS on Amazon linux

Below is the error i get after running sudo install -y kubeadm

http://yum.kubernetes.io/repos/kubernetes-el7-x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for kubernetes

Please help me to get over it.

May 2019 Browse by date July 2019