#kubernetes - April 2019

would it be bad practice to deploy my CI tool in the same k8s cluster as what it is deploying?

Depends on what you want to accomplish with the CI tool

For example if you want the CI tool to upgrade the cluster it operates in, that won’t work

However no reason to limit yourself to one CI service

no

k8s cluster would hold the app and the CI tool that deploys the app

probably ok but maybe in a different namespace?

@Tim Malone that would be the plan yeah

hi all I have a quick question regarding dns zones which I am unsure of, if anyone could help it would be much appreciated.

I want my domain name to be example.com. I have a hosted zone in aws route53 for example.com, which sits in my root account (I can not move it from the root account at this time, because its being used). I have another aws account called production which contains the hosted zone production.example.com, this account is also where my eks cluster is in. In my root account route53 zone example.com i have an NS record production.example.com so that the production account can handle those domains.

If I use external-dns in my eks cluster, and allow it to create records in the production.example.com hosted zone, will my ssl cert hold? The SSL cert I have is a wildcard for *.example.com, and the records that get created from external-dns will look like *.production.example.com

I believe that they wont, but I am not sure. Is there any common way to handle this kind of situtation?

We request SSL certificates In each account separately

The root certificate will not work in different accounts

So prod.example.com will have its own certificate

With wildcard

and even if you created all environments in one account, a cert for *.<http://example.com|example.com> could be used for <http://prod.example.com|prod.example.com>, but will not apply to *.<http://prod.example.com|prod.example.com>

You mean certs? :) They are free on AWS because they can be used only with other AWS resources for which you pay, e.g. load balancers . Not with servers external to AWS

and they are automatically renewed on AWS 🙂 (which will save you a lot of headache because we always forget to renew them 🙂 )

I know kops is somewhat intentionally behind Kubernetes in releases, but it looks like they are only "stable" on 1.11, which technically went EOL when 1.14 went GA if I'm not mistaken. That seems like a really slow release cadence to me. Is it still the go-to for doing home-spun K8S in AWS? EKS isn't keeping up either.

Yea I am not sure why it’s slowed

kube-aws is also worth checking out. @mumoshu is a maintainer.

That's the one you mentioned two weeks ago that I didn't write down. Will take a look. Thanks! 😃

has anyone had any luck installing kiam on eks?

yeah I am using kiam on eks

how did you install it? I tried with with the helmfile in cloudposses repo, but no luck.

Im pretty sure it's because I couldn't run the kiam server on a master node since eks doesnt let you do that

I did a separate node-group for kiam-server that has the required credentials

i used the stable helm chart to install it

what do you mean node-group ?

add --kubelet-extra-args --node-labels=${name_of_node_group} to your userdata

that lets you use a nodeSelector when deploying your pods

ah okay so do you have one node specifically for kiam? or do you run other pods on it as well?

In our case, we run kiam servers on masters

i have a few daemonsets like node exporter, but not any other applications

And agents on all other nodes

yeah you can’t run stuff on eks masters that I am aware of

Oh right

Yea we use kops predominantly

The kiam-server node will have the ability to assume any role, so its best not to run anything else on it

Re kops release schedule having slowed, anything to do with EKS perhaps? i.e. is pickup of kops slowing too?

I wonder... could be

Also the number of options available means there’s not as much support for anyone offering

how do you avoid merging maps in helm?

have a more concrete example? @btai

i think it might be the way the chart is written

@Erik Osterman (Cloud Posse) basically for this chart: https://github.com/helm/charts/blob/master/stable/drone/values.yaml#L153

that DRONE_DATABASE_DATASOURCE can be set as a postgres url i.e. <postgres://username:pw@hostname/dbname>

but I set that as an envSecret value to pull from a k8s secret: https://github.com/helm/charts/blob/master/stable/drone/values.yaml#L159

but i guess when the values get merged for the deployment, the secret env vars get written first then the default env vars: https://github.com/helm/charts/blob/master/stable/drone/templates/deployment-server.yaml#L74

so the DRONE_DATABASE_DATASOURCE value gets overriden by the default sqlite value

possibly just a poorly written helm chart?

or is there a way of ignoring those default values

let me know if that makes any sense..

@btai you can run helm with debug/dryrun enabled - this should show you how the values are being generated which may help you work out what/why a variable value is not as expected

awesome idea! anytime you need a shared secret, generate it ☝️

I use it, very useful : https://github.com/derailed/k9s

@Erik Osterman (Cloud Posse) You were able to convert @stobiewankenobi lolololol

Terraform -> SSM
Then aws-ssm + helm + helmfile at deploy time

SSM is beastmode.. love using serverless.com pulling from there also

hahaha 😃 FTW!

hi all, anyone has experience with eks to assign a pool of static ip address/eni to pods based on the AWS high availability zone?

haven't seen that done before; don't know if it's possible

@Erik Osterman (Cloud Posse) (or anyone) I got a fundamental question for ya..

What's the difference between:
- a canary deployment with 5% increments
- a rolling update with maxUnavailable==0 && maxSurge==5% (with a RR Load Balancer in front)

(no rush.. food for thought)

What I'm thinking about is.. can a native rolling update be used in place of a canary deployment.. Assuming they can monitor the same metrics for health

Because.. spinnaker / istio just for the sake of canary might not be reason enough

🤷‍♂️ midnight thoughts

@Ryan Smith I often use this example when talking to customers that out of the box, kubernetes supports a very basic form of canary+rolling updates. The reason for istio is to have full control over that process. Also, the "gold standard" for canary deployments is to tie it into your monitoring backplane so that you proceed to increase the level of traffic only so long as some KPIs are true. This level of controlled rollouts are more difficult to orchestrate with kubernetes primitives, which is why Istio is used. Also, it doesn't have to be limited to blue/green. It can be a full rainbow of colors, where the traffic is spread across them.

Makes a ton of sense

I think we're all saying the same thing.. or at least on the same train of thought. It's definitely a nice thing to have as business requirements expand (as they always do). Soon enough, it'll become a required thing to have. 😉

what are y’all thoughts on 3 clusters/3az vs 1 cluster/3az? does the 3 cluster approach give us much more reliable availability?

for the 3 cluster approach either using federation or dns round robin

1 cluster, 3 az’s, multi-master setup

if you want more clusters, set those up in different regions instead with dns failover and/or geo/round robin load balancing between them

This sounds insanely cool.

https://rancher.com/blog/2019/announcing-k3os-kubernetes-operating-system/?utm_campaign=Newsletter&utm_source=hs_email&utm_medium=email&utm_content=72089076&_hsenc=p2ANqtz-8QNiAGpBUK__hXfEvzq52iaVINoxhsygCqgfqyOtzf_IvAFn_DzsUyRdyqad5iOpx2T2vXTSOWTTF5Tk5luRdPWGN7Tg&_hsmi=72089030

Anyone run tillerless helm? https://rimusz.net/tillerless-helm/

that's interesting!

does someone have something to say about https://www.ovh.co.uk/kubernetes/ ?

Haven’t used that in particular... I used OVH back in the day with CoreOS. Loved the service for the value. Unlimited bandwidth and beefy bare metal instances.

https://github.com/roboll/helmfile/issues/392#issuecomment-455065039 @Erik Osterman (Cloud Posse) I feel like I need to write some middleware for Helmfile so it can use SSM directly

lol think just right here

Has anyone found a good solution for user management on EKS? The best solution I can find is setting up roles that users can assume, but it doesn’t seem like an optimal solution.

have you looked at Gravitational Teleport?

it supports integration with SSO

and audited session logs with replay

I haven’t, but I will take a look!

Thanks @Erik Osterman (Cloud Posse)

https://github.com/roboll/helmfile/pull/569 SSM integration with Helmfile. Lets see the lash-back. lol