#kubernetes - June 2020

any suggestion for the “mailtrap” self-hosted on k8s app? (mailhog?)

Hi, I am planning to install namespace isolation for my k8s cluster. I normally use ALB Ingress and External-Dns but each of them can only be scoped per namespace. What do you do or have done to make multi namespaced environment? I was planning to install a seperate Ingress ALB (though controller can span multiple namespace an ingress alb can only scope on namespace) and seperate External-DNS (https://github.com/kubernetes-sigs/external-dns/issues/1454 )

and for ALB Ingress : https://github.com/kubernetes-sigs/aws-alb-ingress-controller/issues/329

Since you don't need the statefulness can you just remove the storage protection finalizers?

digging into k8s RBAC a little more, want to create a "read only" role for users - anyone know of any pre-defined helm charts that have different permission groups in them?

Or, since it's not an "app" would a helm chart be the wrong way to deploy it? Just thinking to reuse existing deployment patterns

maybe look at the rbacmanager chart, it will simplify role creation and binding a bit (creates some CRDs for you to use). But if you were just going to use straight k8s manifests you could use the raw chart in your helmfile to define your permissions as well.

Thanks, rbacmanager looks useful

rbacmanager is great, using it along with https://github.com/kubernetes-sigs/aws-iam-authenticator makes rbac so much easier to manage

Anyone care to share their lessons-learned from running k8s on bare-metal?
I'm looking for best practices and potential war stories.

I'm writing a k8s controller which monitors a CRD. I want the CRD definition to wrap the built-in Deployment spec schema without me having to redefine it. Is it possible? I want to be able to do something like this:

apiVersion: <http://example.com/v1alpha1|example.com/v1alpha1>
kind: MyResource
metadata:
  name: myresource
spec:
  deployment:
    replicas: 3
    template:
      metadata:
      ...
      (Just like a normal deployment definition)

i'm curious.. what do you guys use for continuous delivery/continuous integration/continuous release? i work at a smallish (~100 people) company and we're generally jamming pretty hard on delivering product features instead of focusing on infrastructure. we use kubernetes and some folks use a service mesh, but not all, and we don't always use the same clusters for various reasons.

some of the options i've considered (from cncf landscape) are flagger (requires service mesh) and spinnaker (gold standard? heavy weight). i suspect armory is above what i'm able to pay for a saas-type solution, and with limited time to focus on this specifically, flagger (service mesh) and spinnaker (learning curve/setup/maintenance?) i suspect is just overkill.

the other option i think i'm most likely to implement because it's fast and easy is just using a separate deployment in the same cluster and weighted dns to serve some small percentage of traffic to a canary deployment and continuously deploy there first.

any thoughts/feedback appreciated.

anyone have tradeoffs of using EKS managed node groups vs self managed?

Hey 👋 - Is anyone using istio with Ingress resources? My problem is that many helm charts come with Ingress resource definitions and we're using istio-ingressgateway as, well, our ingress gateway. Just can't seem to get it to work as it should at the moment. Any gotchas that others might have encountered would be useful 🙂

Anybody using Lens and sourcing creds via aws-vault? Found https://github.com/99designs/aws-vault/issues/561 but that isn’t doing the trick for me. Wonder if somebody else has stumbled upon trying to make that work.

I have a k8s role with the same name that gets created in every namespace on during cluster provision time (not maintained by helm) that I need to update. Is there a quick / easy way to update the role for all namespaces (without having to go through every namespace)

Has anyone tried this? https://akomljen.com/meet-a-kubernetes-descheduler/

Guys any recommendation of a terraform plugin that can get me started with Kubernetes Nodes on EC2?

I decided to contribute 32 nodes to a community cluster.

And I need to provide them nodes that can join a community Kube Master in about 2 hours.

I really appreciate any suggestions.

Thanks!

You could have a look at: https://github.com/cloudposse/terraform-aws-eks-workers

Thanks.

We aren’t using EKS.

The Master is in another CLoud.

THe Nodes are being contributed from many AWS accounts by volunteers. 😉

And we are currently stuck at this error:

error execution phase preflight: unable to fetch the kubeadm-config ConfigMap: failed to get config map: Get <https://xxx.xxx.xxx.xxx:6443/api/v1/namespaces/kube-system/configmaps/kubeadm-config?timeout=10s>: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Not sure whats causign the timeout.

@Tim Birkett Nice Dog by the way! 🙂

damn quay being down again. 😞

i've never got to the point of actually implementing a pull through cache for docker

has anyone here done so and have lessons to share?

playing with harbor has been on top of my list of things to do for a little while now

maybe i'll do that today

Fellas, I am looking for industry recommendations of handling the alerts in Prometheus Server on EKS.

Currently, I am managing the alerts by editing the configmaps of the specific Prometheus pod and adding the alerts over there.

Is there a simpler way of dealing this type of requests (managing the Prometheus alerts)? Like API or exposing the configmaps through a different channel?

any pgpooler/pgbouncer implementation people recommend for kubernetes?

i had recently been looking into options for essentially doing a poor man's canary as i don't want to add a service mesh or yet-another ingress controller that supports weighting in front of my service. today someone pointed out https://argoproj.github.io/argo-rollouts/getting-started/ that "argo rollouts" is independent of argo cd, and that it does exactly what i want. in at least one case i have to use an alb-ingress-controller despite my distaste for alb's, and it has nifty integration there as well, https://argoproj.github.io/argo-rollouts/features/traffic-management/alb/#aws-application-load-balancer-alb.

what are you using for ingress @joey?

aws-alb-ingress-controller ... alb directly to my pods

ah - i was going to say, flagger supports nginx https://docs.flagger.app/tutorials/nginx-progressive-delivery

if it were my choice i'd be doing many things differently but i don't pay the bills 🤷‍♂️ i'm just jazzed to have learned that argo-rollouts does exactly what i want in the scenario i've been handed.

Hey, all. The k8s set-up that I inherited uses node-level AWS IAM roles. I'd like to set that up, so I can assign different roles to different applications (i.e., deployments/namespaces). Can anyone point me to some docs that will tell me just what that looks like? I've got the policies I need, for the most part. I'm just looking for some helm/k8s config that will apply roles at the service/namespace level. Thanks.

Has anyone tried to update the kernel of a Centos7 server? The image available on AWS is 3.1 and need it to be at least 4.8 in order to use kubespray. When I tried to upgrade it to 5.6 using the ELrepo, it would not complete the status checks after reboot.

I didn't understand the message I was seeing. The kernel requirment came from the network pluging, not kubespray. I switched to the default and the installation was fine.

Hi Folks, One basic question on kubernetes Container resource allocation..what is the default CPU and memory (both request and limit) value if we dont provide any values?

Hi guys
When a container is killed, and his PV/PVC use a storageclass with Retain Policy "Delete", usually the PV/PVC is destroyed from the cluster.
I have a case where the PV and PVC resources are kept and still flagged as "Bound". But the container is really gone.
Any idea what could cause this ?

Hi Everyone!

Please has anyone used Open Access Policy in Kubernetes for memory and cpu resource management and also to check for Readiness Probe?

https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work/ is a fairly clear discussion of PKI certificates used by kubernetes. If you work with the CIS Benchmark, you'll want to read this.