#office-hours - April 2020

@Ken Y.y welcome to #office-hours

any questions you'd like to have answered on our call today?

anyone seen an open source implementation of this api?

Yak Shaving: https://www.youtube.com/watch?v=8fnfeuoh4s8

love this one ☝️

Thanks for a great session!

@David Scott were you asking this week in office hours about how to run Github Action's Runners inside of containers, when the actions themselves run containers?

Hey @Erik Osterman (Cloud Posse), thanks for following up! I’m currently working with terraform github actions. Running them in a hosted runner in EKS fails to mount the code from the Checkout step into the terraform-github-actions container due to docker-in-docker volume mounting issues:

The docker run command tries to volume-mount the content from the Checkout step into the terraform-github-actions container. Because the docker socket is a volume mount from the EKS worker node, it ends up trying to mount -v "/home/github-runner/_work/_temp/_github_home":"/github/home" from the EKS node, not from the github-runner pod where the git Checkout happened, and terraform can’t find any code when it runs.

Sooooo the reason I reached out is @mumoshu just shared he's working on a solution for Kubernetes

@mumoshu has joined the channel

Check out https://github.com/summerwind/actions-runner-controller

I'm not exactly sure it will address your problems, but it was started as a way of addressing how to run GitHub Action's Runner in a containerized environment (#kubernetes )

@David Scott you were also running EKS, right?

Yes! This shows a lot of potential for me. I’ll poke around and see if it can handle things like adding the role annotation to the ServiceAccount, and a few other edge cases.

Thank you for showing me this.

Weren't we just talking about this? https://www.youtube.com/watch?v=Kx110kqoHo0

Btw, here's that chat bot (saas) for aws cloudwatch I wanted to share last week but couldn't think of the name

👉️ New Zoom Recording from our Office Hours session on 2020-04-01 is now available.

Is this once a week?

Yes, next one tomorrow

We have an Nginx ingress port question using websockets. Would this be an appropriate place to ask during the Office?

Thanks Erik. What time or it's an all day kind of thing

Ah 11:30 am PST got it

Tnx

See you there!

Should we post questions here if we can't join with video?

You can ask it here, but it will be easier if you're ont he call so it can be interactive.

Just jumping in car. Post question in 2 secs

In meantime as I type you might like http://getcommandeer.com/

A group of SRE’s in our company wrote this as a templating system useful for getting things out of vault and into helm deploys:
https://github.com/PremiereGlobal/stim
Goal is simplify Jenkinsfiles that are using Jenkins credentials to construct and run helm commands.

So we are trying to deploy a HA RabbitMQ deployment using Rancher, K8s and Docker.

We cannot open the port for RMQ websockets (or at very least it will not connect) port 15674.

The load balancer is landing on the proper port but will not fwd to the proper port. It does not detect anything on the other end. It is not HTTP and what is interesting is that the admin section of RMQ is HTTP, and it's port works.

It is TCP and we use WebSTOMP client library.

We see they port is open on the instance but the ingress LB will not connect. We tried to strip it down still no luck.
Basically a 503.

We tried different ingress' like HAProxy, Traefik, as well, logged in from console, but it didnt work. Even following the RMQ instructions very closely.
Seems like it is the ingress config because it doesn't work with any of them. So it doesn't seem like this is a NGINX issue to me.

Seems like a configuration issue.

Any ideas? Thanks guys.

Digital Ocean atm

Well even if we remove the LB from the discussion, lets just say the issue is the ingress

It is TCP btw

we are using the default NGINX ingress that comes with Rancher

the ingress binds fine to the rancher admin port

it does not work for binding to the rabbit mq webstomp port

WebSTOMP = Simple Text Over Messaging Protocol over TCP

ahhh

what LB did you use?

to summarize we cannot get ws:// or wss:// protocols to work at all
this is an ingress config issue IMO

but which config and how to configure - no idea. Have tried everything

I will get back to you after this. I might have an idea.

Awesome, sorry to overwhelm just thought it was me asking questions. REALLY appreciate that discussion

How did you deploy rabbitmq?

thank you

using the RMQ chat @Pierre Humberdroz

*chart

Have you guys ever used Fanout.io? Enterprise version of Pushpin.

https://github.com/summerwind/actions-runner-controller by @mumoshu

this is how to run github actions (runners) onprem under kubernetes

I ask because with all this RabbitMQ config issues we always had a plan to wait for Kafka to remove the Zookeeper requirement, and then considered Fanout reverse proxy just as good to use as a fanout broker

haha really?

The confluent guys seemed to communicate it was in process and that was late 2019

You guys ever use FoundationDB? We are setting up a JanusGraph on Cassandra and then are considering instead Foundation

Have played quite a bit with Neo4j

It is great, heavy lock in and expensive. We are going with JanusGraph. AWS and IBM using JG (based off old Titan)

It is a GDB

Plus Gremlin Query Lang is great

Well heck we cant get the RMQ stomp PORT open HA!

this is if you want to send slack notifications @mfridh

Have you guys tried Rancher K3's?

only k3sup from alex

Have you ever looked at Begin?

Just a little snippet that I learned to love. 🙂 Makes helm list -A so much nicer to look at:
yq w -i .deployment/$PROJECT_NAME/Chart.yaml appVersion $DOCKER_IMAGE_TAG

serverless stuff https://begin.com/

trying to learn about k3's more see if they cover our K8's use

Has anyone ever hit a Lambda limit?

You guys are awesome, thanks so much!!

yes.. I hate lambdas it gives to many constraints on the developer

Ya we are worried about it

I want to join @Pierre Humberdroz, how do I get the secret hand-shake for that one?

What is your slack handle in the kubernetes slack @Zachary Loeber?

👉️ New Zoom Recording from our Office Hours session on 2020-04-08 is now available.

This message was deleted.

Links from Todays Office Hours:
Highlights from the last week:
https://github.blog/2020-04-14-github-is-now-free-for-teams/
https://about.gitlab.com/blog/2020/03/30/new-features-to-core/
https://slack.engineering/deploys-at-slack-cd0d28c61701
https://zacharyloeber.com/2020/04/devops-patterns/
https://lethain.com//build-vs-buy/
Related to discussions:
https://sweetops.slack.com/archives/CBW0HJDS8/p1586886308064000:
• https://aws.amazon.com/macie/
• https://medium.com/tensorflow/introducing-tensorflow-privacy-learning-with-differential-privacy-for-training-data-b143c5e801b6

@Andrew Roth you mentioned that you would not use gitlab's container registry why is that?

👉️ New Zoom Recording from our Office Hours session on 2020-04-15 is now available.

Ho to all, can anyone help to to fetch the alerting details or data from Elasticsearch and send it to otrs ticketing tool using terraform. I read the aws kinesis using terraform but a confusion is still there. So I am asking here. If anyone worked with on this plz help me.

@Pratap join us tomorrow on the Zoom call and we'll try to provide some guidance on how to accomplish this.

Maybe worth mentioning on office hours today, two items (of several) that made the kubernetes podcast "news of the week' (https://kubernetespodcast.com/episode/100-community-redux/). One is Pluto from Fairwinds. They always seem to produce quality tooling, Pluto is used to discover depreciated apiVersions in Kubernetes . Another is Magicpak, an interesting tool for building minimal docker images without static linking.

This may be a FAQ,,, What's the most sane way to spin up fresh mySQL/MariaDB instances in containers..... the images in Docker hub don't have the right address binding to be accessible from the host OS and it was a P I T A !!

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

zloeber has joined Public "Office Hours"

Mateusz Sobczak has joined Public "Office Hours"

Jawwad Yunus has joined Public "Office Hours"

Sahil has joined Public "Office Hours"

Adam Crews has joined Public "Office Hours"

Neil Gealy has joined Public "Office Hours"

Pierre Humberdroz has joined Public "Office Hours"

Ronak Patel has joined Public "Office Hours"

I have a little show and tell today.

hari babu has joined Public "Office Hours"

Paul Barros has joined Public "Office Hours"

Raghu has joined Public "Office Hours"

Blaise Pabon has joined Public "Office Hours"

Andrew Elkins has joined Public "Office Hours"

Mike Martin has joined Public "Office Hours"

Christian Roy has joined Public "Office Hours"

Sheldon Hull has joined Public "Office Hours"

CHONAN TSAI has joined Public "Office Hours"

Bob Chen has joined Public "Office Hours"

Harry M has joined Public "Office Hours"

Mateusz Sobczak has left Public "Office Hours"

Adam Watson has joined Public "Office Hours"

Matt Gowie has joined Public "Office Hours"

Adedayo Akinpelu has joined Public "Office Hours"

James Wade has joined Public "Office Hours"

Omer Sen has joined Public "Office Hours"

James Wade has left Public "Office Hours"

Andrew Roth has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

https://github.com/pivotal/LicenseFinder <- for closed source

for Js I wrote this little test:
https://github.com/helm-notifier/service-template/blob/master/test/license-check.js

Adam Watson has left Public "Office Hours"

Libert Schmidt has joined Public "Office Hours"

Raghu has left Public "Office Hours"

Andrew Elkins has left Public "Office Hours"

Google's Open Sourcing Guidelines:
https://opensource.google/docs/creating/

Contributing guidelines: https://github.com/zalando-incubator/kopf/blob/master/docs/contributing.rst#sign-your-code

Adam Crews has left Public "Office Hours"

Marc Tamsky has left Public "Office Hours"

CHONAN TSAI has left Public "Office Hours"

Matt Gowie has left Public "Office Hours"

Is there a way to disable some certain fields not to written on terraform state file. For example i put my secrets as encrypted variables on my terraform.tfvars but i can see it unencrypted in state file

What if your tf repo shared with devs. It would be problem with security. I dont prefer secrets in tf vars

They are not clear text

They are encrpyted via KMS and show as encrypted

Chris Topinka has joined Public "Office Hours"

Mikael Fridh has joined Public "Office Hours"

But they can be seen clear text on s3 Bucket(which is also encrypted with KMS but visible on AWS console etc)

I read them and decrpt them on the fly

Personal appeal
I'm (in a rush) working on an interview exercise and need some help UN-phucking a docker-compose:
https://github.com/blaisep/hiring-engineers/blob/feature/blaisep/zero2datadog/docker/docker-compose.yaml
if someone can do me a solid and help me figure out why I can't connect to mysql from the host os, you'll have a friend in Santa Cruz....

Paul Barros has left Public "Office Hours"

hari babu has left Public "Office Hours"

my ci cd list

Adedayo Akinpelu has left Public "Office Hours"

Paul Barros has joined Public "Office Hours"

Are there any opinions about Tekton https://github.com/tektoncd/pipeline/blob/master/docs/tutorial.md

Sahil has left Public "Office Hours"

Ronak Patel has left Public "Office Hours"

Libert Schmidt has left Public "Office Hours"

Sheldon Hull has left Public "Office Hours"

Neil Gealy has left Public "Office Hours"

Bob Chen has left Public "Office Hours"

Christian Roy has left Public "Office Hours"

Pierre Humberdroz has left Public "Office Hours"

Harry M has left Public "Office Hours"

Paul Barros has left Public "Office Hours"

Andrew Roth has left Public "Office Hours"

Chris Topinka has left Public "Office Hours"

Mike Martin has left Public "Office Hours"

Jawwad Yunus has left Public "Office Hours"

zloeber has left Public "Office Hours"

Mikael Fridh has left Public "Office Hours"

Omer Sen has left Public "Office Hours"

Blaise Pabon has left Public "Office Hours"

Erik Osterman (Cloud Posse) has left Public "Office Hours"

👉️ New Zoom Recording from our Office Hours session on 2020-04-22 is now available.

Got an interesting case here. One of our internal applications grew organically over time. as we approached 30 internal users, things was beginning to slow. We can’t do a LB with many smaller instances due to the fact we have a couple of places using databases table as queue (this would result in some race condition - no locking mechanism). So what we ended up doing is separate all the async & recurring tasks and have them run on another server. We did this by setting up an elaborate CICD where the master repo would sync to a secondary repo (98% code is the same), then updates the primary instance and then updates the secondary async-task instance. This is providing user some relieve on the UX. But, the overall architecture is hard to maintain from a devops point of view. What would you do from here on to clean this up?
A) Clean up application logic so that there is no reliance on using db as queue. Using SQS to implement the queue then implement LB.
B) Instead of using a secondary repo, just keep everything in the master repo but have different set of config for the secondary instance.
C) All of above

What tools would you recommend for managing all this?

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Bre Gielissen has joined Public "Office Hours"

Edward Wizelman has joined Public "Office Hours"

@here public #office-hours starting now! join us to talk shop https://zoom.us/j/508587304

Jay Simoni has joined Public "Office Hours"

Pedro Galvão has joined Public "Office Hours"

Mario Feliz has joined Public "Office Hours"

Neil Gealy has joined Public "Office Hours"

Should be able to join in a bit. Can do some show-and-tell with https://github.com/RothAndrew/istio-practice/blob/master/eks/README.md if people are interested

The HTTPS section in particular feels really useful

Nice we can go over this

Mathieu Frenette has joined Public "Office Hours"

Gautam Sidhu has joined Public "Office Hours"

Mike Martin has joined Public "Office Hours"

CHONAN TSAI has joined Public "Office Hours"

Pierre Humberdroz has joined Public "Office Hours"

Cesar Sanchez has joined Public "Office Hours"

Andy Roth has joined Public "Office Hours"

hari babu has joined Public "Office Hours"

Sheldon Hull has joined Public "Office Hours"

Andrea Bolandrina has joined Public "Office Hours"

Sri P has joined Public "Office Hours"

Adam Blackwell has joined Public "Office Hours"

Andrew Elkins has joined Public "Office Hours"

Raghu has joined Public "Office Hours"

Raghu has left Public "Office Hours"

Sheldon Hull has left Public "Office Hours"

Neil Gealy has left Public "Office Hours"

Andrew Elkins has left Public "Office Hours"

❤️ https://github.com/RothAndrew/istio-practice/blob/master/eks/README.md#https I'll definitely run through these docs since I've been meaning to play with Istio more.

Blaise Pabon has joined Public "Office Hours"

@Andrew Roth flagger is what I am thinking about for managing the business logic of canaries

Gautam Sidhu has left Public "Office Hours"

I have to drop off at 3, but https://blog.newrelic.com/product-news/how-to-monitor-istio-service-mesh/

https://github.com/opendistro-for-elasticsearch/community/issues/25#issuecomment-522084664
Btw super awesome way for getting tracing working with elastic apm and jaeger

Cool! We also published some of the videos from observe 20/20 which taught me a lot about tracing: https://www.youtube.com/channel/UC3UV2PEUA9NvUOxTkXuEhuw

CHONAN TSAI has left Public "Office Hours"

raghu aderapalli has joined Public "Office Hours"

Adam Blackwell has left Public "Office Hours"

Cesar Sanchez has left Public "Office Hours"

Is there a tutorial or example of using cert-manager as the local CA ? That would be handy for my home lab....

Sri P has left Public "Office Hours"

raghu aderapalli has left Public "Office Hours"

Blaise Pabon has left Public "Office Hours"

Jay Simoni has left Public "Office Hours"

Mario Feliz has left Public "Office Hours"

Andy Roth has left Public "Office Hours"

Pierre Humberdroz has left Public "Office Hours"

Mathieu Frenette has left Public "Office Hours"

Mike Martin has left Public "Office Hours"

Bre Gielissen has left Public "Office Hours"

Erik Osterman (Cloud Posse) has left Public "Office Hours"

Andrea Bolandrina has left Public "Office Hours"

Edward Wizelman has left Public "Office Hours"

hari babu has left Public "Office Hours"

Pedro Galvão has left Public "Office Hours"

👉️ New Zoom Recording from our Office Hours session on 2020-04-29 is now available.