#general - February 2020

2nd interview for my first DevOps type role today! 😎 Anyone want to share tips?

#jobs had some good discussion on it recently. i'm not sure how much is relevant to a first timer though

Ya some good interview questions were posted in #jobs. Honestly, it comes down to the interviewer and their style of interviewing.

Hey everyone, give a warm welcome to our newest members!
- @MorganGeek
- @Viktors D
- @Prasanna Pawar
- @Almas Hilman Muhtadi
- @Dan Griffiths
- @Sharanya reddy pagidi

Good to have you here =)

thx for the hard work on that elasticsearch module thats really nice.

Thanks @gyoza!

Hey everyone, give a warm welcome to our newest members!
- @cia
- @Josh Hudson
- @BitsnBites
- @Maciej Kozlowski
- @Jason Carter

Good to have you here =)

☺️ thanks!

Thanks 🙂

Hey everyone, give a warm welcome to our newest members!
- @Mike Martin
- @Zack Hewison
- @Miranda Pearson
- @Silke Van den Broeck
- @Tony Stark

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Norbert Fenk
- @Dhrumil Patel

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Rustem Abdrakhmanov
- @Andrew Cameron-Douglas
- @Tom Howarth

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Aaron Lennon

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Stoor

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Geoff Weinhold
- @Julius Blank
- @Conti Mattia
- @Meg Yahl

Good to have you here =)

Dang, the daily new member list is getting larger and larger. The cult of devops is spreading....

come closer to the light!

Question on secrets for things such as DB passwords etc.
Does anyone keep the originals in a Git repo (encrypted of course)?
We do and I'm questioning myself on whether we should.

Pro: We have the master copy offline from the K8S cluster and in the event of a total failure we still have access to the original passwords.

Cons: Do we really need to keep yet another copy of the passwords?
They are in the Azure Keyvault/Google Secret Manager (where apps pull them them from)

Anybody have any good reasons to go either way?

@Patrick M. Slattery I mean, it sounds like you’re facing the burden of keeping passwords in more than place. What about a centralized secrets management solution such as HashiCorp Vault?

Pros:
• Secrets are in one place
• Failure scenario: Vault’s storage can be HA, for example with Consul as the data store. Even in total failure, you will be okay if you have enabled Consul backups to AWS S3.
• Cloud agnostic - i.e. you’re not relying on AWS KMS, GSM, etc
• Dynamic Secrets Engines (You can give admin credentials to Vault and it will issue and automatically revoke temporary DB credentials)
• A k8s mutation webhook controller which essentially acts as an operator to allow injection of secrets to pods https://www.vaultproject.io/docs/platform/k8s/injector/index.html
Cons:
• Expensive to run HashiCorp’s reference architecture of 5 Consul nodes + 3 Vault nodes (https://learn.hashicorp.com/vault/operations/ops-reference-architecture) - Vault integrated Raft storage is still in beta until 1.4.0
• You’re running a third party application on top of everything you’re already running
• Re: Failure scenario and automated backups - for non-Enterprise Consul you need to manage automated backups yourself

Yeah, I've heard a couple of Vault horror stories that keep me away from it.
We ourselves initially used Consul for state management in our app and had several disasters with it. It would be hard to persuade anyone here to run Consul again.

That said the pros are all very much what I really want...

You can use AWS secrets manager (if on AWS), but it becomes really expensive if there are large number of secrets to be stored (we are also in the same boat right now on which secrets manager to use though)

We are currently using Azure Keyvault / Google Secrets Manager (We are moving away from Azure though)
Price for either is not anything crazy but then again we only have a few dozen secrets in each instance at most

Google Secrets Manager is nice but is pretty light on features being so new (Still in beta)

as long as the encryption is well protected i (depending on the workload) don't see a massive issue with using git as the source of truth as long as they are that.
things like needing to rotate at short intervals and needing to audit retrieval of secrets would change that recommendation

vault and consul are great tools, but they're not without operational overheads - they need to be kept running and maintained

if you can't commit to the overhead required to run them you're introducing a weakness into the system rather than a strength as they'll quickly end up in the critical path

what are everyones favorite videos for learning terraform, helm, kubernetes, etc? please share with me!

I don't use videos for learning much but thus far I think that Kubernetes In Action is the current bible of kubernetes.

Same, videos tend to be too slow-paced for me, and I'm a quick reader. I'd rather read, pause where I want, test, play, search for answers...

A coworker of mine has enjoyed the LinuxAcademy content, I can't vouge for it but it's helped him learn stuff.

Kubernetes - https://github.com/kelseyhightower/kubernetes-the-hard-way

Also, courses on udemy from Edward Viaene, for Kubernetes, terraform and Prometheus are good ones I can think of

Hey everyone, give a warm welcome to our newest members!
- @Siraj Rauff
- @Eric Yang
- @sekhar modu
- @Vidhya Vijayakumar

Good to have you here =)

Helpful feedback @Zachary Loeber @Alex Siegman

Let me rephrase this to what learning materials have ya’ll found the most helpful

I am trying to curate some content to help newcomers to grok the concepts and give them a place to get started

I am personally searching for content and overwhelmed with how bad most of the stuff is (well, to be kind - just not the way I would explain it), so it’s not surprising that if someone is a newcomer to this stuff how hard it is to find good learning materials

For people brand new to kubernetes, I've found "Kubernetes Up and Running" pretty invaluable for learning the basic concepts. It's available for free from a few places, I haven't found a give-information-free link in my quick google, but https://azure.microsoft.com/en-us/resources/kubernetes-up-and-running/

Or you can buy it from o'reilly / amazon

I think it helps to be focused on what you're trying to learn, and more importantly, for what purpose. Example being, Up and Running is a great book for anyone who has need to understand basic kubernetes concepts. But if you're a developer who has to make the program and stuff it in to a container, will that necessarily help you? I find it harder to find learning materials at the level and "depth" around a given concept than materials for a tool or concept in general

that's a good point.

Another great example is python books. There's a million "learn python" books, but all of them cover the same boring stuff. What about a "learn python for someone who already knows python" book - where it goes more in to style, architecture choices for python programs, good designs for whatever, etc. I don't need to be taught python, I want to learn/explore idiomatic approaches to problems in python, learn to use the language elegantly and efficiently, etc.

"here's what you need to know as a developer...:"

@Erik Osterman (Cloud Posse) can i be added to the terraform channel, kicked myself out by mistake 🤦🏻

Hrmmm anyone can join any channel :-)

Even if you leave one...

good to know, thanks

hii 🙂

Hey everyone, give a warm welcome to our newest members!
- @Sudesh Lalmal Pathirana
- @Mattia Bertorello
- @Olivier

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Kevin Hetman
- @devops-bot
- @acs
- @Örjan Sjöholm

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Jim B
- @Christoph Gerkens

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Frankie.li
- @CC
- @Pawel
- @Alex Kiss

Good to have you here =)

Thanks!

good to be here

Is this the right way to version pin helm to v2 in buildharness .33?

apk —update —no-cache helm2@cloudposse && update-alternatives —set helm /use/share/helm/2/bin/helm2

?

After this helm version -c correctly shows version 2.16.1

Couldn’t figure out if there was a make target that would allow packages to overwrite helm symlink to the specific one I want

@Vincent De Smet there are a few different concepts that are getting mixed together

there's the build-harness docker image that has packages installed

then there are the install targets. the install targets are designed to work across platforms, thus do not use apk which is specific to alpine linux

Hey everyone, give a warm welcome to our newest members!
- @ngan nguyen
- @Vincent De Smet
- @Rico
- @Ievgenii Shepeliuk

Good to have you here =)

I whipped up a list of practical recommendations for "12 factor" apps on Kubernetes. Appreciate any feedback. Did I miss anything?

https://cloudposse.com/12-factor-app/

(the poly repo thing is opinionated - realize that's not a requirement; this list is meant for our customers)

Flags should enable/disable functionality without knowledge of stage or environment (e.g. do not use if ($environment == 'dev') { ... })

THANK YOU!

it's a good opinion in my book - mono repo is an overhead, doesn't make sense for most systems - and when it does you should know that it does

i do think that things that are versioned together should be in the same repo though - if you've got two services that are tightly coupled and need to be released together (you shouldn't paint yourself into this corner) then it will probably be easier to control that if they're in the same repo

https://www.hashicorp.com/resources/closing-keynote-terraform-at-google
Kind of an interesting take on the monorepo

(see Single version policy in the transcript)

Health checks should not depend on the health of the backing services - this is an interesting conversation to have. usually boils down to the difference between "healthy" and "ready"

I’ve seen something along those lines here https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q - Paper written by a former Google SRE

@Yonatan Koren is apparently the human encyclopedia of relevant links at a moments notice

lol this is what i get paid for 😅

jk

Hey everyone, give a warm welcome to our newest members!
- @Mario Gonzales
- @George Kontridze
- @Vinayak Singh
- @Anthony
- @Daniel Blue

Good to have you here =)

Is this the place to ask a trouble shooting devOps technical question. I don't want to pollute

Yep! There are different channels for specific issues (e.g. #aws, #release-engineering, #terraform, #terragrunt, and so many more)

Hi Matty, yes I didn't see the one that might be appropriate for a Rancher, RMQ port issue

ya, maybe #kubernetes

some rancher users there

@UUB28NLDS can you keep tabs? 😃

Hey everyone, give a warm welcome to our newest members!
- @Chuck B
- @Marco Ceppi
- @nafas muhammad
- @UUB28NLDS
- @Nabil Becker

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Mohamed Meabed
- @Richard Gomes

Good to have you here =)

Hey! awesomes - glad to be here 🙂

asdf-vm, a wicked cool project I tinkered with last night comes with a backstory written as a ballad -> https://github.com/asdf-vm/asdf/blob/master/ballad-of-asdf.md

Hey everyone, give a warm welcome to our newest members!
- @Stef
- @Sia
- @kimxogus
- @Jey Ayyalu
- @Zorin Wade
- @Francisco Montada
- @Sreekumar
- @Andy To
- @Abhishek Gupta

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Milosb
- @Dragos Andronache
- @Jeremy Schuller
- @dan
- @Thuong
- @Moritz S

Good to have you here =)

TY TY

Hey everyone, give a warm welcome to our newest members!
- @bougyman
- @joey
- @Oliver Smit
- @RB (pat leave again)
- @Kendall Link

Good to have you here =)

Hello all. I keep coming across SweetOps archives in Google searches, and thought I would check out the Slack. Judging from my brief skimming of #random… I’m glad I did.

Wondering if something about chart versions I'm not grok'ing... I've seen some charts where its version looks intentionally locked/coupled to its appVersion . Seems like a lot of potential to muck with the semver in my helm values. Hasn't happened to me personally yet, but in a forked chart that I maintain, I'm debating whether to break that coupling to appVersion. Curious if anyone hear has thoughts on that.

github is having issues. does anyone setup a readonly internal mirror for when this happens?

Nah, the alternative is much better, pack your things and go home 😄

im a remote engineer, so im already home. 😄

ive been googling and havent seem to find anything. maybe github up time has been so good that no one had to worry abou tit

We had it once in a company

I dont remember the details since it was setup before I came

i usually have all my repos pretty up-to-date locally, and write tests and build scripts in a way they can be run locally instead of only through CI

but it was a cron that synced one critical repo

sounds super bespoke

as for the actual server https://git-scm.com/book/en/v2/Git-on-the-Server-Setting-Up-the-Server

should be straight forward

ah ok. i was hoping for something more cookie cutter, off-the-shelf. basically i give it our enterprise org name and an auth token and it grabs all our repos and has an endpoint

could run a cron that just loops through your local clones and runs git fetch periodically?

yea but then thats a solution only for myself

ahh

well gitlab has some pretty great webhook based options to mirror from github

every push to github is then synced to gitlab

@RB (pat leave again) you jinxed it https://www.githubstatus.com/

lol... it was causing issues this morning for us on the east coast so i was looking at alternatives in case it happens again

would be nice to have an artifactory like approach for git* service providers like github

so you first clone/pr/commit/etc to the mirror, which then redirects the changes upstream to github. if github is having issues, it won't stop developers. we can simply wait for github to come back, and resync our mirror with upstream

but it doesn't seem like this solution exists... so if anyone wants to make a lot of money or contribute to the open source community by hedging that github will have issues like this in the future, then ill be your customer 🙂

git proxy or git cache turn up a couple options. not sure how robust they would be for a team, or if availability would be any better than GH

Hey everyone, give a warm welcome to our newest members!
- @Steve Neuschotz
- @SATYA PATI
- @Jesse
- @Bogdan Lata
- @Steen
- @Martin Leopold
- @igal
- @hari
- @Alan Rickman
- @Scott Barr

Good to have you here =)

got a question for anyone using GCP. if you have an app running under a service account in project A, but you want to access resources in project B (from that app), what's the best way to do this? i've seen API keys, service account impersonation, etc. thrown out there. curious what people are actually doing in real applications, though.

Hey everyone, give a warm welcome to our newest members!
- @Jawwad Yunus
- @Castro Mbithii
- @Nishant Gupta
- @Raymond Butcher
- @Adam Perry
- @Marcin Brański
- @Tan Quach

Good to have you here =)

Hey everyone, give a warm welcome to our newest members!
- @Victor Wong

Good to have you here =)