#office-hours - October 2021

@here office hours is starting in 30 minutes! Remember to post your questions here.

Sahil Kamboj has joined Public "Office Hours"

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Andy Miguel (Cloud Posse) has joined Public "Office Hours"

Michael Jenkins has joined Public "Office Hours"

Alexandr Vorona has joined Public "Office Hours"

Benjamin Smith has joined Public "Office Hours"

Florain Drescher has joined Public "Office Hours"

Ross Rfd has joined Public "Office Hours"

David Hawthorne has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Michael Manganiello has joined Public "Office Hours"

Ryan Ryke has joined Public "Office Hours"

Justin Davis has joined Public "Office Hours"

Erin Angerer has joined Public "Office Hours"

Yusuf has joined Public "Office Hours"

Adam Blackwell has joined Public "Office Hours"

Mukesh Sharma has joined Public "Office Hours"

Nate Faerber has joined Public "Office Hours"

Tim Gourley has joined Public "Office Hours"

Mike Crowe has joined Public "Office Hours"

Antarr Byrd has joined Public "Office Hours"

Arjun Venkatesh has joined Public "Office Hours"

Mike Crowe has joined Public "Office Hours"

Andrew Thompson has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

links from today's session:
• https://twitter.com/lukehoban/status/1443658655467900936?s=21
• https://github.com/hashicorp/terraform-provider-awscc
• https://www.hashicorp.com/blog/announcing-terraform-aws-cloud-control-provider-tech-preview
• https://www.pulumi.com/blog/announcing-aws-native/
• https://aws.amazon.com/about-aws/whats-new/2021/10/programmatically-manage-alternate-contacts-aws-accounts/
• https://blog.cloudflare.com/introducing-r2-object-storage/
• https://aws.amazon.com/blogs/mt/how-to-deploy-aws-config-conformance-packs-using-terraform/?utm_campaign=weekly.tf&utm_medium=email&utm_source=Revue%20newsletter
• https://github.com/cloudposse/terraform-aws-config/tree/master/modules/conformance-pack
• https://www.modulehub.io/
• https://github.blog/changelog/2021-10-05-github-actions-dry-your-github-actions-configuration-by-reusing-workflows/

Erin Angerer has joined Public "Office Hours"

Tony Scott has joined Public "Office Hours"

Geoff Weinhold has joined Public "Office Hours"

Patrick Joyce has joined Public "Office Hours"

Chocks Subramanian has joined Public "Office Hours"

Oscar Blanco has joined Public "Office Hours"

Zadkiel AHARONIAN has joined Public "Office Hours"

Mohammed Yahya has joined Public "Office Hours"

Patrick Joyce has joined Public "Office Hours"

Oscar Blanco has joined Public "Office Hours"

@Matt Calhoun, I feel like I hit a related issue regarding encrypted AMI (or at least EBS volume) in EKS using eksctl. I ended up falling back to some defaults that got me by so I can't say this would have solved my problem but maybe it's a data point: https://github.com/weaveworks/eksctl/issues/952#issuecomment-507219406

Is the principal listed here one you have tried?

Agnello Dsouza has joined Public "Office Hours"

@here office hours is starting in 30 minutes! Remember to post your questions here.

Oscar Blanco has joined Public "Office Hours"

Matteo Migliaccio has joined Public "Office Hours"

Jim Park has joined Public "Office Hours"

Mukesh Sharma has joined Public "Office Hours"

Phillip Hocking has joined Public "Office Hours"

Oscar Blanco has joined Public "Office Hours"

Carlos Ybere Rodrigues has joined Public "Office Hours"

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

Andy Miguel (Cloud Posse) has joined Public "Office Hours"

Tim Gourley has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

Benjamin Smith has joined Public "Office Hours"

venkata mutyala has joined Public "Office Hours"

Daniel Miller (CloudPosse) has joined Public "Office Hours"

Justin Davis has joined Public "Office Hours"

emem peace has joined Public "Office Hours"

Michael Manganiello has joined Public "Office Hours"

Sarah Donehower has joined Public "Office Hours"

Jason Skidmore has joined Public "Office Hours"

Agnello dsouza has joined Public "Office Hours"

links from today's session:
• https://aws.amazon.com/about-aws/whats-new/2021/10/aws-backup-backup-protection-aws-backup-vault-lock/
• https://sweetops.slack.com/archives/CCT1E7JJY/p1634138183131300
• https://rancherdesktop.io/
• https://www.atlassian.com/software/halp/jira

James Haughey has joined Public "Office Hours"

Patrick Joyce has joined Public "Office Hours"

Oskar Maria Grande has joined Public "Office Hours"

Mauricio Wyler has joined Public "Office Hours"

Othman Musleh has joined Public "Office Hours"

Florain Drescher has joined Public "Office Hours"

Marc Slayton has joined Public "Office Hours"

yong chen has joined Public "Office Hours"

Andrew Thompson has joined Public "Office Hours"

yong chen has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

I feel like Theo Schlossnagle's Art and Science of the Service Level Objective is a solid reference: https://www.circonus.com/whitepapers/art-and-science-of-the-service-level-objective.pdf

Ian Bartholomew has joined Public "Office Hours"

Next Week: What is DevOps 🧌

Does anyone have a demo app framework to recommend? Something along the lines of https://codebase.show/projects/realworld
but with an emphasis on infrastructure?

KHAI Anis has joined Public "Office Hours"

@here office hours is starting in 30 minutes! Remember to post your questions here.

Oliver Schoenborn has joined Public "Office Hours"

Guilherme Borges has joined Public "Office Hours"

David Hawthorne has joined Public "Office Hours"

Charles Sperbeck has joined Public "Office Hours"

Vicken Simonian has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

venkata mutyala has joined Public "Office Hours"

Mike Cox has joined Public "Office Hours"

Mauricio Wyler has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

Jason Skidmore has joined Public "Office Hours"

Ian Bartholomew has joined Public "Office Hours"

Nate Faerber has joined Public "Office Hours"

Jim Park has joined Public "Office Hours"

Aleksandr Fofanov has joined Public "Office Hours"

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Sarah Donehower has joined Public "Office Hours"

Andy Miguel (Cloud Posse) has joined Public "Office Hours"

Daniel Miller (CloudPosse) has joined Public "Office Hours"

Matteo has joined Public "Office Hours"

Tim Gourley has joined Public "Office Hours"

Matt Gowie has joined Public "Office Hours"

Justin Davis has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Antarr Byrd has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

Yusuf has joined Public "Office Hours"

links from today's session:
• https://github.com/mitchellh/terraform-provider-multispace
• https://docs.microsoft.com/en-us/exchange/recipients-in-exchange-online/plus-addressing-in-exchange-online
• https://www.pulumi.com/blog/introducing-pulumi-registry/
• https://github.com/FairwindsOps/saffire
• https://registry.terraform.io/providers/hashicorp/time/latest
• https://blog.cloudflare.com/automatic-remediation-of-kubernetes-nodes/

Mukesh Sharma has joined Public "Office Hours"

Obi One has joined Public "Office Hours"

Eric Berg has joined Public "Office Hours"

Florain Drescher has joined Public "Office Hours"

Vicken Simonian has joined Public "Office Hours"

Yo, Sherif here 👋
I am looking forward to join today’s office hours, I just discovered cloudposse community (although I’ve been using the modules for quite sometime 😄 )
I work as a SRE/DevOps/Infa/Platform ( whatever the name is it’s confusing 😂 )

I apologies for the wall of text 😄

---

My current work is moving our Infrastructure to a more modern setup, this include using Terraform, Kubernetes, and all the things in between.
We’ve built something very cool and I am proud of it.

We do have a Production, Staging and QA Environments ( 1 Per QA Member, about ~30 in total), and we have about 30 Service.

Ironically I modeled our Infrastructure in a way very similar to SweetOps 4 Layer model.
Aside from the Foundational, Platform, and Shared Services part, my main point of struggle that I am very eager to improve is the Application Layer.

Especially the Application Infrastructure (call it AppInfra) which are the Application 1-to-1 Infrastructure components such as S3 Buckets, RDS, CloudFront. and Let me explain what I want to achieve.

I just made a deal with my workplace to give me a AWS Account to build a Greenfield virtual company that I’ll try to explore good practices using. And I am trying to solve some problems we’re already having (but without caring so much about disturbing business, it will be a greenfield project)

So

Our current way
1. Applications are Deployed using Push Model, thus mean, CI (jenkins sadly 💔 ) will Apply the Terraform part, then proceed with Applying the Helm Chart.
2. We use Terraform to release the helmchart, which is a very intuitive combo in theory especially when passing Terraform outputs to the Helm Values.
My Goals:
1. GitOps GitOps GitOps… I want to implement GitOps ( the reconciliation part is what is missing atm)
2. Implement Feature Branch Preview Environments
3. I tried to shift any thing possible to Kubernetes, using the service’s helm chart (it’s just soo intuitive imo), this includes the:
a. All application components like Webservers, Consumers, Jobs, and CronJobs are in the Chart.
b. Route53 Records using External DNS
c. Monitoring and Alerting Using Prometheus Operator
d. Grafana Dashboard using Grafana Operator
e. And of course Kubernetes stuff like Autoscaling, KEDA, etc are all part of the Helm Chart.
4. However… I can’t use Helm Chart for AppInfra yet…
a. I looked into Crossplane.io, and AWS Operator, but they’re not there yet imo. Infrastructure is different, The CRDs are fine for the optimistic situations, but letting my Kubernetes Operator do RDS Restore, Scaling, or other tricky Infrastructure tweaks ( The DB Migration analogy comes to mind), I don’t feel Kubernetes Operator pattern is the best method here.
b. Using Terraform to deploy changes in the PRs (before merging and before Argo does its sync) seems fine. We won’t deploy our App to Kubernetes unless Terraform is in sync.
c. However… This breaks when dealing with highly dynamic environments, when I have like ~5 interdependent services, each one has their own AppInfra, I can’t use GitOps methods to just point Argo to sync them. I’ll need to deploy them 1 by 1, as well as Cleaning Them Up.
d. And now I will use my CI to handle this deployments and ask it to trigger 5 services, then ask it later to go destroy them too. and I am drifting away from Kubernetes, and more coupled to my CI to do all the operations.
I am asking, is there is a better way to do it ? What do you think ?

Questions ( for OfficeHours)
1. How do I deploy on-demand environments (say ~5 interdependent services) using GitOps & Terraform together ?
a. Using:
i. Terraform to Apply the App Infrastructure (1-to1 Infra)
ii. ArgoCD for syncing the Chart itself.
b. And avoiding maintaining the services dependency tree and triggering my CI in specific sequence to provision an environment ?
2. How to pass information between Terraform and the Helm Chart (deployed via Argo) ?
a. For Testing/QA we have DB Snapshot Volumes that we will use Terraform to clone, and pass it for the HelmChart
b. The HelmChart then uses to create a PV & PVC for the DB Pod.

@here office hours is starting in 30 minutes! Remember to post your questions here.

Emile Fugulin has joined Public "Office Hours"

Emile Fugulin has joined Public "Office Hours"

Yusuf has joined Public "Office Hours"

Erik Osterman (Cloud Posse) has joined Public "Office Hours"

Vlad Ionescu has joined Public "Office Hours"

Zachary Loeber has joined Public "Office Hours"

Andy Miguel (Cloud Posse) has joined Public "Office Hours"

Sherif Abdel-Naby has joined Public "Office Hours"

Benjamin Smith has joined Public "Office Hours"

Guilherme Borges has joined Public "Office Hours"

Jim Park has joined Public "Office Hours"

James Haughey has joined Public "Office Hours"

Oskar Maria Grande has joined Public "Office Hours"

David Hawthorne has joined Public "Office Hours"

Justin Davis has joined Public "Office Hours"

Matt Calhoun has joined Public "Office Hours"

JB Beck has joined Public "Office Hours"

Kyle Korth has joined Public "Office Hours"

Yusuf has joined Public "Office Hours"

Daniel Miller (CloudPosse) has joined Public "Office Hours"

Antarr Byrd has joined Public "Office Hours"

Matthew Zeemann has joined Public "Office Hours"

Tim Gourley has joined Public "Office Hours"

Jim Antoniou has joined Public "Office Hours"

links from today's session:
• https://www.datadoghq.com/blog/dash-2021-new-feature-roundup/
• https://www.datadoghq.com/blog/universal-service-monitoring-datadog/
• https://www.armosec.io/blog/new-kubernetes-high-severity-vulnerability-alert-cve-2021-25742
• https://aws.amazon.com/about-aws/whats-new/2021/10/amazon-rds-proxy-supports-amazon-rds-mysql-version-8-0/

Mohammed Almusaddar has joined Public "Office Hours"

dario erregue has joined Public "Office Hours"

Isa Aguilar has joined Public "Office Hours"

Marc Tamsky has joined Public "Office Hours"

Hao Wang has joined Public "Office Hours"

Sam C has joined Public "Office Hours"

Tony Scott has joined Public "Office Hours"

Taylor M has joined Public "Office Hours"

Github actions telemetry => Datadog by Scribd (https://github.com/scribd/github-action-datadog-reporting)

https://aws.amazon.com/blogs/compute/accelerating-serverless-development-with-aws-sam-accelerate/ and https://aws.amazon.com/about-aws/whats-new/2021/10/aws-load-balancer-2-3-alb-ipv6-targets/

Patrick Joyce has joined Public "Office Hours"

Sherif Abdel-Naby has joined Public "Office Hours"

Michael Jenkins has joined Public "Office Hours"

Anere Faithful has joined Public "Office Hours"

Vicken Simonian has joined Public "Office Hours"

Blaise pabon has joined Public "Office Hours"

Just (officially) announced: https://github.blog/changelog/2021-10-27-github-actions-secure-cloud-deployments-with-openid-connect/

Eric Berg has joined Public "Office Hours"

@Erik Osterman (Cloud Posse) you might like that

it is new

^ @Erik Osterman (Cloud Posse) I think the video is corrupted 😄