#aws - April 2020

Hi just wondering, anyone having some terraform modules to setup an EKS cluster with ondemand (only when needed) gpu scaling for KubeFlow?

Hey Folks
i need to make docker swarm cluster on EC2 with auto scaling in Terraform scripts
how can i add new scaled up Ec2 to docker manager setup

NOOB here please help

How can I update an RDS master password in a way that my backend connections don't have downtime? We currently are passing in the database URI onto our servers via an SSM param

Hey guys, IAM related question. Suppose company A aws account has for instance three users who access the company A aws accounts using company SSO (auth linked to Azure AD). These 3 users additionally want access to a different company B AWS account (access to S3 buckets in company B account, both read/write). Requirement is, to use the SSO based credentials of company A. How can this be achieved? Requreiment is NOT to create separate IAM users in company A (and then trusted relationship), but to use SSO based temp credentials (24 hour tokens). Does anyone has any insight on how it can be achieved?

I'm looking for scripts to make 2 common AWS actions easier:

1. A script to simplifiy starting a session on an instance, ideally something that lists instances, autocompletes names, etc.
2. A script that will 1) execute a state manager association and then poll until it finishes and output the log (from the s3 bucket) to my console
Anyone know if something like these exist?

Hi, We have been using SSM parameter store and chamber with clousposse modules and everything is good but now we have multiple teams that need access to RDS and we have testing users added in and we added the password to Parameter store and we allow each user to assume a role that can decrypt and read the parameter but it is pretty cumbersome and prone to user error. We have been looking at RDS IAM auth which is too cumbersome so I was wondering what you guys will recommend in this situation ? I thought about setting the users with chamber to read the secrets or something like that

I did a thing which might be relevant to y'all: https://twitter.com/iamvlaaaaaaad/status/1247929989137403905 ( hope useful self-promotion is allowed here --- if I've failed please let me know )

Anyone here have experience with NLBs? I’m looking at trying to expose an elasticache redis instance to the internet by putting an NLB in front of it

The NLB’s target group sees the redis instance as healthy, using an IP target and TCP health check. But hitting the NLB with redis-cli ping, I get a timeout

I’m sure there’s a reason why this won’t work, and just curious what it is

do NLBs handle the egress traffic as well as the ingress? Or does an NLB only handle ingress?

Hey Guys
need help with RDS backup
i have scenario in which many site data is stored in single RDS
need to automate backup for particular site data and store it to S3

How are folks monitoring their Fargate clusters/containers?

Hey folks — This is more of a bash question, but I’m sure there are aws-vault users here and folks have dealt with this before so I figure I’ll have a quick answer…

Issue is if I’m using aws-vault exec gowiem --no-session -- $ALIAS_OR_FUNCTION I get errors similar to the following:

aws-vault: error: exec: Error execing process: exec: "tfplan": executable file not found in $PATH

I have a bunch of functions and aliases for my Terraform workflow, so this throws a wrench into my works when I don’t want to specify a specific profile in my provider and instead want to use aws-vault. Does anybody know how I can have aws-vault pick up my non-PATH based commands?

example of your alias command?

Hey everyone 😃

Are there any examples of a good EC2 AMI delivery pipeline across multiple stages/accounts? Our system is all EC2 instance based, running services in ASGs behind an ALB. The ‘pipeline’ such as it is at the moment, is just building AMIs with packer and then running terraform to swap the ASG out (we have the ASG name linked to the AMI creation date, so it creates a ‘blue green’ AS swap). Fairly manual at the moment although we’ve got some limited scripting around these to allow jobs in Rundeck. I’m just not sure how we’d best move an AMI through several stages in an more automated fashion. We don’t release very quickly is the other problem (which I’m hoping to accelerate)

guys is there a standard on CloudPosse terraform outpus ? I see it is not Json compliant

for example:

[
  {
    "autoscaling_groups" = [
      {
        "name" = "eks-72b8b964-bfb4-e582-36d5-a0cb231a4ab5"
      },
    ]
    "remote_access_security_group_id" = "sg-01f490d95dcc8f35b"
  },
]

which command you use to parse this ?

jq is not parsing this

@Omer Sen The JSON is malformed

terraform output -json eks_node_group_resources | jq -r
[

this does the trick

Just found it

Anyone here familiar with RDS Event Subscriptions? I have one made by Terraform that says Status is NULL and im unable to edit it. If I make one manually in the console it works just fine. Same settings.

hey guys,
running ssm run(Applyansibleplaybook) command but ansible using python2 but when i run locally(EC2) its working fine
help

You want ansible to run with python3?

Ensure ansible_python_interpreter: /usr/bin/python3 is in your vars

thnx @Abel Luck

Hello Guys
I m looking for scripts that can trigger Gitlab-CI from AWS lambda and trigger lambda via slack

please

I don't know of any off the shelf scripts that do that, you might have to write your own. Gitlab has docs for triggering pipelines via the API here https://docs.gitlab.com/ee/ci/triggers/README.html

We have a CICD pipeline on AWS. Written entirely in cloudformation templates. But the annoying thing is we can’t delete stuff in S3 after the stack is done. Any idea how to work around this so we delete everything when the stack is no longer needed.

The bucket you tried to delete is not empty. You must delete all versions in the bucket. (Service: Amazon S3; Status Code: 409; Error Code: BucketNotEmpty; Request ID

Hello,
I am a bit confused with how Cloudfront works with an S3 SSE-KMS bucket (Cloudfront uses OAI to access the bucket). I am able to fetch an index.html file perfectly fine by hitting the CDN DNS in the browser when AES encryption is applied. However when SSE-KMS encryption is enabled on the bucket, kms.unrecognizedclientexception shows up for the same page. What needs to be done in this case to enable access to the html page residing in the bucket to be accessible by CDN ? Why is the decryption not transparent in this case ? Any pointers appreciated.

Here’s a weird scenario. We’re playing with AWS Global Accelerator and have it pointed at a public facing ALB, which is fronting a bunch of EC2 instances. The GA is referenced by a Route53 record. We have another Route53 record that points directly at the ALB.
ga.mydomain --> globalaccel --> ALB
alb.mydomain --> ALB

In the course of some other network troubleshooting, my colleague went onto one the EC2 instances and ran a long series of queries and pings to the Global Accelerator R53 record, and to the ALB R53 record. When we compared the results, the Global Accelerator was 2x faster than going ‘directly’ to the ALB itself. We’re not sure how that works, since the GA is based on ‘edge’ locations, while the ALB nodes should be co-located with our instances. This seems like voodoo magic and we’re not sure what to make of it.

Global accelerator is a basically layer 4 CDN in a way

that is why we use it

it will route your request trough a faster aws network (same as cloudfront) and will server the request back from a router closer to your location

same as a CDN

yah but if the requestor is already on the aws network (ie, an ec2 instance) and the target is the ALB, shouldn’t that already be the fastest route possible?

it just seems to defy our undestanding

ohhh you were doing your test from whiting aws ?

yes thats what was bonkers about it

nothing should be faster than doing it from the same vpc

except the GA was 2x faster

mmm but you were doing tests to the public ip of the alb from a private subnet inside the vpc ?

Yes - we have a not-so-great setup right now with our APIs

you could do a traceroute and see what is the path is taking from each place

and you could add an internal alb and do the same test

I wonder if the “firewall” effect of the IGW could be adding to much time to your request

that’s related to why we were checking this in the first place. We have a bunch of go apps and the net/http lib was complaining of ‘client header timeouts’ (we had it set to a 5sec max time for the response) - we did a whole bunch of testing, and we found that sometimes the ALB would take as long as 8 seconds to ‘ack’ the request

if you do a test from the private subnet, it needs to go out trough the nat gateway then igw then BGP aws routers , back to the IGW, ALB backend and then response out again to the IGW , aws BGP , IGW, NAT gateway and finally instance

so we wired up some tracing and found that these bad connections in the applications were taking up to a minute to get the response back from the alb. We suspected maybe we had a bad node or something, so we blew away the ALB and tried a new one. Same issue.

if you have internal service that talk to each other you should never use the a public alb for intercommunication , that is going to be slow a not so secure

only internal albs

or you can use app-mesh too

yah, the slowness is now evident. the security we do an auth check on every call - which is how we found this problem because that suddenly started killing a lot of requests

when I worked at a big game company, we used to have like 5 public ALBs and like 20 internal albs

we even have ALB with special rules for the same service but different clients

kinda like treating the ALBs like api gateways

but internal facing only

Yes thats how the ALB is run right now, its basically an API Gateway

bu if the alb does not need access from clients or external integrators, then it should be only internal

but, all our services are run off EC2 right now, I’m working to migrate that

I see ok

yah agreed - the problem I stumbled upon is that our actual client apps talk to these internals too, so I still need the public facing ALB to route to the internal services. I’ve been pretty suspicious of having our apps talk to each other “internally” through the public ALB, so this is good confirmation

And unfortunately App Mesh isn’t HIPAA eligible yet

so basically you will endup with the setup we had a public facing alb for external clients and and internal for inter-app communication

you will double up the cost of albs but I think it will be faster

ygotcha

thanks for confirming, we’re gonna run some tests and I’ll start working on a reworked architecture…

let us know how the tests go

about the GA speed is because the ACK will come directly from the GA node and it will route it to the ALB

so you mean it will not go out to the internet and then to GA ?

I think it does, but the GA does a ‘parallel’ connection to the ALB while its sending you the ACK.
But even still, the GA endpoint is ‘edge’ and so I would still have thought that the ALB endpoint in the VPC itself would respond faster

Also still unclear why we have these weird connections that just ‘dangle’ for almost a minute before the ALB even responds to the initial connection

time to enable vpc flow logs

I think our use of the public facing ALB for ‘internal API chatter’ breaks the usefulness of the flow logs to a degree, since the traffic keeps going out and back in. I’m just going to take this is the time to get this broken apart correctly, it had irked me when I first saw it and now I have some actual weight to say its not right

also, if you're going internal instead of Internet egress and back in, there's at least a chance that you'll be within the same AZ and thus won't pay 100% cross-AZ traffic.

From time to time I get bots trying to scan our AWS servers. I often get this error in the application log. I am trying to find effective ways to stop this higher up in the stack. Maybe at the nginx level or cloudflare level. Any recommendations? This is obviously not AWS specific but that’s where I am seeing happening most frequently for us.

Invalid HTTP_HOST header: '<http://example.com|example.com>'. You may need to add u'<http://example.com|example.com>' to ALLOWED_HOSTS.

Hey folks, does anyone know what the typical way to run cloud-nuke or similar tools is? Is everyone re-inventing their own solution to run that on a schedule or is there a terraform (or other) solution out there that will spin up a small instance and run that periodically without nuking the instance as well?

How would you approached RDS (mysql) migration to another account while introducting cloudformation and minimazing the downtime?

Vpc peering and read replicas is how we do it if we can't have downtime

yeah, if there wouldn’t be a cloudformation in the mix it would be quite simple. AFAIK (and correct me if I’m wrong) if you have a cloudformation managed read replica when you remove the sourcedbidentifier and deploy cloudformation will delete the read replica and create a new database

so maybe instead of relying on the aws to instantiate a read replica I could just create an instance using cloudformation and then configure it manually to replicate from master

I hate cloudformation so I can’t give you an answer but you could split it in a different cloudformation and have a nested stack or something like that ?

I have not used cloudformation in a long time

I remember there was something like a reference stack

I’m already using nested stacks. AFAIK what you’re proposing is not possible because cloudformation will delete an instance when you change/delete source_db_identifier attribute and will create a new resource in place of read replica.

What I could actually do is the create a read replica manually, promote it to be master and then pull it in using cloudformation’s import resource, but it’s such a pain to work with import and I would like to skip that part (-:

and people complain about terraform…..

yeah it’s using the most shitty dsl you can have😄 i feel like writing puppet manifests when I’m working with terraform …. sooo 2005

well I can tell you that in TF I have multi regions cluster and if I delete the source_db_identifier it does not delete the instance

lol

I don’t mind either of them. it’s just a tool and depending on the context you can use it on not.

what about aws cdk? Is it the same as using vanilla CF ?

in this case cloudformation is used across all other projects and it’s easier for everyone else to pick it up if needed.

@Nikola Velkovski no, AFAIR in cdk you define your infra in a language you use (e.g. in my case it would be python) and then it’s generating the templates for you in the process

IMO that's really the only valid way to use CFN... render the templates. the templates are an artifact of the release. trying to manage them directly is a big fail

but, side tracking. i have nothing helpful to contribute to the question at hand. sry, just early morning venting for me

@loren IMO by rendering templates by tools like cdk you remove the ability to fine tune your environment because not every option is supported.

Also with cdk generating the cloudformation templates you end up in a situation that some features are not available in cdk which are alrady available in cloudformation so you create additional boiler plates just to make it work. at least in native cloudformation you have custom resources which are closing the gap of stuff not available natively in cloudformation

just my experience talking. cloudformation is way too limited without something writing out the template for you ¯\(ツ)/¯

as I said, I don’t mind it the same way as I don’t mind working with terraform or any other tool. it’s just a tool which depending on the context will be used or not

i just get tired of the boilerplate in CF. i want a way to define a library of common use cases and organizational standards, and use those across all my templates. copying and pasting it throughout every CF template is unmanageable. we've tried getting there with other CF features, like transforms, but those require very special handling and their own deployment/versioning workflow. it's just annoying. much easier to do though with something like the CDK

there is this thing called sparkleformation

it tries to make CF better

invented before CDK

now CDK is far better than CF it self

but TF is better 🙂

has anyone tried pulumi ?

a none of this solves @Maciek Strömich problem

I have not tried but I have been looking at it

@Nikola Velkovski I was looking at it recently but the pricing put me off a little bit.

Just discovered AWS param store does not allow you to store values containing '{{{ .. }}' in them

has anyone got a simple little tool for rolling launch template changes through an auto-scaling group

can you use Terraform for that ?

Hello SweetOps people. Is there a way to see within the logs all the permissions required to constrict IAM roles to only the permissions they use ?

If this is all logged, I was hoping to see the past 3 months for a particular ECS task to see what perms it uses, create a new IAM role with only those perms, apply the role to the task, see if anything breaks, if so, add additional perms. If not, move to the next ECS task.

👋 hi folks, while refactoring for 0.12 noticed https://github.com/cloudposse/terraform-aws-ssm-parameter-store doesn't appear ready yet. there are a couple PRs which seem to address the issue. any reason those haven't been merged? is there a better module to be using for this now? TIA!

Hey folks — I posted this over the weekend. It (and the accompanying module) might be of interest to some folks who are running a Bastion host as part of their stack: https://medium.com/masterpoint/time-to-ditch-your-bastion-host-f22cc782de71

Terraform Module: https://github.com/masterpointio/terraform-aws-ssm-agent (Shout out to CP in there for using a few of their modules as well 🙂)

@johncblandii has joined the channel

Guys, for s3 and static website if I remember well if we want to use custom domain we need to name s3 bucket same as domain name?
Is it possible to name it differently, and to serve assets from there only with custom domain, together with cloudfront, and without any webserver?

Hi, we have some huge s3 bucket copies that happen every night between accounts and is run from jenkins as a cron and I was wondering if there is a way to do this in aws where is automated and runs on a schedule

Is anyone aware of a way to track unused IP address' by subnet?

anyone know if there is a public repository of common useful iam policies ?

also how do your companies allow devs access to resources ? we use hologram at my company to give ppl creds. engineers use the engineer role and sres use the sre role. currently iam is whitelist only and usually when engineers hit a wall, they pr into our terraform module, we review, merge, and apply the change. is there a better way to manage creds for engineers ?

hi everyone, have you find any proper way to manage the custom index mapping on AWS Elastic Search ? I think the most easiest way is store all those JSON template under your source control, every time when we need to manipulate something, just pick them up and using CURL. But you know we are all failing in CI/CD world. Therefore I am still finding any proper way to handle those stuffs under proper pipeline which provide us more robust to handle. Am I correct or just over engineering ?

Hi everyone, how could a "normal end user" access files on s3 using SAML authentication?

is there a way within elasticbeanstalk to configure the application load balancer for authentication via cognito user pool? or do i have to configure the ALB outside of EB?

i'm looking for a way but I don't see one on the load balancer config page in elasticbeanstalk

RDS question: I want to spin up an RDS read replica that will generally have pretty low read iops (1,000~2,000). However, during deployment time, I will be running a bunch of concurrent backups against the read replica, causing the read iops to spike to upwards of 10,000 during usually a window of less than an hour. What should I provision my read replica as? I’m thinking gp2 with a high storage size (> 1TB).

What I’m kind of confused about is I currently have a 500GB gp2 RDS instance which spikes to 8,000~10,000 read iops during deployment time but the burst balance is still pretty high post deployment (roughly low 90s). But documentation says the maximum iops will be 3,000. Anyone familiar w/ RDS iops credits can explain?
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html

Hey, I’m migrating one application which has a long running import task triggered by one of our views from beanstalk to fargate. this view when called enables scale in protection for the instance that runs the import task. I went through the logs available at https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cluster-auto-scaling.html but I can’t find anything useful related to fargate. Has anyone solved a similar issue? Also rewriting it in a way that it can be triggered via run-task is currently heavily time constrained and I don’t want to do it.

Have any using aws sso through cli? if so, how the switch b/w accounts happens?

Trying to create S3 bucket with state using terraform, and create dynamodb with some tables. Works fine on one region, got this error trying on another one. What should I do next? Google alot.

Error: error using credentials to get account ID: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.

hi

just a quick one how can i sync s3 subdirectories like that:

aws s3 sync --delete <s3://bucket1/subdir1/*> s3:/bucket2/subdir2/

by removing * 😉

grr took a long time to figure it out

https://github.com/aws/containers-roadmap/issues/487#issuecomment-622135394

AWS actually shipped 1.16

is it just me or are they getting faster?