#geodesic - November 2019 | Slack Archive

geodesic

135 messages

Discussions related to https://github.com/cloudposse/geodesic

Archive: https://archive.sweetops.com/geodesic/

Oct 2019

November 2019

Dec 2019

SweetOps #geodesicalmost 7 years ago

There are no events this week

Cloud Possealmost 7 years ago

Join us for "Office Hours" every Wednesday 11:30AM (PST, GMT-7) via Zoom.

This is an opportunity to ask us questions about INLINECODE_0, get live demos and learn from others using it. Next one is Nov 13, 2019 11:30AM.

👉️ Register for Webinar

#office-hours (our channel)

Joe Nilandalmost 7 years ago

When using the reference-architectures project, would it be possible to adopt an existing account?

Erik Osterman (Cloud Posse)almost 7 years ago

@Joe Niland technically yes, but we don't optimize for that since it's not how we (as a company) work. There are too many variables to consider.

Erik Osterman (Cloud Posse)almost 7 years ago

It's best to look at what geodesic represents: a strategy for bundling and shipping the tools and configuration for an environment

Erik Osterman (Cloud Posse)almost 7 years ago

we happen to have our flavor of how to do that, but nothing restricts how it is used

Joe Nilandalmost 7 years ago

@Erik Osterman (Cloud Posse) thanks. After I wrote that, I had a look through the account module and realised it's not so easy.

Asking because I have a new client who have been working for a while in a single account and have AWS credits tied to it.

I'm working on creating the other accounts using your process and then will create a geodesic repo manually for the existing account. That's the plan anyway!

Erik Osterman (Cloud Posse)almost 7 years ago

Yes, I think manually creating the geodesic repos in this case is the way to go

Erik Osterman (Cloud Posse)almost 7 years ago

https://calendly.com/cloudposse if you want some quick pointers

Erik Osterman (Cloud Posse)almost 7 years ago

https://github.com/cloudposse/packages

Erik Osterman (Cloud Posse)almost 7 years ago

packages are now updated nightly using GitHub actions!

Erik Osterman (Cloud Posse)almost 7 years ago

See: https://github.com/cloudposse/packages/pull/265

Erik Osterman (Cloud Posse)almost 7 years ago

@Andriy Knysh (Cloud Posse) can you test out @tamsky change: https://github.com/cloudposse/geodesic/pull/534

Andriy Knysh (Cloud Posse)almost 7 years ago

Will test

SweetOps #geodesicalmost 7 years ago

There are no events this week

Cloud Possealmost 7 years ago

👉️ Register for Webinar

#office-hours (our channel)

Blaise Pabonalmost 7 years ago

Sooo... the docs don't actually describe how to get started with geodesic...
I guessed that git clone <https://github.com/cloudposse/geodesic.git> && cd geodesic && geodesic would get me launch the shell.
Next, it asked me for my passphrase (which I think I remember)

Blaise Pabonalmost 7 years ago(edited)

I'll tell you what, if I agree to be initiated me into the sacred brotherhood of geodesic and I swear that I will not reveal the sacred mysteries to anyone; will you show me the answer to the riddle:


-> Run 'assume-role' to login to AWS
 ⧉  geodesic
 ✗   (none) ~ ⨠

Jeremy G (Cloud Posse)almost 7 years ago

@Blaise Pabon Geodesic makes a few assumptions that are not really documented AFAIK but flow from the way things are set up by the CloudPosse Reference Architecture: https://github.com/cloudposse/reference-architectures

Jeremy G (Cloud Posse)almost 7 years ago

Geodesic assumes that you are using AWS, and that you will get AWS credentials dynamically injected into your environment using some tool that allows you to assume an AWS IAM role. Typically that is aws-vault, but it can be aws-okta or anything. All that has to happen is that once you have your AWS credentials set up in your environment, you need to set the environment variable ASSUME_ROLE to the role name, after which the "Run assume-role" prompt will go away and the role name specified by ASSUME_ROLE will become part of the prompt.

Blaise Pabonalmost 7 years ago

Oh! OK, I have been setting my AWS creds in $AWS_ACCESS_KEY etc. I will look over the reference arch setup more closely. Thanks for the tips. FWIW, I'm a fan of keeping a higher barrier to entry.... it keeps the day-trippers and riff-raff away.
It's not that Unix isn't user-friendly, it's just picky about who its friends are

Joe Nilandalmost 7 years ago(edited)

The module directories created by reference-architectures are all for Terraform 0.11. I’m creating my own for my app but I want to use 0.12.

I saw that @Andriy Knysh (Cloud Posse) recommended to change the Makefile like this:


-include ${TF_MODULE_CACHE}/Makefile

And to add export TF_MODULE_CACHE=.module to terraform.envrc

But I want to create ${TF_MODULE_CACHE} if it’s not there.

So far I have this:


$(shell mkdir -p ${TF_MODULE_CACHE})
-include ${TF_MODULE_CACHE}/Makefile

## Fetch the remote terraform module
deps:
	terraform init

## Reset this project
reset:
	rm -rf Makefile *.tf .terraform

Is there a better way?

chrismalmost 7 years ago

How do you run "make lint" without it running terraform 0.11?

chrismalmost 7 years ago

tried passing in env var but its got a fetish for 0.11

chrismalmost 7 years ago

in relation to https://github.com/cloudposse/terraform-aws-ec2-autoscale-group/pull/17

chrismalmost 7 years ago

meh doesnt do much anyway as it falls over using 12 at the validate step

chrismalmost 7 years ago

is gomplate supposed to be in that mass of stuff in the build harness?
/bin/bash: gomplate: command not found
I've installed it and generated the file; still no idea why the build harness kept defaulting to v11 in the project but I broke my way passed it so 🤷

joshmyersalmost 7 years ago

AFAICR gomplate isn’t installed inside geodesic, so you need that on your host

chrismalmost 7 years ago

Its a tf repo; so its more the "build tools" bit; i suppose its awkward to get the right binary unless you presume (like the terraform side) its all linux

chrismalmost 7 years ago

I use go get ... in the end and ran it. instructions were unclear so I winged it 😆

Jeremy G (Cloud Posse)almost 7 years ago

In general, you run make init to load the build harness, but then you still may need to run make readme/deps to get gomplate so you can run make readme.

SweetOps #geodesicalmost 7 years ago

There are no events this week

Cloud Possealmost 7 years ago

👉️ Register for Webinar

#office-hours (our channel)

chrismalmost 7 years ago

any quick tools to generate a "provider file" with the versions it recommends pinning

chrismalmost 7 years ago

pre-empting https://www.hashicorp.com/blog/deprecating-terraform-0-11-support-in-terraform-providers/

Joe Nilandalmost 7 years ago

I’ve used the reference-architectures to set up the base repos for each account. I’m now adding specific modules to build app infrastructure. I have a couple of general questions:

1. Once everything is created, should I remove the <namespace>-root-bootstrap user and role?
2. Say I create a module at dev.example.net/conf/network/ which creates VPC & subnets, then I have another at dev.example.net/conf/data, would you create a main.tf in the conf/ dir which references the above and can be used to create all the modules with one terraform apply? This would mainly be for convenience, but also to make sure you deployed every module in the repo. I’m used to Salt states, where you often apply many states with a single command.
3. I noticed that the <mailto:user@example.com|user@example.com> users created by Cold Start can assume role to arn:aws:iam::XXXXX:role/example-root-admin which has the AdministratorAccess policy but this policy is not directly attached to the example-root-admin group. I know we try not to use the Console often but sometimes it is required/useful. Also clients may prefer it. It’s an extra step to assume role in the console so just wondering if it would be bad to assign the AdministratorAccess policy directly the to example-root-admin group, since the net effect is the same.

Erik Osterman (Cloud Posse)almost 7 years ago

(btw, if you break your message into multiple msgs, easier to respond)

Erik Osterman (Cloud Posse)almost 7 years ago

Once everything is created, should I remove the <namespace>-root-bootstrap user and role?

Erik Osterman (Cloud Posse)almost 7 years ago

it's only needed in the very, very beginning

Erik Osterman (Cloud Posse)almost 7 years ago

Say I create a module at dev.example.net/conf/network/ which creates VPC & subnets, then I have another at dev.example.net/conf/data, would you create a main.tf in the conf/ dir which references the above and can be used to create all the modules with one terraform apply? This would mainly be for convenience, but also to make sure you deployed every module in the repo. I’m used to Salt states, where you often apply many states with a single command.

This is where a task runner comes in

Erik Osterman (Cloud Posse)almost 7 years ago

using something like make or terragrunt

Erik Osterman (Cloud Posse)almost 7 years ago

make has gotten a lot of attention lately (and for good reason)

Erik Osterman (Cloud Posse)almost 7 years ago

but one thing that's often neglected (and not obvious to newcomers), is that targets are first and foremost files

Erik Osterman (Cloud Posse)almost 7 years ago

e.g.

Erik Osterman (Cloud Posse)almost 7 years ago(edited)

file.txt: foo.txt
  cp -a foo.txt $@

Erik Osterman (Cloud Posse)almost 7 years ago

my point is that if file.txt is older than foo.txt, then it perfoms the action

Erik Osterman (Cloud Posse)almost 7 years ago

this is interesting if you think about it for terraform

Erik Osterman (Cloud Posse)almost 7 years ago

since using modification times, it can be derived if things have changed and need updating

Erik Osterman (Cloud Posse)almost 7 years ago

anyways, see #terragrunt for the apply-all type stuff

Erik Osterman (Cloud Posse)almost 7 years ago

re: 3, we prefer to always require assuming a role

Erik Osterman (Cloud Posse)almost 7 years ago

so things are deliberate

Joe Nilandalmost 7 years ago

Thanks very much @Erik Osterman (Cloud Posse). Sorry about the multi-part message.

Joe Nilandalmost 7 years ago

I've been getting to know make since finding your stuff, so I'll trying creating a Makefile for each account repo and see how it goes.

Andrew Rothalmost 7 years ago

Made a project to play around with CodeFresh, and get a jump on some future work. I'd love some feedback. It's a philosophical cousin to Geodesic, with slightly different goals.

https://github.com/dadsgarage/dadsgarage

Erik Osterman (Cloud Posse)almost 7 years ago

lol!

Erik Osterman (Cloud Posse)almost 7 years ago

love the name. so perfectly fitting.

Erik Osterman (Cloud Posse)almost 7 years ago

@Andrew Roth will take a look tomorrow

Joe Nilandalmost 7 years ago

Is there a recommended way to add the S3 state backend to the module structure generated by reference-architectures ? I had a look at https://docs.cloudposse.com/geodesic/module/with-terraform/ but it seems to be describing the process that is already automated by Cold Start

Joe Nilandalmost 7 years ago

I can see all the required env vars seem to have been created, e.g.

TF_CLI_ARGS_init=-backend-config=region=ap-southeast-2 -backend-config=key=network/terraform.tfstate -backend-config=bucket=example-dev-terraform-state -backend-config=dynamodb_table=example-dev-terraform-state-lock -backend-config=encrypt=true -from-module=git::<ssh://git>@bitbucket.org/team-example/infra.modules.git//modules/network?ref=master .module
TF_CLI_INIT_BACKEND_CONFIG_DYNAMODB_TABLE=example-dev-terraform-state-lock
TF_CLI_INIT_BACKEND_CONFIG_BUCKET=example-dev-terraform-state
TF_CLI_INIT_BACKEND_CONFIG_KEY=network/terraform.tfstate
... etc

However when I run make which runs terraform init Terraform prompts for the bucket name.

Just wondering why it’s not picking up the values from the TF_CLI vars

Joe Nilandalmost 7 years ago(edited)

I should also add, my module is initialised from git, is using v0.12, and I added backend.tf containing the following to the module directory.

terraform {
    backend "s3" {}
}

Erik Osterman (Cloud Posse)almost 7 years ago

@Joe Niland hit me up tomorrow and can take a look

Erik Osterman (Cloud Posse)almost 7 years ago

Make sure you have “use tfenv” in your envrc

SweetOps #geodesicalmost 7 years ago

There are no events this week

Cloud Possealmost 7 years ago

👉️ Register for Webinar

#office-hours (our channel)

Joe Nilandalmost 7 years ago

I saw that you guys have the following in your Makefile to use local state if it’s there otherwise remote.

init:
	[ -f .terraform/terraform.tfstate ] || terraform $@

Joe Nilandalmost 7 years ago

So that takes care of local development, right?

Erik Osterman (Cloud Posse)almost 7 years ago

No, just an optimization to only init once

Joe Nilandalmost 7 years ago

Ah yeah. I totally misread that. Too much code today.

Joe Hostenyalmost 7 years ago

How are folks handling getting a unique TF_BUCKET during bootstrap? I may be missing something, but it looks like the chance for collision on the names is fairly high.

Erik Osterman (Cloud Posse)almost 7 years ago

that's we we've stuck to the namespace-stage-name convention

Erik Osterman (Cloud Posse)almost 7 years ago

which is enforced by labels

Erik Osterman (Cloud Posse)almost 7 years ago

We use this module: https://github.com/cloudposse/terraform-aws-tfstate-backend

Erik Osterman (Cloud Posse)almost 7 years ago

to provision the statebucket (once per AWS account)

Erik Osterman (Cloud Posse)almost 7 years ago

and then use folders for each project in that account

Joe Hostenyalmost 7 years ago

Thanks @Erik Osterman (Cloud Posse). Does it make sense to include the account id in the name? I picked a fairly common namespace, and make children failed halfway through on just one of the accounts.

Erik Osterman (Cloud Posse)almost 7 years ago

aha, you could, but we don't provide a way to parameterize that.

Erik Osterman (Cloud Posse)almost 7 years ago

the main reason we avoided that is account id's have little meaning to humans

Erik Osterman (Cloud Posse)almost 7 years ago

so reviewing terraform plans that contain them doesn't really help code review

Erik Osterman (Cloud Posse)almost 7 years ago

it's certainly a common pattern

Erik Osterman (Cloud Posse)almost 7 years ago

and i would probably recommend the pattern whereby you're generating a slew of accounts (E.g. for developers or apps)

Erik Osterman (Cloud Posse)almost 7 years ago

but stick to named accounts for your "core" accounts

Erik Osterman (Cloud Posse)almost 7 years ago

we tend to use acronyms for namespace

Erik Osterman (Cloud Posse)almost 7 years ago

for example "sweetops cloud infrastructure", we'd use soci

Erik Osterman (Cloud Posse)almost 7 years ago

so it's "optimistically" unique given all the other dimensions

Joe Hostenyalmost 7 years ago(edited)

Right, I was thinking of just having the account id as a uniquifier in the state file bucket name, no where else

Joe Hostenyalmost 7 years ago

Is there any guidance on how to unwind a partially bootstrapped set of accounts?

Erik Osterman (Cloud Posse)almost 7 years ago

that'sthe thing though

Erik Osterman (Cloud Posse)almost 7 years ago

we use N buckets

Erik Osterman (Cloud Posse)almost 7 years ago

one per aws account (to share nothing)

Joe Hostenyalmost 7 years ago

Right, so you would have soci-dev-<dev acct id>-terraform-state and soci-prod-<prod acct id>-terraform-state, for example

Erik Osterman (Cloud Posse)almost 7 years ago

you can give it a shot! 😃 haven't tried it, so not sure what sort of edge cases might come up

Erik Osterman (Cloud Posse)almost 7 years ago

i like that it provides a nice way to disambiguate multiple prod accounts

Joe Hostenyalmost 7 years ago

For a fresh bootstrap, will the label get the attributes from the TF_BUCKET envvar?

Joe Hostenyalmost 7 years ago

That is set in the template Dockerfile? If so, it is an easy change I think

Erik Osterman (Cloud Posse)almost 7 years ago

yes/no, yes it will use it for projects

Erik Osterman (Cloud Posse)almost 7 years ago(edited)

but no it won't use it for creation

Erik Osterman (Cloud Posse)almost 7 years ago(edited)

b/c the terraform-aws-tfstate-backend module is opinionated and always uses the terraform-null-label format

Joe Hostenyalmost 7 years ago

Yup, I think I need to read a bit more to find out how that is invoked. It looks like it accepts the attributes var on the label module, so perhaps that can be used

Erik Osterman (Cloud Posse)almost 7 years ago

I would accept a PR to the tfstate backend module that added support for macros

Erik Osterman (Cloud Posse)almost 7 years ago

Of which account id could be one

Erik Osterman (Cloud Posse)almost 7 years ago

So you could add %account_id%

Erik Osterman (Cloud Posse)almost 7 years ago

Or maybe we should add this to the null label

Erik Osterman (Cloud Posse)almost 7 years ago

That way we can use it else where

Erik Osterman (Cloud Posse)almost 7 years ago

@Andriy Knysh (Cloud Posse) what do you think? ☝️

Andriy Knysh (Cloud Posse)almost 7 years ago

you can still use namespace-stage-name-attributes format

Andriy Knysh (Cloud Posse)almost 7 years ago

you can use account ID as name

Andriy Knysh (Cloud Posse)almost 7 years ago

or add acount ID to the attributes

Andriy Knysh (Cloud Posse)almost 7 years ago

otherwise, I don’t see a use case for that. Do you want to use a few of those buckets in one account and for the same environment (stage)?

Andriy Knysh (Cloud Posse)almost 7 years ago

as we do it, we distinguish environments by stage name

Andriy Knysh (Cloud Posse)almost 7 years ago

they can be deployed into one account or into separate accounts

Andriy Knysh (Cloud Posse)almost 7 years ago

but still the resource names will be unique

Andriy Knysh (Cloud Posse)almost 7 years ago(edited)

if you happen to deploy the same set of resources into the same namespace and stage with the same names, you can add additional attributes e.g. namespace-stage-name-blue namespace-stage-name-green``

Joe Hostenyalmost 7 years ago

This is just for the state bucket. I am hitting a global collision with someone else's bucket on the name, since I chose devops as the name space (so the bucket it attempts to create is devops-prod-terraform-state).

Joe Hostenyalmost 7 years ago

I think I can accomplish this with a two line change to the reference architectures repo. Thanks for the help, and the great modules!

Andriy Knysh (Cloud Posse)almost 7 years ago

why not just change the namespace?

Andriy Knysh (Cloud Posse)almost 7 years ago

It should be unique

Andriy Knysh (Cloud Posse)almost 7 years ago

devops is not

Andriy Knysh (Cloud Posse)almost 7 years ago

Namespace should be your company name or even better company abbreviation

Andriy Knysh (Cloud Posse)almost 7 years ago

Which decreases the possibility of collision

Joe Hostenyalmost 7 years ago

Yes, we could do that. I was avoiding it as our company name is fairly long (and the abbreviation is not uncommon).

Andriy Knysh (Cloud Posse)almost 7 years ago

You should try to keep it as short as possible because some AWS resources have limits on names and IDs length

Andriy Knysh (Cloud Posse)almost 7 years ago

But nothing prevents you from using namespace like ctcr2 or similar

Joe Hostenyalmost 7 years ago

Yup - I have used terragrunt on other work projects, along with terraform-null-label. Typically, I use a namespace of “crl”, so the names are reasonable in length. I’ve just been using the account ID (only) in the tf state bucket name to avoid collisions. It’s not a dealbreaker by any means, it was just a bit unfortunate since it ran into the collision only after the 4th account was being created.

Erik Osterman (Cloud Posse)almost 7 years ago

yea, if it's any consolation, "you are not alone"

Erik Osterman (Cloud Posse)almost 7 years ago

😃

Erik Osterman (Cloud Posse)almost 7 years ago(edited)

I have run into it as well using the even namespace

Erik Osterman (Cloud Posse)almost 7 years ago(edited)

I was blown away that there would be a conflict

Stéphane Bernieralmost 7 years ago

Hi, I wanted to build Geodesic today, but I'm facing issue with helm v3 support. make docker/build is complaining that --client-only is an unknown flag.
I was wondering if there is a way to specify the helm version 2 from the lis of packages to download from cloudposs ?. Or do I need to download a specific cloudposs version ? thanks

Erik Osterman (Cloud Posse)almost 7 years ago

Hrmmmm so in your Dockerfile you would put something like RUN apk add helm==2.15.2

Erik Osterman (Cloud Posse)almost 7 years ago

I'm not sure what our latest 2.x release of helm is though

Erik Osterman (Cloud Posse)almost 7 years ago

you can maybe run apk info helm to see available versions

Erik Osterman (Cloud Posse)almost 7 years ago

oh, this might work too:

Erik Osterman (Cloud Posse)almost 7 years ago(edited)

apk add 'helm<3.0.0'

Erik Osterman (Cloud Posse)almost 7 years ago

via https://superuser.com/a/1058665

October 2019 Browse by date December 2019