#sre - April 2020

Prometheus + EFS users, so nfs isn’t considered a supported storage for Prometheus i guess? Have you guys have any problems with data corruption/data loss?

Dirty shutdowns will leave wall files around

Also, I imagine if you had two Prometheus operators writing to the same exact file system you would have corruption

Oof. #2 could happen when I do cluster cutovers 🤔

When I spin up a new cluster. There’s a short period of time that new cluster and the old cluster both have prom-operator talking to the same EFS

Yup, that would be my guess. That’s going to lead to corruption.

Learned about this today in the kubernetes office hours

Anyone know of any projects out there that would support longer-term prometheus metrics storage for small deployments?

Thanos is much to complex for us. Timescaledb seems promising, but cannot be used with RDS.

We don't need HA.

Historical rollups like datadog would be a huge plus.

EFS works like a charm

What’s EFS? Google only turns up ways to monitor AWS’s EFS.

We're ready to move from a homegrown alert system to a more "proper" service like pagerduty/victorops/opsgenie. Does this group have any strong feelings one way or another about one of these (or another) services? Team of 4-8 engineers geo-distributed. We use prometheus/alertmanager and cloudwatch.

I find it kind of silly that in order to do end-to-end tests of the alerting system you have to add another SASS like Dead Man's Snitch.

What are people using to monitor the apps on Fargate? APM solution like NewRelic/DataDog?

Pro tip: subscribe to the RSS feed for status pages you depend on. Send those updates to a slack channel. https://slack.com/help/articles/218688467-Add-RSS-feeds-to-Slack
e.g.
• https://www.githubstatus.com/history.rss
• https://status.aws.amazon.com/rss/ec2-us-east-1.rss
• https://status.pagerduty.com/history.rss
• https://status.cloud.google.com/feed.atom

anyone here using http://timber.io/

i am not so sure, but it looks a dead product

documentation outdate . some broken links also

but what they offer seem not bad.