#sre - December 2019

Hey @Erik Osterman (Cloud Posse) here is a slightly biased, coming from the core developer 😉 of Victoria Metrics, comparison with Thanos.

“Comparing Thanos to VictoriaMetrics cluster” by Aliaksandr Valialkin https://link.medium.com/ltvYvhvrj2

This is great! Thanks for sharing @Steve Boardwell!

IMO this is the downside. Why be in the business of managing EBS disks.

I like the simpler architecture of VictoriaMetrics, but the big appeal for me regarding Thanos was the S3 backend. Basically, "set it and forget it"

yeh S3 for storage is a much lower ops burden and much more straight forward to lifecycle - the last thing i want my ops tooling to do is unduly increase the burden of operations

i'd be interested to see cortex in that comparison

Great points - I hadn't considered the overhead involved in managing the EBS disks. Cortex looks interesting as well, thanks Chris.

Anyone aware of a tool that can take kubernetes action based on prometheus alerts/metrics?

I'd like to kill a pod when it stops processing, which is tracked by a prometheus metric

Depending on what you want to achieve this can do that: https://github.com/weaveworks/flagger

Or use something like https://github.com/stefanprodan/k8s-prom-hpa to scale replicas to zero

Thank you Erik. I don't think HPA autoscaler will help, because I don't want to scale to zero, I just want the pods replaced/restarted. I'll take a look at flagger, but I'm not sure if I can use it to kill/restart the pod

@Nelson Jeppesen it sounds like it maybe be better describe the problem you’re trying to solve than the technical implementation of a solution

Ok thats a very good point. I have some logstash pods. Between 2 and 20 days, the pod will stop processing logs (writing to elasticsearch). We alert on this and kill the pods manualy. As of yet, we've not be able to figure out why this happens as we see no errors

For example, “we have a problem with our application where the services stop responding processing customer orders. The health checks are all okay, but if we look at prometheus we see it is no longer doing anything. If we restart the service, everything is fine. So I want to implement something that restarts our service based on metrics in Prometheus”