#kubecost - August 2019

kubecostArchived
45 messages

Kubernetes resource and cost management
Archive: https://archive.sweetops.com/kubecost/

August 2019
Jan
Janabout 7 years ago
any one ever run into the cost-analyzer-frontend container spitting the following error ?

[emerg] host not found in upstream "cost-analyzer-grafana.default.svc.cluster.local" in /etc/nginx/conf.d/default.conf:11
Jan
Janabout 7 years ago
ah oki found teh issue
Jan
Janabout 7 years ago
I hand global.grafana.enabled: false yet was passing a custom domain
Jan
Janabout 7 years ago
mmm no thats not it
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)about 7 years ago
do you have a service called cost-analyzer-grafana in the default namespace?
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)about 7 years ago
@Jan are you still using helmfile?
Jan
Janabout 7 years ago
Will need to check
Jan
Janabout 7 years ago
Tomorrow
Jan
Janabout 7 years ago
We don't use helm file at all
Jeremy G
Jeremy Gabout 7 years ago
@Jeremy G has joined the channel
Jan
Janabout 7 years ago
So despite setting the grafana.domainName
Jan
Janabout 7 years ago
the nginx-conf config map still uses

upstream grafana {
        server cost-analyzer-grafana.default.svc.cluster.local;
Jan
Janabout 7 years ago
how do I change these settings from the chart?
Jeremy G (Cloud Posse)
Jeremy G (Cloud Posse)about 7 years ago
The settings are under global. You need to set

global:
  grafana:
    enabled: false
    domainName: <external domain name of your Grafana>
    scheme: <http or https for accessing your Grafana>

https://github.com/cloudposse/helmfiles/blob/103b1c3b5ab68b568307017f159f07431c07d8b3/releases/kubecost.yaml#L45-L48
Erik Osterman (Cloud Posse)
Erik Osterman (Cloud Posse)about 7 years ago
@Jan ☝️
Jan
Janabout 7 years ago
So I have tried those, at least in our case, I can't seem to populate those parameters from passing arguments to the helm chart
Jan
Janabout 7 years ago
We deploy the chart passing a values.yaml file and additional override specific things (grafana address etc)
Jan
Janabout 7 years ago
By setting specific parameters as arguments
Jan
Janabout 7 years ago
Even after doing so if I edit the Nginx-conf config map I see the upstream grafana value isn't set
Webb
Webbabout 7 years ago
@Jan, any chance you can share your values.yaml file? I can try to reproduce.
Jan
Janabout 7 years ago
I can do in a few hours
Jan
Janabout 7 years ago
When my kids are asleep
Jan
Janabout 7 years ago
Will need to show you how we are installing the chart
Jan
Janabout 7 years ago
so we are installing the chart via terraform and setting two parameters while installing as well as providing two values files
Jan
Janabout 7 years ago(edited)

data "helm_repository" "kubecost" {
  url  = "<https://kubecost.github.io/cost-analyzer>"
  name = "kubecost"
}

resource "helm_release" "kubecost" {
  name       = "kubecost"
  namespace  = "monitoring"
  chart      = "kubecost/cost-analyzer"
  repository = "${data.helm_repository.kubecost.name}"

  set {
    name  = "ingress.hosts[0]"
    value = "kubecost.${local.cluster_domain}"
  }

  set {
    name  = "grafana.domainName[0]"
    value = "grafana.${local.cluster_domain}"
  }

  values = [
    "${file("defaults.yaml")}",
    "${file("stage.yaml")}",
  ]
}
Jan
Janabout 7 years ago
so ingress.hosts works 100% as expected
Jan
Janabout 7 years ago
grafana.domainName doesn not as I would expect
Jan
Janabout 7 years ago
defaults.yaml:

global:
  prometheus:
    enabled: false
    fqdn: <http://promop-prometheus-operator-prometheus.monitoring.svc.cluster.local:9090>

  grafana:
    enabled: false
    scheme: "http"

  notifications:
    alertmanager:
      fqdn: <http://promop-prometheus-operator-alertmanager.monitoring.svc.cluster.local:9093>

# <https://kubecost.com/install>
# This token is not really secret and can be exchanged at any time
kubecostToken: xxxxxx
serviceMonitor:
  enabled: true

kubecost:
  limits:
    cpu: 300m
    memory: 128Mi

kubecostFrontend:
  limits:
    cpu: 30m
    memory: 56Mi

kubecostModel:
  limits:
    cpu: 400m
    memory: 128Mi

ingress:
  enabled: true
  annotations:
    <http://kubernetes.io/ingress.class|kubernetes.io/ingress.class>: traefik

grafana:
  sidecar:
    dashboards:
      enabled: true
    datasources:
      enabled: true
Jan
Janabout 7 years ago
stage.yaml is only used to enable slack notifications for specific accounts
Jan
Janabout 7 years ago

global:
  notifications:
    slack:
      enabled: true
      webhook: <https://hooks.slack.com/services/xxxxxx>
Jan
Janabout 7 years ago
that does almost not seem to be working currently
Jan
Janabout 7 years ago
well it doesn't get set
Jan
Janabout 7 years ago
works if i set it via the kubecost site after
Webb
Webbabout 7 years ago
Thanks for sharing! You’re meaning to be using a ServiceMonitor correct? That looks like the only part I haven’t tested recently. @Jeremy G actually has a similar config I believe.
Jan
Janabout 7 years ago
so everything else is working fine
Jan
Janabout 7 years ago
I just dont seem to be able to preset those parameters (slack and grafana address)
Jan
Janabout 7 years ago
that said I have not yet pulled the chart apart
Jeremy G (Cloud Posse)
Jeremy G (Cloud Posse)about 7 years ago(edited)
@Jan As I explained before, the setting you need is global.grafana.domainName but you are only setting grafana.domainName
Jan
Janabout 7 years ago
which I should probably just do
Jan
Janabout 7 years ago
any idea on the slack notification where im going wrong?
Jeremy G (Cloud Posse)
Jeremy G (Cloud Posse)about 7 years ago
The slack notification is a feature of AlertManager. If you are using your own Prometheus then you are also using your own AlertManager and have to configure Slack there.
Jan
Janabout 7 years ago
ah I see
Jan
Janabout 7 years ago
cheers mate
Jan
Janabout 7 years ago
much appreciated!