Docs: [mimir-distributed] Meta-Monitoring seems incomplete

[OlliMartin]

Is your documentation request related to a feature? If so, which one?

I have set up a mimir-distributed (5.9.0-weekly.364) and followed the documentation to set up k8s-monitoring.

The documentation mentions that no ServiceMonitors are needed and that no metaMonitoring settings should be present.

Having k8s-monitoring set up exactly like in the linked docs (With changed destinations - of course), I cannot get any metrics to show up.

Inspecting the alloy-singleton logs, I can see that the metrics are missing job and instance values, the logs read:

1 error occurred:\n\t* job or instance cannot be found from labels\n\n

I am quite new to kubernetes, so I can only speculate that usually prometheus would populate those while scraping mimir, but mimir (in this setup? or alloy?) does not populate them.

In any case, it seems the provided documentation is not covering everything to set up meta-monitoring for mimir.

What is the solution that you would like or the expected outcome?

(Working) Documentation detailing how to set up meta-monitoring for mimir

What did you think would happen?

k8s-monitoring pushing mimir metrics to my destination

What was your environment?

Kubernetes on Azure + ArgoCD

Any additional context to share?

No response

[narqo]

Thank you for the feedback. We should improve the documentation, moving forward.

Meanwhile, these are the steps I used to verify the integration between the Mimir and k8s-monitoring works

1. Install Mimir helm chart (note the mimir-test namespace)

helm upgrade mimir . --install --create-namespace --namespace mimir-test

2. Install alloy-operator CRD, as required by k8s-monitoring's integration

kubectl apply -f https://github.com/grafana/alloy-operator/releases/download/alloy-operator-0.3.11/collectors.grafana.com_alloy.yaml

3. Install k8s-monitoring helm chart

helm repo update grafana
helm install meta-monitoring grafana/k8s-monitoring --values 3-meta-monitoring-values.yaml

I used these simplified set of values to configure the Mimir integration:

3-meta-monitoring-values.yaml

---
cluster:
  name: local-1

destinations:
  - name: mimir
    type: prometheus
    url: http://mimir-gateway.mimir-test.svc:80/api/v1/push
    tenantId: 0 # Optional tenant id

integrations:
  collector: alloy-singleton
  # attach available node labels to all integration metrics collection
  nodeLabels:
    nodepool: true
    region: true
    availability_zone: true
    node_role: true
    os: true
    architecture: true
    instance_type: true

  mimir:
    instances:
      - name: mimir
        namespaces:
          - mimir-test # The namespace to where this Mimir instance was installed in step 1
        labelSelectors:
          app.kubernetes.io/name: mimir
        logs:
          enabled: false
        # Enable metrics collection from the mimir instance
        metrics:
          enabled: true
          scrapeInterval: 30s

clusterMetrics:
  enabled: true

applicationObservability:
  enabled: false

# Collectors
alloy-singleton:
  enabled: true

alloy-metrics:
  enabled: true

alloy-logs:
  enabled: false

alloy-profiles:
  enabled: false

alloy-receiver:
  enabled: false

4. Verify the collection works

Expose the port of the mimir-gateway to query metrics from the Mimir instance,

kubectl -n mimir-test port-forward svc/mimir-gateway 8080:80 --address 0.0.0.0

Add a new Prometheus datasource to Grafana

• URL http://192.168.0.1:8080/prometheus
• X-Scope-OrgID: 0 (optional, ref to 3-meta-monitoring-values.yaml for the tenant id set)

---

The alloy, the k8s-monitoring manages, has a debug UI, that can be handy to debug possible issues with the integration's config. You access the UI by port-forwarding to the svc/meta-monitoring-alloy-singleton:12345. This was how the alloy looked for my test above,

[OlliMartin]

Hi,

thanks a lot of your showcase, really appreciate it.
I think the problem I faced comes from k8s-monitoring.

I am running mimir-distributed deployment mode and I after some digging I found that there seems to be an issue in the alloy config, see here.

Yet, now I wonder why the scraping works for you.
I'm running the latest mimir helm chart (prerelease 5.9.0-weekly.364)

~~> Ohhh.. I mixed up components of the LGTM stack again.. mimir's distributed mode is a separate helm chart and we're not installing the same thing.

I think with k8s-monitoring adjusted (PR is available, I think) this issue is obsolete (sorry!), but I'm happy to confirm once the new k8s-monitoring chart is available, if you want me to. :)