Grafana

​

Overview

​

Prerequisites

• Grafana instance URL (Cloud stack URL or self-hosted origin)
• Service account token with read access — see Credentials

​

Setup

​

Option 1: Interactive CLI

opensre integrations setup grafana

​

Option 2: Environment variables

GRAFANA_INSTANCE_URL=https://your-stack.grafana.net
GRAFANA_READ_TOKEN=glsa_your_service_account_token
GRAFANA_VERIFY_SSL=true                    # optional — set false only for local/lab
GRAFANA_CA_BUNDLE=/path/to/internal-ca.pem # optional — internal CA

​

Option 3: Persistent store

{
  "version": 1,
  "integrations": [
    {
      "id": "grafana-prod",
      "service": "grafana",
      "status": "active",
      "credentials": {
        "endpoint": "https://your-stack.grafana.net",
        "api_key": "glsa_your_token",
        "verify_ssl": true
      }
    }
  ]
}

​

Option 4: Hosted web app (OpenSRE Cloud)

1. In app.tracer.cloud, go to Integrations → Grafana
2. Enter a name, instance URL, and service account token
3. Click Save

​

Self-signed or internal CA certificates

​

Credentials

1. In Grafana, open Administration → Service accounts (or your stack’s equivalent).
2. Create a service account with read access to the datasources you want OpenSRE to query.
3. Add a token to that service account and copy it (shown once).

​

Investigation tools

​

Verify

opensre integrations verify grafana

​

Troubleshooting

​

Security

• Prefer a dedicated service account token with read-only access for investigations.
• Prefer GRAFANA_CA_BUNDLE over disabling TLS verification for real internal Grafana.
• Set GRAFANA_VERIFY_SSL=false only for local/lab instances.
• Store tokens in .env or your secret manager — not in source control.

​

Extras

​

Local Grafana setup (Minikube example)

​

Steps

1. Start Minikube:
   minikube start
   
2. Add Helm repositories and update:
   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo add podinfo https://stefanprodan.github.io/podinfo
   helm repo update
   
3. Install the kube-prometheus stack:
   kubectl create namespace monitoring
   helm install kube-stack prometheus-community/kube-prometheus-stack -n monitoring
   
4. Install the podinfo sample app:
   kubectl create namespace podinfo
   helm install podinfo podinfo/podinfo -n podinfo --set serviceMonitor.enabled=true
   
5. (Optional) Check pods:
   kubectl -n podinfo get pods
   kubectl -n monitoring get pods
   
6. Port-forward podinfo (separate terminal):
   kubectl -n podinfo port-forward deploy/podinfo 8080:9898
   
7. Port-forward Prometheus (separate terminal):
   kubectl -n monitoring port-forward svc/kube-stack-kube-prometheus-prometheus 9090:9090
   
8. Port-forward Grafana on all interfaces (separate terminal):
   kubectl -n monitoring port-forward --address 0.0.0.0 svc/kube-stack-grafana 3000:80
   
9. Allow Prometheus to scrape podinfo ServiceMonitors:
   kubectl patch prometheus kube-stack-kube-prometheus-prometheus \
   -n monitoring \
   --type=merge \
   -p '{"spec":{"serviceMonitorSelector":{},"serviceMonitorNamespaceSelector":{}}}'
   

​

Grafana credentials

kubectl get secret kube-stack-grafana --namespace monitoring -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

​

Access

​

Simulate load

while true; do curl -s http://localhost:8080/status/500 >/dev/null; sleep 1; done &
while true; do curl -s http://localhost:8080/delay/5 >/dev/null; sleep 1; done &

​

Sample Grafana queries

sum(rate(http_requests_total{job="podinfo"}[1m])) by (status)

histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{job="podinfo"}[1m])) by (le))

​

Prometheus alert for high error rate

kubectl apply -f - <<'YAML'
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: podinfo-alerts
  namespace: monitoring
  labels:
    release: kube-stack
spec:
  groups:
  - name: podinfo.rules
    rules:
    - alert: PodinfoHighErrorRate
      expr: increase(http_requests_total{status="500"}[5m]) > 5
      for: 30s
      labels:
        severity: critical
      annotations:
        summary: "Podinfo error rate is high"
YAML

​

Connect OpenSRE to the lab Grafana

1. Get your machine’s LAN IP:
   hostname -I | awk '{print $1}' # Linux
   ipconfig getifaddr en0 # macOS
   (Get-NetIPAddress -InterfaceAlias "Wi-Fi" -AddressFamily IPv4).IPAddress # Windows PowerShell
   # Replace "Wi-Fi" with your interface name if needed.
   
2. Create a Grafana service account token (Grafana docs).
3. Run setup and enter:
   opensre integrations setup grafana
   

   Prompt	Value
   Instance URL	http://<LAN_IP_ADDRESS>:3000
   Service account token	Token from step 2

   For local/lab TLS issues, set GRAFANA_VERIFY_SSL=false or answer the SSL prompts as described in Setup.