Better Stack Telemetry

OpenSRE uses Better Stack’s ClickHouse SQL Query API to read log evidence during investigations. It queries the configured source via remote(<source>_logs) for recent rows and s3Cluster(primary, <source>_s3) for historical rows, bounded by the alert window.

Prerequisites

A Better Stack account with at least one Telemetry source collecting logs
A ClickHouse HTTP client credential pair (username + password) generated from the dashboard
The region-specific query endpoint for your workspace (e.g. https://eu-nbg-2-connect.betterstackdata.com)
Network access from the OpenSRE environment to that endpoint over HTTPS

Setup

Option 1: Interactive CLI

opensre integrations setup betterstack

You will be prompted for the query endpoint, username, password, and an optional comma-separated list of source IDs (planner hint).

Option 2: Environment variables

Add to your .env:

BETTERSTACK_QUERY_ENDPOINT=https://eu-nbg-2-connect.betterstackdata.com
BETTERSTACK_USERNAME=<clickhouse-http-username>
BETTERSTACK_PASSWORD=<clickhouse-http-password>
BETTERSTACK_SOURCES=t123456_myapp,t123456_gateway

Variable	Default	Description
`BETTERSTACK_QUERY_ENDPOINT`	—	Required. Region-specific SQL API host (e.g. `https://eu-nbg-2-connect.betterstackdata.com`)
`BETTERSTACK_USERNAME`	—	Required. Username from Connect ClickHouse HTTP client
`BETTERSTACK_PASSWORD`	—	Required. Password from the same dashboard flow
`BETTERSTACK_SOURCES`	(empty)	Optional comma-separated list of base source IDs (e.g. `t123456_myapp`). The integration appends `_logs` / `_s3` internally. If omitted, the planner must derive the source from alert metadata (via a `betterstack_source` annotation)

Option 3: Persistent store

Credentials are persisted to ~/.opensre/integrations.json with 0o600 permissions:

{
  "version": 1,
  "integrations": [
    {
      "id": "betterstack-prod",
      "service": "betterstack",
      "status": "active",
      "credentials": {
        "query_endpoint": "https://eu-nbg-2-connect.betterstackdata.com",
        "username": "<clickhouse-http-username>",
        "password": "<clickhouse-http-password>",
        "sources": ["t123456_myapp"]
      }
    }
  ]
}

Generating credentials

From the Better Stack dashboard:

Open Telemetry and pick the source you want OpenSRE to query.
In the source sidebar, open Integrations → Connect ClickHouse HTTP client.
Copy the generated username, password, and query endpoint. The endpoint’s subdomain encodes the region (e.g. eu-nbg-2-connect, us-connect).
The base source ID (e.g. t123456_myapp) is shown above the integration panel and is the value to supply for BETTERSTACK_SOURCES. Use the base name only — OpenSRE appends _logs and _s3 internally.

Investigation tool

OpenSRE exposes one tool against a Better Stack source:

`query_betterstack_logs`

Returns (dt, raw) pairs by UNIONing:

Recent rows from remote(<source>_logs)
Historical rows from s3Cluster(primary, <source>_s3) WHERE _row_type = 1

Arguments the planner supplies:

source — the base identifier (e.g. t123456_myapp). Falls back to the first configured sources entry when omitted.
since / until — ISO-8601 timestamps that bound the dt column. Optional, typically derived from the alert window.
limit — row cap; defaults to 500.

All queries run with FORMAT JSONEachRow and output_format_pretty_row_numbers=0. Source names are validated against ^[A-Za-z0-9_]+$ to prevent identifier injection.

Triggering investigation from an alert

When an alert carries a betterstack_source annotation, the planner wires it through to query_betterstack_logs automatically:

{
  "title": "[betterstack] Agent silent (no info logs in 5min)",
  "state": "alerting",
  "alert_source": "betterstack",
  "commonAnnotations": {
    "summary": "Agent silent (no info logs in 5min): info < 1",
    "betterstack_source": "t123456_myapp"
  }
}

Verify

opensre integrations verify betterstack

Expected output:

SERVICE       SOURCE    STATUS    DETAIL
betterstack   store     passed    Connected to Better Stack SQL API at https://eu-nbg-2-connect.betterstackdata.com

The verify step issues a cheap probe (SELECT 1 FORMAT JSONEachRow) against the configured endpoint using the stored credentials.

Troubleshooting

Symptom	Fix
Authentication failed (401)	Regenerate credentials via the dashboard’s Connect ClickHouse HTTP client flow; confirm `BETTERSTACK_USERNAME` / `BETTERSTACK_PASSWORD` match exactly.
Endpoint not found / DNS error	The region subdomain is wrong. Copy the endpoint directly from the dashboard (e.g. `eu-nbg-2-connect`, `us-connect`, `eu-fsn-3-connect`).
`invalid source name` error on query	The `source` argument contains characters outside `[A-Za-z0-9_]`. Use the base ID shown in the dashboard — no dashes, no quotes, no whitespace.
Empty result set for a known-busy source	Check that the alert window (`since` / `until`) actually overlaps with the source’s `dt` range; historical rows older than recent retention live in `s3Cluster(primary, <source>_s3)`.
`planner did not configure a source` error	Either set `BETTERSTACK_SOURCES`, or ensure the alert payload includes a `betterstack_source` annotation.

Security best practices

Use a dedicated ClickHouse HTTP client credential for OpenSRE — not your personal dashboard login.
Keep the credential pair out of source control — use .env or the persistent store (~/.opensre/integrations.json).
The integration is read-only: OpenSRE only issues SELECT statements against remote(...) and s3Cluster(...) table functions.
Source identifiers are allowlisted against ^[A-Za-z0-9_]+$ before being interpolated into SQL, preventing identifier-injection attacks from the alert payload.
Rotate credentials periodically via the Better Stack dashboard.

Overview

LLM providers

Observability and incidents

Cloud, code, and collaboration

Messaging

Data and workflow systems

Prerequisites

Setup

Option 1: Interactive CLI

Option 2: Environment variables

Option 3: Persistent store

Generating credentials

Investigation tool

`query_betterstack_logs`

Triggering investigation from an alert

Verify

Troubleshooting

Security best practices

​Prerequisites

​Setup

​Option 1: Interactive CLI

​Option 2: Environment variables

​Option 3: Persistent store

​Generating credentials

​Investigation tool

​query_betterstack_logs

​Triggering investigation from an alert

​Verify

​Troubleshooting

​Security best practices

Prerequisites

Setup

Option 1: Interactive CLI

Option 2: Environment variables

Option 3: Persistent store

Generating credentials

Investigation tool

`query_betterstack_logs`

Triggering investigation from an alert

Verify

Troubleshooting

Security best practices