89 lines
2.0 KiB
Markdown
Executable File
89 lines
2.0 KiB
Markdown
Executable File
# Setting Up Monitoring
|
|
|
|
This guide covers the monitoring and observability surface that ships with the
|
|
repository.
|
|
|
|
## 1. Start the observability profile
|
|
|
|
```bash
|
|
docker compose --profile observability up -d
|
|
```
|
|
|
|
This brings up:
|
|
|
|
- Prometheus
|
|
- Grafana
|
|
- OTEL Collector
|
|
- MinIO
|
|
- Tempo distributor, ingester, querier, query frontend, and compactor
|
|
|
|
## 2. Point the gateway at the collector
|
|
|
|
Set:
|
|
|
|
```dotenv
|
|
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
|
```
|
|
|
|
Optional tuning:
|
|
|
|
```dotenv
|
|
OTEL_FLUSH_INTERVAL_MS=5000
|
|
FROSTY_OTEL_MODEL_CARDINALITY_CAP=11
|
|
```
|
|
|
|
## 3. Start or restart the gateway
|
|
|
|
```bash
|
|
docker compose up -d gateway
|
|
```
|
|
|
|
## 4. Verify the surfaces
|
|
|
|
- Gateway health: `http://localhost:8080/healthz`
|
|
- Gateway metrics: `http://localhost:8080/metrics`
|
|
- Grafana: `http://localhost:3000`
|
|
- Prometheus: `http://localhost:9090`
|
|
- Tempo query frontend: `http://localhost:3200`
|
|
- MinIO console: `http://localhost:9001`
|
|
|
|
## 5. Understand what is captured
|
|
|
|
### Metrics
|
|
|
|
The gateway exposes Prometheus metrics from `/metrics`, including request,
|
|
cache, plugin, MCP, and cost-related counters and histograms.
|
|
|
|
### Traces
|
|
|
|
When OTLP is enabled, the gateway exports spans through the collector into
|
|
Tempo.
|
|
|
|
### Logs and analytics
|
|
|
|
Operator-visible log and analytics surfaces remain available from the gateway
|
|
itself:
|
|
|
|
- `/api/logs`
|
|
- `/api/logs/stored`
|
|
- `/api/analytics`
|
|
- `/api/runtime`
|
|
|
|
## 6. Storage considerations
|
|
|
|
- Prometheus uses the `prometheus-data` volume.
|
|
- Grafana uses the `grafana-data` volume.
|
|
- Tempo's WAL uses `tempo-wal`.
|
|
- Trace blocks are stored in MinIO, backed by
|
|
`${FROSTY_TEMPO_STORAGE_PATH:-./data/tempo}`.
|
|
|
|
## 7. Troubleshooting monitoring
|
|
|
|
- No traces in Tempo: verify `OTEL_EXPORTER_OTLP_ENDPOINT` and check
|
|
`otel-collector` logs.
|
|
- Metrics missing: check `GET /metrics` directly before blaming Prometheus.
|
|
- Grafana up but empty: confirm the observability profile is running and
|
|
Prometheus is scraping successfully.
|
|
- MinIO issues: verify the bucket bootstrap step from `minio-init` completed
|
|
successfully.
|