docs(grafana): Add comprehensive Loki integration guide
This commit is contained in:
Claude AI
388
apps/grafana/README-LOKI-INTEGRATION.md
Normal file
388
apps/grafana/README-LOKI-INTEGRATION.md
Normal file
@@ -0,0 +1,388 @@
|
|||||||
|
# 📊 Grafana + Loki Integration
|
||||||
|
|
||||||
|
## ✅ What Was Configured
|
||||||
|
|
||||||
|
### 1. Loki Data Source
|
||||||
|
**File**: `apps/grafana/loki-datasource.yaml`
|
||||||
|
|
||||||
|
Automatically adds Loki as a data source in Grafana:
|
||||||
|
- **URL**: http://loki.loki.svc.cluster.local:3100
|
||||||
|
- **Type**: Loki
|
||||||
|
- **Access**: Proxy (internal cluster access)
|
||||||
|
|
||||||
|
### 2. Loki Logs Dashboard
|
||||||
|
**File**: `apps/grafana/loki-dashboard.yaml`
|
||||||
|
|
||||||
|
Comprehensive dashboard with 7 panels:
|
||||||
|
|
||||||
|
#### 📈 Panel 1: Log Rate by Namespace
|
||||||
|
- Real-time log ingestion rate
|
||||||
|
- Grouped by namespace
|
||||||
|
- Shows logs/second
|
||||||
|
|
||||||
|
#### 🔥 Panel 2: Error Rate by Namespace
|
||||||
|
- Errors, exceptions, and fatal messages
|
||||||
|
- Per namespace breakdown
|
||||||
|
- 5-minute rate
|
||||||
|
|
||||||
|
#### ⚠️ Panel 3: Total Errors (Last Hour)
|
||||||
|
- Gauge showing total error count
|
||||||
|
- Color-coded thresholds:
|
||||||
|
- Green: < 10 errors
|
||||||
|
- Yellow: 10-50 errors
|
||||||
|
- Red: > 50 errors
|
||||||
|
|
||||||
|
#### 🔍 Panel 4: Log Browser
|
||||||
|
- Interactive log viewer
|
||||||
|
- Filterable by:
|
||||||
|
- Namespace (dropdown)
|
||||||
|
- Pod (dropdown)
|
||||||
|
- Search text (free text)
|
||||||
|
- Live tail capability
|
||||||
|
|
||||||
|
#### 📊 Panel 5: Top 10 Namespaces by Log Volume
|
||||||
|
- Pie chart showing which namespaces generate most logs
|
||||||
|
- Based on last hour
|
||||||
|
|
||||||
|
#### 🎯 Panel 6: Top 10 Pods by Log Volume
|
||||||
|
- Pie chart of chattiest pods
|
||||||
|
- Filtered by selected namespace
|
||||||
|
|
||||||
|
#### 🚨 Panel 7: Errors & Warnings
|
||||||
|
- All errors/warnings across cluster
|
||||||
|
- Full log details
|
||||||
|
- Sortable and searchable
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 How to Access
|
||||||
|
|
||||||
|
### 1. Wait for ArgoCD Sync
|
||||||
|
ArgoCD will automatically apply the changes (~2-3 minutes).
|
||||||
|
|
||||||
|
### 2. Access Grafana
|
||||||
|
```bash
|
||||||
|
# Get Grafana URL
|
||||||
|
kubectl get ingress -n monitoring
|
||||||
|
|
||||||
|
# Or port-forward
|
||||||
|
kubectl port-forward -n monitoring svc/k8s-monitoring-grafana 3000:80
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Find the Dashboard
|
||||||
|
In Grafana:
|
||||||
|
1. Click **Dashboards** (left menu)
|
||||||
|
2. Search for **"Loki Logs Dashboard"**
|
||||||
|
3. Or navigate to: **Dashboards → Browse → loki-logs**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔍 How to Use the Dashboard
|
||||||
|
|
||||||
|
### Filters at the Top
|
||||||
|
|
||||||
|
**Namespace Filter:**
|
||||||
|
- Select one or multiple namespaces
|
||||||
|
- Default: All namespaces
|
||||||
|
|
||||||
|
**Pod Filter:**
|
||||||
|
- Dynamically updates based on selected namespace(s)
|
||||||
|
- Default: All pods
|
||||||
|
|
||||||
|
**Search Box:**
|
||||||
|
- Free-text search across all logs
|
||||||
|
- Examples:
|
||||||
|
- `error` - find errors
|
||||||
|
- `timeout` - find timeouts
|
||||||
|
- `sync` - find ArgoCD syncs
|
||||||
|
|
||||||
|
### Example Workflows
|
||||||
|
|
||||||
|
#### 1. Debug Application Errors
|
||||||
|
```
|
||||||
|
1. Select namespace: "default"
|
||||||
|
2. Select pod: "myapp-xyz"
|
||||||
|
3. Search: "error"
|
||||||
|
4. Look at "Log Browser" panel
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 2. Monitor ArgoCD
|
||||||
|
```
|
||||||
|
1. Select namespace: "argocd"
|
||||||
|
2. Search: "sync"
|
||||||
|
3. Check "Error Rate" and "Log Browser"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 3. Find Noisy Pods
|
||||||
|
```
|
||||||
|
1. Look at "Top 10 Pods by Log Volume"
|
||||||
|
2. Click on highest pod
|
||||||
|
3. Use "Log Browser" to see what it's logging
|
||||||
|
```
|
||||||
|
|
||||||
|
#### 4. Cluster-Wide Error Monitoring
|
||||||
|
```
|
||||||
|
1. Set namespace to "All"
|
||||||
|
2. Check "Total Errors" gauge
|
||||||
|
3. Look at "Errors & Warnings" panel at bottom
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📝 LogQL Query Examples
|
||||||
|
|
||||||
|
The dashboard uses LogQL (Loki Query Language). Here are some queries you can use:
|
||||||
|
|
||||||
|
### Basic Queries
|
||||||
|
```logql
|
||||||
|
# All logs from a namespace
|
||||||
|
{namespace="loki"}
|
||||||
|
|
||||||
|
# Logs from specific pod
|
||||||
|
{pod="loki-0"}
|
||||||
|
|
||||||
|
# Multiple namespaces
|
||||||
|
{namespace=~"loki|argocd|grafana"}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Filtering
|
||||||
|
```logql
|
||||||
|
# Contains "error"
|
||||||
|
{namespace="default"} |= "error"
|
||||||
|
|
||||||
|
# Regex match
|
||||||
|
{namespace="argocd"} |~ "sync|deploy"
|
||||||
|
|
||||||
|
# NOT containing
|
||||||
|
{namespace="loki"} != "debug"
|
||||||
|
|
||||||
|
# Case insensitive
|
||||||
|
{namespace="default"} |~ "(?i)error"
|
||||||
|
```
|
||||||
|
|
||||||
|
### Metrics Queries
|
||||||
|
```logql
|
||||||
|
# Log rate per namespace
|
||||||
|
sum by (namespace) (rate({namespace=~".+"}[1m]))
|
||||||
|
|
||||||
|
# Error count
|
||||||
|
sum(count_over_time({namespace=~".+"} |~ "(?i)error"[5m]))
|
||||||
|
|
||||||
|
# Top namespaces
|
||||||
|
topk(10, sum by (namespace) (count_over_time({namespace=~".+"}[1h])))
|
||||||
|
```
|
||||||
|
|
||||||
|
### JSON Parsing
|
||||||
|
```logql
|
||||||
|
# If logs are JSON
|
||||||
|
{namespace="app"} | json | level="error"
|
||||||
|
|
||||||
|
# Extract fields
|
||||||
|
{namespace="app"} | json | line_format "{{.message}}"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎨 Dashboard Customization
|
||||||
|
|
||||||
|
### Add New Panel
|
||||||
|
|
||||||
|
1. Click **"Add panel"** (top right)
|
||||||
|
2. Select **"Loki"** as data source
|
||||||
|
3. Write your LogQL query
|
||||||
|
4. Choose visualization type
|
||||||
|
5. Save
|
||||||
|
|
||||||
|
### Useful Panel Types
|
||||||
|
|
||||||
|
- **Time series**: For rates and counts over time
|
||||||
|
- **Logs**: For viewing actual log lines
|
||||||
|
- **Stat**: For single values (like total errors)
|
||||||
|
- **Gauge**: For thresholds (like error counts)
|
||||||
|
- **Table**: For structured data
|
||||||
|
- **Pie chart**: For distribution
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔧 Troubleshooting
|
||||||
|
|
||||||
|
### Dashboard Not Appearing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check ConfigMap created
|
||||||
|
kubectl get configmap -n monitoring loki-logs-dashboard
|
||||||
|
|
||||||
|
# Check Grafana pod logs
|
||||||
|
kubectl logs -n monitoring -l app.kubernetes.io/name=grafana
|
||||||
|
|
||||||
|
# Restart Grafana
|
||||||
|
kubectl rollout restart deployment/k8s-monitoring-grafana -n monitoring
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Source Not Working
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Test Loki from Grafana pod
|
||||||
|
kubectl exec -n monitoring -it deployment/k8s-monitoring-grafana -- \
|
||||||
|
curl http://loki.loki.svc.cluster.local:3100/ready
|
||||||
|
|
||||||
|
# Should return: ready
|
||||||
|
```
|
||||||
|
|
||||||
|
### No Logs Showing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check Promtail is running
|
||||||
|
kubectl get pods -n loki -l app.kubernetes.io/name=promtail
|
||||||
|
|
||||||
|
# Check Promtail logs
|
||||||
|
kubectl logs -n loki -l app.kubernetes.io/name=promtail --tail=50
|
||||||
|
|
||||||
|
# Test query directly
|
||||||
|
kubectl exec -n monitoring -it deployment/k8s-monitoring-grafana -- \
|
||||||
|
curl "http://loki.loki.svc.cluster.local:3100/loki/api/v1/labels"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 What Logs Are Collected
|
||||||
|
|
||||||
|
Promtail collects logs from:
|
||||||
|
|
||||||
|
### 1. All Pod Logs
|
||||||
|
```
|
||||||
|
/var/log/pods/<namespace>_<pod>_<uid>/<container>/*.log
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Labels Added Automatically
|
||||||
|
Every log line gets these labels:
|
||||||
|
- `namespace` - Kubernetes namespace
|
||||||
|
- `pod` - Pod name
|
||||||
|
- `container` - Container name
|
||||||
|
- `node` - Node where pod runs
|
||||||
|
- `job` - Always "kubernetes-pods"
|
||||||
|
|
||||||
|
### 3. Example Log Entry
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"namespace": "loki",
|
||||||
|
"pod": "loki-0",
|
||||||
|
"container": "loki",
|
||||||
|
"node": "master1",
|
||||||
|
"timestamp": "2026-01-05T13:30:00Z",
|
||||||
|
"line": "level=info msg=\"flushing stream\""
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Advanced Features
|
||||||
|
|
||||||
|
### Live Tail
|
||||||
|
Click **"Live"** button in Log Browser panel to stream logs in real-time.
|
||||||
|
|
||||||
|
### Context
|
||||||
|
Click on any log line → "Show context" to see surrounding logs.
|
||||||
|
|
||||||
|
### Log Details
|
||||||
|
Click on any log line to see:
|
||||||
|
- All labels
|
||||||
|
- Parsed fields (if JSON)
|
||||||
|
- Timestamp
|
||||||
|
- Full message
|
||||||
|
|
||||||
|
### Sharing
|
||||||
|
Click **"Share"** (top right) to:
|
||||||
|
- Copy link
|
||||||
|
- Create snapshot
|
||||||
|
- Export as JSON
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚨 Alerting (Optional)
|
||||||
|
|
||||||
|
You can create alerts based on log patterns:
|
||||||
|
|
||||||
|
### Example: Alert on High Error Rate
|
||||||
|
```yaml
|
||||||
|
alert: HighErrorRate
|
||||||
|
expr: sum(rate({namespace=~".+"} |~ "(?i)error"[5m])) > 10
|
||||||
|
for: 5m
|
||||||
|
annotations:
|
||||||
|
summary: "High error rate detected"
|
||||||
|
description: "{{ $value }} errors/sec"
|
||||||
|
```
|
||||||
|
|
||||||
|
To add alerts:
|
||||||
|
1. Go to dashboard panel
|
||||||
|
2. Click "Alert" tab
|
||||||
|
3. Configure threshold
|
||||||
|
4. Set notification channel
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📈 Performance Tips
|
||||||
|
|
||||||
|
### 1. Limit Time Range
|
||||||
|
- Use smaller time ranges for faster queries
|
||||||
|
- Default: 1 hour (good balance)
|
||||||
|
|
||||||
|
### 2. Use Filters
|
||||||
|
- Always filter by namespace/pod when possible
|
||||||
|
- Reduces data scanned
|
||||||
|
|
||||||
|
### 3. Dashboard Refresh
|
||||||
|
- Default: 10 seconds
|
||||||
|
- Increase if experiencing lag
|
||||||
|
|
||||||
|
### 4. Log Volume
|
||||||
|
- Monitor "Top 10" panels
|
||||||
|
- Consider log retention policy if volume is high
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔗 Useful Links
|
||||||
|
|
||||||
|
- **Loki API**: http://loki.loki.svc.cluster.local:3100
|
||||||
|
- **Loki Ready**: http://loki.loki.svc.cluster.local:3100/ready
|
||||||
|
- **Loki Metrics**: http://loki.loki.svc.cluster.local:3100/metrics
|
||||||
|
- **LogQL Docs**: https://grafana.com/docs/loki/latest/logql/
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📋 Quick Reference
|
||||||
|
|
||||||
|
### LogQL Operators
|
||||||
|
- `|=` - Contains (exact)
|
||||||
|
- `!=` - Does not contain
|
||||||
|
- `|~` - Regex match
|
||||||
|
- `!~` - Regex not match
|
||||||
|
- `| json` - Parse JSON
|
||||||
|
- `| logfmt` - Parse logfmt
|
||||||
|
- `| line_format` - Format output
|
||||||
|
|
||||||
|
### Rate Functions
|
||||||
|
- `rate()` - Per-second rate
|
||||||
|
- `count_over_time()` - Total count
|
||||||
|
- `bytes_over_time()` - Total bytes
|
||||||
|
- `bytes_rate()` - Bytes per second
|
||||||
|
|
||||||
|
### Aggregations
|
||||||
|
- `sum by (label)` - Sum grouped by label
|
||||||
|
- `count by (label)` - Count grouped
|
||||||
|
- `avg by (label)` - Average
|
||||||
|
- `max by (label)` - Maximum
|
||||||
|
- `topk(n, query)` - Top N results
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Dashboard is ready! It will appear in Grafana after ArgoCD syncs (~2-3 minutes).** 🎉
|
||||||
|
|
||||||
|
## Next Steps
|
||||||
|
|
||||||
|
1. ✅ Wait for ArgoCD sync
|
||||||
|
2. ✅ Open Grafana
|
||||||
|
3. ✅ Find "Loki Logs Dashboard"
|
||||||
|
4. ✅ Start exploring your logs!
|
||||||
|
|
||||||
|
Want me to add more panels or create specific queries for your use case?
|
||||||
Reference in New Issue
Block a user