k3s-gitops/apps/ollama-mcp/QUICKSTART.md

# MCP Stack — Installation Guide

> Covers: mcp-kubernetes · mcp-gitea · mcp-prometheus · mcp-loki

---

## Prerequisites

| Requirement | Check |
|---|---|
| Docker + Docker Compose | `docker compose version` |
| Git | `git --version` |
| kubectl + kubeconfig | `kubectl get nodes` |
| Access to the Gitea repo | https://git.thedevops.dev |

---

## Step 1 — Clone the repository

```bash
git clone https://git.thedevops.dev/admin/k3s-gitops.git
cd k3s-gitops/apps/ollama-mcp
```

---

## Step 2 — Configure environment

```bash
cp .env.example .env
nano .env
```

Fill in the required values:

```env
# ── Gitea ────────────────────────────────────────────────────
GITEA_URL=https://git.thedevops.dev
GITEA_TOKEN=<your-token>        # Settings → Applications → Generate Token
GITEA_OWNER=admin               # your Gitea username

# ── Kubernetes ───────────────────────────────────────────────
K8S_CONTEXT=                    # leave empty for default context
K8S_NAMESPACE=default

# ── Ports ────────────────────────────────────────────────────
MCP_K8S_PORT=3001
MCP_GITEA_PORT=3002
MCP_PROMETHEUS_PORT=3003
MCP_LOKI_PORT=3005

# ── Prometheus ───────────────────────────────────────────────
# Find your NodePort: kubectl -n monitoring get svc kube-prometheus-stack-prometheus
PROMETHEUS_URL=http://<node-ip>:<nodeport>
GRAFANA_URL=http://<node-ip>:<grafana-nodeport>
GRAFANA_TOKEN=                  # optional — Grafana API key

# ── Loki ─────────────────────────────────────────────────────
# Find your NodePort: kubectl -n monitoring get svc loki
LOKI_URL=http://<node-ip>:<nodeport>
```

**Get your Gitea token:**
```bash
# Via browser:
# https://git.thedevops.dev/user/settings/applications
# → Generate New Token → select: repo, admin:org, write:repository

# Verify the token works:
curl -H "Authorization: token <your-token>" \
  https://git.thedevops.dev/api/v1/user
```

**Find Prometheus and Loki NodePorts:**
```bash
# Get node IP
kubectl get nodes -o wide

# Prometheus NodePort
kubectl -n monitoring get svc kube-prometheus-stack-prometheus

# Loki NodePort
kubectl -n monitoring get svc loki

# Grafana NodePort (optional)
kubectl -n monitoring get svc kube-prometheus-stack-grafana
```

---

## Step 3 — Copy kubeconfig

```bash
mkdir -p config
cp ~/.kube/config config/kubeconfig
chmod 600 config/kubeconfig

# Verify it works
kubectl --kubeconfig=config/kubeconfig get nodes
```

---

## Step 4 — Build images

```bash
docker compose build
```

Or build individually if you only want specific MCPs:

```bash
docker compose build mcp-kubernetes
docker compose build mcp-gitea
docker compose build mcp-prometheus
docker compose build mcp-loki
```

---

## Step 5 — Start services

```bash
# Start all
docker compose up -d

# Or start only the monitoring MCPs
docker compose up -d mcp-prometheus mcp-loki
```

---

## Step 6 — Verify everything is running

```bash
docker compose ps
```

Expected output:
```
NAME                      STATUS          PORTS
ollama-mcp-kubernetes     running         0.0.0.0:3001->3000/tcp
ollama-mcp-gitea          running         0.0.0.0:3002->3000/tcp
ollama-mcp-prometheus     running         0.0.0.0:3003->3000/tcp
ollama-mcp-loki           running         0.0.0.0:3005->3000/tcp
```

Health check all endpoints:
```bash
curl -s http://localhost:3001/health | jq .   # Kubernetes MCP
curl -s http://localhost:3002/health | jq .   # Gitea MCP
curl -s http://localhost:3003/health | jq .   # Prometheus MCP
curl -s http://localhost:3005/health | jq .   # Loki MCP
```

---

## Step 7 — Test each MCP

### Kubernetes MCP
```bash
# List all namespaces
curl -s -X POST http://localhost:3001/api/namespaces/list | jq .

# List pods in monitoring
curl -s -X POST http://localhost:3001/api/pods/list \
  -H "Content-Type: application/json" \
  -d '{"namespace": "monitoring"}' | jq .
```

### Gitea MCP
```bash
# List repositories
curl -s -X POST http://localhost:3002/api/repos/list \
  -H "Content-Type: application/json" \
  -d '{"owner": "admin"}' | jq .
```

### Prometheus MCP
```bash
# Firing alerts
curl -s -X POST http://localhost:3003/api/alerts \
  -H "Content-Type: application/json" \
  -d '{"firingOnly": true}' | jq .

# Top pods by CPU
curl -s -X POST http://localhost:3003/api/pod_cpu \
  -H "Content-Type: application/json" \
  -d '{"namespace": "monitoring"}' | jq .

# Node resources
curl -s -X POST http://localhost:3003/api/node_resources | jq .
```

### Loki MCP
```bash
# List namespaces with logs
curl -s -X POST http://localhost:3005/api/namespaces | jq .

# Errors in argocd last 30 min
curl -s -X POST http://localhost:3005/api/errors \
  -H "Content-Type: application/json" \
  -d '{"namespace": "argocd", "minutes": 30}' | jq .

# OOMKill events last 24h
curl -s -X POST http://localhost:3005/api/oomkilled \
  -H "Content-Type: application/json" \
  -d '{"hours": 24}' | jq .
```

---

## Updating after a git pull

```bash
cd k3s-gitops
git pull
cd apps/ollama-mcp

# Rebuild changed services
docker compose build mcp-prometheus mcp-loki

# Restart with new images
docker compose up -d mcp-prometheus mcp-loki
```

---

## Troubleshooting

### Container won't start
```bash
docker compose logs mcp-prometheus
docker compose logs mcp-loki
```

### Prometheus MCP returns connection errors
```bash
# Check Prometheus is reachable from the Docker host
curl http://<node-ip>:<nodeport>/api/v1/status/runtimeinfo

# If using cluster DNS (svc.cluster.local), it won't work from Docker.
# Use the NodePort IP in PROMETHEUS_URL instead.
```

### Loki returns empty results
```bash
# Confirm Loki has data
curl "http://<loki-ip>:<port>/loki/api/v1/labels"

# Check Promtail is shipping logs
kubectl -n monitoring logs -l app=promtail --tail=50
```

### Port already in use
```bash
# Find what's using a port
ss -tlnp | grep 3003

# Change the port in .env
MCP_PROMETHEUS_PORT=3013
docker compose up -d mcp-prometheus
```

### Kubeconfig permission denied
```bash
chmod 600 config/kubeconfig
docker compose restart mcp-kubernetes
```

---

## Management commands

```bash
cd k3s-gitops/apps/ollama-mcp

# Status
docker compose ps

# Live logs
docker compose logs -f
docker compose logs -f mcp-prometheus
docker compose logs -f mcp-loki

# Restart a specific service
docker compose restart mcp-prometheus

# Full rebuild
docker compose down
docker compose build --no-cache
docker compose up -d

# Stop everything
docker compose down
```

---

## Available endpoints reference

### Prometheus MCP (`localhost:3003`)
| Endpoint | Description |
|---|---|
| `POST /api/alerts` | Firing alerts |
| `POST /api/targets` | Scrape target health |
| `POST /api/pod_cpu` | CPU usage by pod/namespace |
| `POST /api/pod_memory` | Memory usage by pod/namespace |
| `POST /api/pod_restarts` | Restart counts |
| `POST /api/node_resources` | Node CPU/mem/disk % |
| `POST /api/pvc_usage` | PVC disk usage % |
| `POST /api/http_errors` | Nginx ingress 5xx rate |
| `POST /api/query` | Raw PromQL instant query |
| `POST /api/query_range` | Raw PromQL range query |
| `POST /api/grafana_dashboards` | List Grafana dashboards |

### Loki MCP (`localhost:3005`)
| Endpoint | Description |
|---|---|
| `POST /api/pod_logs` | Logs for a specific pod |
| `POST /api/namespace_logs` | All logs in a namespace |
| `POST /api/errors` | ERROR/WARN lines (namespace or cluster) |
| `POST /api/search` | Full-text search across logs |
| `POST /api/oomkilled` | OOMKilled events |
| `POST /api/crash_loops` | CrashLoopBackOff events |
| `POST /api/rate` | Log ingestion rate by namespace |
| `POST /api/labels` | Available Loki label names |
| `POST /api/namespaces` | Namespaces with logs |
| `POST /api/query` | Raw LogQL query |