k3s-gitops/apps/loki/README-EXTERNAL-ACCESS.md

Loki External Access Setup

Overview

Loki is now accessible externally via: https://loki.thedevops.dev

Configuration

Ingress

• Domain: loki.thedevops.dev
• TLS: Enabled with Let's Encrypt (cert-manager)
• Authentication: Basic Auth
• Service: loki:3100

Authentication

Default credentials:

• Username: admin
• Password: lokipass123

⚠️ IMPORTANT: Change the password after deployment!

Files Created

1. ingress.yaml - Main ingress configuration
2. middleware-auth.yaml - Traefik basic auth middleware
3. secret-basic-auth.yaml - Basic auth credentials

DNS Configuration

Add this A record to your DNS:

loki.thedevops.dev  →  5.182.17.194

Replace 5.182.17.194 with your actual cluster IP.

Testing Access

1. Check Loki Health

curl -u admin:lokipass123 https://loki.thedevops.dev/ready

Expected response: ready

2. Query Loki

# Get labels
curl -u admin:lokipass123 https://loki.thedevops.dev/loki/api/v1/labels

# Query logs
curl -u admin:lokipass123 -G https://loki.thedevops.dev/loki/api/v1/query \
  --data-urlencode 'query={namespace="loki"}'

3. Test from Grafana

Add Loki as a data source in Grafana:

URL: https://loki.thedevops.dev
Auth: Basic Auth
  User: admin
  Password: lokipass123

Changing the Password

Method 1: Generate new password locally

# Generate new password hash
htpasswd -nb admin your-new-password | base64

# Update secret-basic-auth.yaml with new hash
kubectl apply -f apps/loki/secret-basic-auth.yaml

Method 2: Using kubectl directly

# Create new secret
kubectl create secret generic loki-basic-auth \
  --from-literal=users=$(htpasswd -nb admin your-new-password) \
  --namespace loki \
  --dry-run=client -o yaml | kubectl apply -f -

Troubleshooting

Ingress not working

# Check ingress
kubectl get ingress -n loki

# Check certificate
kubectl get certificate -n loki

# Check if Loki is running
kubectl get pods -n loki

Certificate not issued

# Check cert-manager
kubectl get certificaterequest -n loki
kubectl describe certificate loki-tls -n loki

# Check Let's Encrypt challenge
kubectl get challenges -n loki

Authentication not working

# Check secret exists
kubectl get secret loki-basic-auth -n loki

# Check middleware
kubectl get middleware -n loki

# Verify secret content
kubectl get secret loki-basic-auth -n loki -o jsonpath='{.data.users}' | base64 -d

Architecture

Internet
   ↓
DNS (loki.thedevops.dev)
   ↓
Traefik Ingress Controller
   ↓
TLS Termination (Let's Encrypt)
   ↓
Basic Auth Middleware
   ↓
Loki Service (ClusterIP:3100)
   ↓
Loki StatefulSet

Security Considerations

1. TLS: All traffic encrypted with Let's Encrypt certificate
2. Authentication: Basic Auth protects access
3. Network Policy: Consider adding network policies for additional security
4. Password Rotation: Change default password immediately
5. Rate Limiting: Consider adding rate limiting middleware

Integration with Grafana

If you want to access Loki from Grafana (already in cluster):

Option 1: Internal access (recommended)

Use internal service URL: http://loki.loki.svc.cluster.local:3100 No authentication needed for in-cluster access.

Option 2: External access

Use: https://loki.thedevops.dev Requires basic auth credentials.

ArgoCD Sync

ArgoCD will automatically sync these changes:

• Ingress will be created
• TLS certificate will be requested
• Basic auth will be configured

Wait ~2-3 minutes for:

1. Ingress to be created
2. Let's Encrypt to issue certificate
3. DNS propagation (if DNS was just updated)

Verification Checklist

• DNS A record configured
• ArgoCD synced successfully
• Certificate issued (check cert-manager)
• Loki pods running
• Ingress created
• Can access https://loki.thedevops.dev
• Basic auth working
• Default password changed
• Grafana data source configured (if applicable)

Useful Commands

# Watch ArgoCD sync
argocd app get loki --refresh

# Check Loki logs
kubectl logs -n loki -l app.kubernetes.io/name=loki --tail=50

# Test Loki internally (from within cluster)
kubectl run test-loki --rm -it --image=curlimages/curl -- \
  curl http://loki.loki.svc.cluster.local:3100/ready

# Check ingress events
kubectl describe ingress loki -n loki

# Force certificate renewal
kubectl delete certificate loki-tls -n loki

Next Steps

1. Configure DNS A record
2. Wait for ArgoCD to sync (~3 minutes)
3. Wait for Let's Encrypt certificate (~2 minutes)
4. Test access with curl
5. Change default password
6. Configure Grafana data source (if needed)

---

Created: 2026-01-05 Maintained by: DevOps Team