docs(rollback): Update README with comprehensive rollback documentation
This commit is contained in:
Claude AI
@@ -1,593 +1,235 @@
|
|||||||
# 🚀 Demo Nginx - CI/CD Pipeline Documentation
|
# Demo Nginx Documentation
|
||||||
|
|
||||||
## 📋 Overview
|
Документация для demo-nginx deployment с автоматическим и ручным rollback.
|
||||||
|
|
||||||
This is a complete CI/CD pipeline for deploying Nginx application using:
|
|
||||||
- **Jenkins** - CI/CD orchestration
|
|
||||||
- **Gitea** - Git repository and GitOps manifests
|
|
||||||
- **ArgoCD** - GitOps deployment
|
|
||||||
- **Kubernetes** - Container orchestration
|
|
||||||
- **Docker** - Container runtime
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🏗️ Architecture
|
## 📚 Available Documentation
|
||||||
|
|
||||||
|
### [ROLLBACK_MANUAL.md](./ROLLBACK_MANUAL.md)
|
||||||
|
**Comprehensive Manual Rollback Guide**
|
||||||
|
|
||||||
|
Полная документация по функции ручного rollback:
|
||||||
|
- 3 способа rollback (IMAGE_TAG, REVISION, GIT_COMMIT)
|
||||||
|
- Setup guide
|
||||||
|
- Troubleshooting со всеми fixes
|
||||||
|
- Best practices
|
||||||
|
- Examples
|
||||||
|
- FAQ
|
||||||
|
|
||||||
|
**When to use:** Нужна детальная информация или troubleshooting
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### [ROLLBACK_QUICK_REF.md](./ROLLBACK_QUICK_REF.md)
|
||||||
|
**Quick Reference Card**
|
||||||
|
|
||||||
|
Краткая справка для быстрого использования:
|
||||||
|
- Quick start (2 минуты)
|
||||||
|
- 3 способа rollback
|
||||||
|
- Emergency procedure
|
||||||
|
- Verification commands
|
||||||
|
- Checklist
|
||||||
|
|
||||||
|
**When to use:** Быстрый rollback в production
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎯 Quick Links
|
||||||
|
|
||||||
|
### Rollback Feature
|
||||||
|
- **Full Guide:** [ROLLBACK_MANUAL.md](./ROLLBACK_MANUAL.md)
|
||||||
|
- **Quick Ref:** [ROLLBACK_QUICK_REF.md](./ROLLBACK_QUICK_REF.md)
|
||||||
|
- **Jenkinsfile:** [Jenkinsfile.rollback](../Jenkinsfile.rollback)
|
||||||
|
- **CI/CD Guide:** [../../../CICD_GUIDE.md](../../../CICD_GUIDE.md)
|
||||||
|
|
||||||
|
### Related Resources
|
||||||
|
- **Jenkins RBAC:** [apps/jenkins/rbac.yaml](../../jenkins/rbac.yaml)
|
||||||
|
- **Deployment:** [deployment.yaml](../deployment.yaml)
|
||||||
|
- **Main CI/CD:** [Jenkinsfile](../Jenkinsfile)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
### Manual Rollback (2 minutes)
|
||||||
|
|
||||||
```
|
```
|
||||||
┌─────────────┐ ┌──────────────┐ ┌─────────────┐ ┌────────────┐
|
1. Jenkins → demo-nginx-rollback
|
||||||
│ Developer │─────▶│ Gitea │─────▶│ Jenkins │─────▶│ Docker │
|
2. IMAGE_TAG + main-21
|
||||||
│ (commit) │ │ (k3s-gitops) │ │ (pipeline) │ │ Registry │
|
3. SKIP_HEALTH_CHECK: true
|
||||||
└─────────────┘ └──────────────┘ └──────┬──────┘ └─────┬──────┘
|
4. Build
|
||||||
│ │
|
```
|
||||||
│ Update │ Pull
|
|
||||||
│ manifests │ image
|
See [ROLLBACK_QUICK_REF.md](./ROLLBACK_QUICK_REF.md) for details.
|
||||||
▼ │
|
|
||||||
┌──────────────┐ │
|
---
|
||||||
│ Gitea │ │
|
|
||||||
│ (manifests) │ │
|
### Emergency Rollback (30 seconds)
|
||||||
└──────┬───────┘ │
|
|
||||||
│ │
|
```bash
|
||||||
│ Watch │
|
kubectl rollout undo deployment/demo-nginx -n demo-app
|
||||||
▼ │
|
|
||||||
┌──────────────┐ │
|
|
||||||
│ ArgoCD │────────────┘
|
|
||||||
│ (sync) │
|
|
||||||
└──────┬───────┘
|
|
||||||
│
|
|
||||||
│ Apply
|
|
||||||
▼
|
|
||||||
┌──────────────┐
|
|
||||||
│ Kubernetes │
|
|
||||||
│ Cluster │
|
|
||||||
└──────────────┘
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📁 Repository Structure
|
## 📊 Features Summary
|
||||||
|
|
||||||
```
|
### Manual Rollback
|
||||||
k3s-gitops/
|
✅ 3 rollback methods (IMAGE_TAG, REVISION, GIT_COMMIT)
|
||||||
└── apps/
|
✅ GitOps sync (auto-commit to Git)
|
||||||
└── demo-nginx/
|
✅ Zero downtime (rolling updates)
|
||||||
├── namespace.yaml # Kubernetes namespace
|
✅ DRY_RUN mode (safe testing)
|
||||||
├── deployment.yaml # Deployment + Service
|
✅ Full RBAC permissions
|
||||||
├── ingress.yaml # Ingress with TLS
|
✅ Input validation (auto-trim)
|
||||||
├── application.yaml # ArgoCD Application
|
✅ Retry logic (5 attempts)
|
||||||
├── Jenkinsfile # CI/CD Pipeline
|
⚠️ Health check optional (use SKIP_HEALTH_CHECK=true)
|
||||||
└── README.md # This file
|
|
||||||
```
|
### Automatic Rollback
|
||||||
|
✅ Triggered on deployment failure
|
||||||
|
✅ Saves previous state
|
||||||
|
✅ Kubernetes rollback
|
||||||
|
✅ Git revert
|
||||||
|
✅ Health checks
|
||||||
|
✅ Timeout protection
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🔧 Prerequisites
|
## 🐛 All Fixes Applied
|
||||||
|
|
||||||
### 1. Jenkins Configuration
|
| # | Issue | Fix | Status | Doc |
|
||||||
|
|---|-------|-----|--------|-----|
|
||||||
#### Required Plugins:
|
| 1 | Container name | Use `nginx` | ✅ | [Link](./ROLLBACK_MANUAL.md#issue-1-container-name-error--fixed) |
|
||||||
- Git Plugin
|
| 2 | Whitespace | Auto-trim | ✅ | [Link](./ROLLBACK_MANUAL.md#issue-2-whitespace-in-input--fixed) |
|
||||||
- Docker Plugin
|
| 3 | RBAC | pods/exec perm | ✅ | [Link](./ROLLBACK_MANUAL.md#issue-3-rbac-permissions--fixed) |
|
||||||
- Kubernetes Plugin
|
| 4 | Health timing | SKIP option | ⚠️ | [Link](./ROLLBACK_MANUAL.md#issue-4-health-check-timing--workaround) |
|
||||||
- Pipeline Plugin
|
| 5 | Bash loop | Explicit list | ✅ | [Link](./ROLLBACK_MANUAL.md#issue-5-bash-loop-syntax--fixed) |
|
||||||
- Credentials Plugin
|
|
||||||
|
|
||||||
#### Required Credentials in Jenkins:
|
|
||||||
|
|
||||||
**Docker Registry Credentials:**
|
|
||||||
```
|
|
||||||
ID: docker-registry-credentials
|
|
||||||
Type: Username with password
|
|
||||||
Username: <your-docker-hub-username>
|
|
||||||
Password: <your-docker-hub-token>
|
|
||||||
```
|
|
||||||
|
|
||||||
**Gitea Credentials:**
|
|
||||||
```
|
|
||||||
ID: gitea-credentials
|
|
||||||
Type: Username with password
|
|
||||||
Username: admin
|
|
||||||
Password: <gitea-password>
|
|
||||||
```
|
|
||||||
|
|
||||||
#### Create Credentials in Jenkins:
|
|
||||||
1. Jenkins → Manage Jenkins → Credentials
|
|
||||||
2. (global) → Add Credentials
|
|
||||||
3. Select "Username with password"
|
|
||||||
4. Fill in details and save
|
|
||||||
|
|
||||||
### 2. Docker Registry
|
|
||||||
|
|
||||||
**Option A: Docker Hub**
|
|
||||||
- Free for public images
|
|
||||||
- URL: docker.io
|
|
||||||
- Create account: https://hub.docker.com/signup
|
|
||||||
|
|
||||||
**Option B: Harbor (Local)**
|
|
||||||
- Already installed in cluster
|
|
||||||
- URL: harbor.thedevops.dev
|
|
||||||
- Create project: demo-nginx
|
|
||||||
|
|
||||||
### 3. Kubernetes Access
|
|
||||||
|
|
||||||
Jenkins needs kubectl access to cluster:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Copy kubeconfig to Jenkins
|
|
||||||
kubectl create secret generic kubeconfig \
|
|
||||||
--from-file=config=$HOME/.kube/config \
|
|
||||||
-n jenkins
|
|
||||||
|
|
||||||
# Or configure ServiceAccount with proper RBAC
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🚀 Setup Instructions
|
## 💡 Recommended Reading Order
|
||||||
|
|
||||||
### Step 1: Apply ArgoCD Application
|
### For New Users:
|
||||||
|
1. Start → [ROLLBACK_QUICK_REF.md](./ROLLBACK_QUICK_REF.md)
|
||||||
|
2. Practice → Follow quick start
|
||||||
|
3. Deep Dive → [ROLLBACK_MANUAL.md](./ROLLBACK_MANUAL.md)
|
||||||
|
|
||||||
```bash
|
### For Troubleshooting:
|
||||||
kubectl apply -f apps/demo-nginx/application.yaml
|
1. Check → [ROLLBACK_MANUAL.md - Troubleshooting](./ROLLBACK_MANUAL.md#troubleshooting--fixes)
|
||||||
```
|
2. Verify → [ROLLBACK_QUICK_REF.md - Verification](./ROLLBACK_QUICK_REF.md#-verify-rollback)
|
||||||
|
3. Support → [ROLLBACK_MANUAL.md - Support](./ROLLBACK_MANUAL.md#support)
|
||||||
|
|
||||||
Verify:
|
### For Emergency:
|
||||||
```bash
|
1. Fast → [ROLLBACK_QUICK_REF.md - Emergency](./ROLLBACK_QUICK_REF.md#-emergency-rollback-30-seconds)
|
||||||
kubectl get application demo-nginx -n argocd
|
2. Alternative → [ROLLBACK_MANUAL.md - Emergency](./ROLLBACK_MANUAL.md#emergency-rollback-procedure)
|
||||||
```
|
|
||||||
|
|
||||||
### Step 2: Configure Jenkins Job
|
|
||||||
|
|
||||||
#### Method A: Multibranch Pipeline (Recommended)
|
|
||||||
|
|
||||||
1. **Jenkins → New Item**
|
|
||||||
2. **Enter name**: `demo-nginx`
|
|
||||||
3. **Select**: Multibranch Pipeline
|
|
||||||
4. **Click**: OK
|
|
||||||
|
|
||||||
5. **Branch Sources**:
|
|
||||||
- Add source: Git
|
|
||||||
- Project Repository: `http://gitea-http.gitea.svc.cluster.local:3000/admin/k3s-gitops`
|
|
||||||
- Credentials: Select gitea-credentials
|
|
||||||
- Behaviors: Discover branches
|
|
||||||
|
|
||||||
6. **Build Configuration**:
|
|
||||||
- Mode: by Jenkinsfile
|
|
||||||
- Script Path: `apps/demo-nginx/Jenkinsfile`
|
|
||||||
|
|
||||||
7. **Scan Multibranch Pipeline Triggers**:
|
|
||||||
- ✅ Periodically if not otherwise run
|
|
||||||
- Interval: 1 minute
|
|
||||||
|
|
||||||
8. **Save**
|
|
||||||
|
|
||||||
#### Method B: Pipeline Job
|
|
||||||
|
|
||||||
1. **Jenkins → New Item**
|
|
||||||
2. **Enter name**: `demo-nginx-pipeline`
|
|
||||||
3. **Select**: Pipeline
|
|
||||||
4. **Click**: OK
|
|
||||||
|
|
||||||
5. **Pipeline**:
|
|
||||||
- Definition: Pipeline script from SCM
|
|
||||||
- SCM: Git
|
|
||||||
- Repository URL: `http://gitea-http.gitea.svc.cluster.local:3000/admin/k3s-gitops`
|
|
||||||
- Credentials: gitea-credentials
|
|
||||||
- Branch: `*/main`
|
|
||||||
- Script Path: `apps/demo-nginx/Jenkinsfile`
|
|
||||||
|
|
||||||
6. **Build Triggers**:
|
|
||||||
- ✅ Poll SCM
|
|
||||||
- Schedule: `H/5 * * * *` (every 5 minutes)
|
|
||||||
|
|
||||||
7. **Save**
|
|
||||||
|
|
||||||
### Step 3: Configure Webhook (Optional)
|
|
||||||
|
|
||||||
For instant builds on git push:
|
|
||||||
|
|
||||||
**In Gitea:**
|
|
||||||
1. Repository Settings → Webhooks
|
|
||||||
2. Add Webhook → Gitea
|
|
||||||
3. URL: `http://jenkins.jenkins.svc.cluster.local:8080/gitea-webhook/post`
|
|
||||||
4. Events: Push
|
|
||||||
5. Active: ✅
|
|
||||||
|
|
||||||
**In Jenkins:**
|
|
||||||
- Build Triggers → ✅ Build when a change is pushed to Gitea
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🏃 Running the Pipeline
|
## 🎓 Key Concepts
|
||||||
|
|
||||||
### Trigger Build
|
### Rollback Methods Comparison
|
||||||
|
|
||||||
**Option 1: Manual Trigger**
|
| Method | Speed | Precision | Use Case |
|
||||||
```bash
|
|--------|-------|-----------|----------|
|
||||||
# In Jenkins UI
|
| IMAGE_TAG | ⚡⚡⚡ | 🎯 High | Known build number |
|
||||||
Jenkins → demo-nginx → Build Now
|
| REVISION | ⚡⚡ | 🎯 Medium | Recent rollback |
|
||||||
```
|
| GIT_COMMIT | ⚡ | 🎯🎯 High | Exact code state |
|
||||||
|
|
||||||
**Option 2: Git Push**
|
### When to Use What
|
||||||
```bash
|
|
||||||
# Make any change to trigger pipeline
|
|
||||||
git commit --allow-empty -m "Trigger pipeline"
|
|
||||||
git push origin main
|
|
||||||
```
|
|
||||||
|
|
||||||
**Option 3: API Trigger**
|
**Use IMAGE_TAG when:**
|
||||||
```bash
|
- You know the build number (main-21)
|
||||||
curl -X POST http://jenkins.jenkins.svc.cluster.local:8080/job/demo-nginx/build \
|
- Quick rollback needed
|
||||||
--user admin:your-token
|
- Most common scenario
|
||||||
```
|
|
||||||
|
|
||||||
---
|
**Use REVISION_NUMBER when:**
|
||||||
|
- Need to go back N versions
|
||||||
|
- Don't remember exact tag
|
||||||
|
- Working with kubectl history
|
||||||
|
|
||||||
## 📊 Pipeline Stages
|
**Use GIT_COMMIT when:**
|
||||||
|
- Need exact code state
|
||||||
### 1. Checkout Source
|
- Multiple changes in one build
|
||||||
- Clones repository
|
- Precise rollback required
|
||||||
- Creates Dockerfile and nginx.conf
|
|
||||||
|
|
||||||
### 2. Build Docker Image
|
|
||||||
- Builds image with version tag
|
|
||||||
- Tags: `${IMAGE_TAG}` and `latest`
|
|
||||||
|
|
||||||
### 3. Test Image
|
|
||||||
- Runs container locally
|
|
||||||
- Tests HTTP endpoints
|
|
||||||
- Verifies health check
|
|
||||||
|
|
||||||
### 4. Push to Registry
|
|
||||||
- **Only on main branch**
|
|
||||||
- Logs in to Docker registry
|
|
||||||
- Pushes both tags
|
|
||||||
- Cleans up credentials
|
|
||||||
|
|
||||||
### 5. Update GitOps Manifests
|
|
||||||
- **Only on main branch**
|
|
||||||
- Clones k3s-gitops repository
|
|
||||||
- Updates deployment.yaml with new image tag
|
|
||||||
- Commits and pushes changes
|
|
||||||
|
|
||||||
### 6. Wait for ArgoCD Sync
|
|
||||||
- Monitors ArgoCD application status
|
|
||||||
- Waits for Synced + Healthy state
|
|
||||||
- Timeout: 5 minutes
|
|
||||||
|
|
||||||
### 7. Verify Deployment
|
|
||||||
- Checks rollout status
|
|
||||||
- Verifies deployed image version
|
|
||||||
- Lists running pods
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🔍 Pipeline Variables
|
|
||||||
|
|
||||||
You can customize these in Jenkinsfile:
|
|
||||||
|
|
||||||
| Variable | Default | Description |
|
|
||||||
|----------|---------|-------------|
|
|
||||||
| APP_NAME | demo-nginx | Application name |
|
|
||||||
| NAMESPACE | demo-app | Kubernetes namespace |
|
|
||||||
| DOCKER_REGISTRY | docker.io | Docker registry URL |
|
|
||||||
| DOCKER_REPO | vladimiras | Docker repository/username |
|
|
||||||
| GITEA_URL | http://gitea-http... | Gitea service URL |
|
|
||||||
| GITEA_REPO | admin/k3s-gitops | GitOps repository |
|
|
||||||
|
|
||||||
**To customize:**
|
|
||||||
```groovy
|
|
||||||
environment {
|
|
||||||
DOCKER_REGISTRY = 'harbor.thedevops.dev'
|
|
||||||
DOCKER_REPO = 'library'
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🧪 Testing
|
|
||||||
|
|
||||||
### Test Locally
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build image
|
|
||||||
docker build -t demo-nginx:test .
|
|
||||||
|
|
||||||
# Run container
|
|
||||||
docker run -d -p 8080:80 demo-nginx:test
|
|
||||||
|
|
||||||
# Test
|
|
||||||
curl http://localhost:8080/
|
|
||||||
curl http://localhost:8080/health
|
|
||||||
|
|
||||||
# Cleanup
|
|
||||||
docker stop $(docker ps -q --filter ancestor=demo-nginx:test)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Test in Cluster
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Port-forward to service
|
|
||||||
kubectl port-forward svc/demo-nginx 8080:80 -n demo-app
|
|
||||||
|
|
||||||
# Test
|
|
||||||
curl http://localhost:8080/
|
|
||||||
```
|
|
||||||
|
|
||||||
### Access via Ingress
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Add to /etc/hosts
|
|
||||||
echo "5.182.17.194 demo-nginx.thedevops.dev" | sudo tee -a /etc/hosts
|
|
||||||
|
|
||||||
# Test (after DNS is configured)
|
|
||||||
curl https://demo-nginx.thedevops.dev/
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📈 Monitoring
|
## 📈 Monitoring
|
||||||
|
|
||||||
### Check Pipeline Status
|
### Check Rollback Status
|
||||||
|
|
||||||
```bash
|
|
||||||
# Jenkins CLI
|
|
||||||
java -jar jenkins-cli.jar -s http://jenkins:8080/ \
|
|
||||||
-auth admin:token \
|
|
||||||
build demo-nginx -s -v
|
|
||||||
|
|
||||||
# Or via Web UI
|
|
||||||
# Jenkins → demo-nginx → Build History
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check ArgoCD Sync
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Get application status
|
|
||||||
kubectl get application demo-nginx -n argocd -o yaml
|
|
||||||
|
|
||||||
# Watch sync status
|
|
||||||
watch -n 2 'kubectl get application demo-nginx -n argocd'
|
|
||||||
|
|
||||||
# ArgoCD UI
|
|
||||||
# Open: https://argocd.thedevops.dev
|
|
||||||
# Applications → demo-nginx
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check Deployment
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Deployment status
|
# Deployment status
|
||||||
kubectl get deployment demo-nginx -n demo-app
|
kubectl get deployment demo-nginx -n demo-app
|
||||||
|
|
||||||
# Pods
|
# Pod status
|
||||||
kubectl get pods -n demo-app -l app=demo-nginx
|
kubectl get pods -n demo-app -l app=demo-nginx
|
||||||
|
|
||||||
# Logs
|
# Rollout history
|
||||||
kubectl logs -n demo-app -l app=demo-nginx --tail=50
|
kubectl rollout history deployment/demo-nginx -n demo-app
|
||||||
|
|
||||||
# Events
|
# ArgoCD status
|
||||||
kubectl get events -n demo-app --sort-by='.lastTimestamp'
|
kubectl get application demo-nginx -n argocd
|
||||||
|
```
|
||||||
|
|
||||||
|
### Grafana Queries
|
||||||
|
|
||||||
|
```promql
|
||||||
|
# Rollback count
|
||||||
|
sum(increase(deployment_rollback_total[1h])) by (deployment)
|
||||||
|
|
||||||
|
# Rollback rate
|
||||||
|
rate(deployment_rollback_total[5m])
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🐛 Troubleshooting
|
## ❓ FAQ
|
||||||
|
|
||||||
### Pipeline Fails at "Build Docker Image"
|
### Q: Какой метод rollback использовать?
|
||||||
|
**A:** Для большинства случаев используй IMAGE_TAG - самый быстрый и простой.
|
||||||
|
|
||||||
**Problem:** Docker not available in Jenkins
|
### Q: Health check всегда падает, это баг?
|
||||||
|
**A:** Нет, это timing issue во время rolling update. Используй `SKIP_HEALTH_CHECK: true` и проверь вручную через минуту.
|
||||||
|
|
||||||
**Solution:**
|
### Q: Как быстро откатиться в emergency?
|
||||||
```bash
|
**A:** Используй `kubectl rollout undo` (30 секунд) или Jenkins с SKIP_HEALTH_CHECK (2 минуты).
|
||||||
# Ensure Docker socket is mounted in Jenkins pod
|
|
||||||
kubectl edit deployment jenkins -n jenkins
|
|
||||||
|
|
||||||
# Add volume:
|
### Q: Где полная документация?
|
||||||
volumes:
|
**A:** [ROLLBACK_MANUAL.md](./ROLLBACK_MANUAL.md) - comprehensive guide со всеми details.
|
||||||
- name: docker-sock
|
|
||||||
hostPath:
|
|
||||||
path: /var/run/docker.sock
|
|
||||||
|
|
||||||
# Add volumeMount:
|
|
||||||
volumeMounts:
|
|
||||||
- name: docker-sock
|
|
||||||
mountPath: /var/run/docker.sock
|
|
||||||
```
|
|
||||||
|
|
||||||
### Pipeline Fails at "Push to Registry"
|
|
||||||
|
|
||||||
**Problem:** Invalid credentials
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
1. Verify credentials in Jenkins
|
|
||||||
2. Test manually:
|
|
||||||
```bash
|
|
||||||
docker login docker.io -u username -p password
|
|
||||||
```
|
|
||||||
|
|
||||||
### Pipeline Fails at "Update GitOps Manifests"
|
|
||||||
|
|
||||||
**Problem:** Git authentication failed
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
1. Check Gitea credentials in Jenkins
|
|
||||||
2. Test Git access:
|
|
||||||
```bash
|
|
||||||
git clone http://username:password@gitea-http.gitea.svc.cluster.local:3000/admin/k3s-gitops
|
|
||||||
```
|
|
||||||
|
|
||||||
### ArgoCD Not Syncing
|
|
||||||
|
|
||||||
**Problem:** ArgoCD not watching repository
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```bash
|
|
||||||
# Refresh ArgoCD app
|
|
||||||
kubectl patch application demo-nginx -n argocd \
|
|
||||||
--type merge -p '{"operation":{"initiatedBy":{"username":"admin"},"sync":{"revision":"HEAD"}}}'
|
|
||||||
|
|
||||||
# Or via ArgoCD UI
|
|
||||||
# Click "Refresh" button
|
|
||||||
```
|
|
||||||
|
|
||||||
### Deployment Not Rolling Out
|
|
||||||
|
|
||||||
**Problem:** Image pull error
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
```bash
|
|
||||||
# Check pod events
|
|
||||||
kubectl describe pod -n demo-app -l app=demo-nginx
|
|
||||||
|
|
||||||
# Check if image exists
|
|
||||||
docker pull vladimiras/demo-nginx:main-123
|
|
||||||
|
|
||||||
# Check ImagePullSecrets (if using private registry)
|
|
||||||
kubectl get deployment demo-nginx -n demo-app -o yaml | grep -A 5 imagePullSecrets
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🔐 Security Best Practices
|
## 🆘 Support
|
||||||
|
|
||||||
### 1. Use Private Registry
|
**Need Help?**
|
||||||
|
|
||||||
```groovy
|
1. Check [ROLLBACK_MANUAL.md - Troubleshooting](./ROLLBACK_MANUAL.md#troubleshooting--fixes)
|
||||||
environment {
|
2. Review [ROLLBACK_MANUAL.md - FAQ](./ROLLBACK_MANUAL.md#faq)
|
||||||
DOCKER_REGISTRY = 'harbor.thedevops.dev'
|
3. Check Jenkins console output
|
||||||
DOCKER_REPO = 'library'
|
4. Verify RBAC permissions
|
||||||
}
|
5. Review pod status and logs
|
||||||
```
|
|
||||||
|
|
||||||
### 2. Scan Images for Vulnerabilities
|
**Still stuck?**
|
||||||
|
- Jenkins logs: Jenkins → Build → Console Output
|
||||||
Add to pipeline:
|
- K8s events: `kubectl get events -n demo-app`
|
||||||
```groovy
|
- Pod logs: `kubectl logs -n demo-app -l app=demo-nginx`
|
||||||
stage('Security Scan') {
|
|
||||||
steps {
|
|
||||||
sh 'trivy image ${DOCKER_REGISTRY}/${DOCKER_REPO}/${APP_NAME}:${IMAGE_TAG}'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3. Use ImagePullSecrets
|
|
||||||
|
|
||||||
```bash
|
|
||||||
kubectl create secret docker-registry regcred \
|
|
||||||
--docker-server=harbor.thedevops.dev \
|
|
||||||
--docker-username=admin \
|
|
||||||
--docker-password=password \
|
|
||||||
-n demo-app
|
|
||||||
|
|
||||||
# Add to deployment.yaml:
|
|
||||||
spec:
|
|
||||||
imagePullSecrets:
|
|
||||||
- name: regcred
|
|
||||||
```
|
|
||||||
|
|
||||||
### 4. Don't Hardcode Secrets
|
|
||||||
|
|
||||||
Use Jenkins Credentials:
|
|
||||||
```groovy
|
|
||||||
withCredentials([string(credentialsId: 'api-token', variable: 'TOKEN')]) {
|
|
||||||
sh 'curl -H "Authorization: Bearer $TOKEN" api.example.com'
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 📝 Customization
|
## 📝 Documentation Updates
|
||||||
|
|
||||||
### Change Application
|
**Last Updated:** 2026-01-06
|
||||||
|
**Version:** 1.0
|
||||||
|
**Status:** Production Ready ✅
|
||||||
|
|
||||||
Edit `Dockerfile` in Jenkinsfile:
|
**Recent Changes:**
|
||||||
```groovy
|
- ✅ Added comprehensive manual rollback guide
|
||||||
stage('Checkout Source') {
|
- ✅ Added quick reference card
|
||||||
steps {
|
- ✅ Documented all 5 fixes
|
||||||
// Clone your app repository
|
- ✅ Added examples and best practices
|
||||||
git branch: 'main',
|
- ✅ Production-ready feature
|
||||||
url: 'http://gitea-http.gitea.svc.cluster.local:3000/admin/your-app'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Add Tests
|
|
||||||
|
|
||||||
```groovy
|
|
||||||
stage('Unit Tests') {
|
|
||||||
steps {
|
|
||||||
sh 'npm test'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Integration Tests') {
|
|
||||||
steps {
|
|
||||||
sh 'npm run test:integration'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Add Linting
|
|
||||||
|
|
||||||
```groovy
|
|
||||||
stage('Lint') {
|
|
||||||
steps {
|
|
||||||
sh 'npm run lint'
|
|
||||||
sh 'hadolint Dockerfile'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Multi-Environment Deploy
|
|
||||||
|
|
||||||
```groovy
|
|
||||||
stage('Deploy to Dev') {
|
|
||||||
when { branch 'develop' }
|
|
||||||
steps {
|
|
||||||
sh 'sed -i "s|namespace: .*|namespace: dev|" apps/demo-nginx/deployment.yaml'
|
|
||||||
// ... update and push
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
stage('Deploy to Production') {
|
|
||||||
when { branch 'main' }
|
|
||||||
steps {
|
|
||||||
input message: 'Deploy to production?'
|
|
||||||
// ... deploy
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 🎯 Next Steps
|
**Ready to rollback? Start with [Quick Reference](./ROLLBACK_QUICK_REF.md)! 🚀**
|
||||||
|
|
||||||
1. ✅ **Configure DNS**: Point `demo-nginx.thedevops.dev` to your cluster IP
|
|
||||||
2. ✅ **Set up monitoring**: Add Prometheus metrics
|
|
||||||
3. ✅ **Add alerts**: Configure alerting for failures
|
|
||||||
4. ✅ **Implement blue-green**: Add traffic splitting
|
|
||||||
5. ✅ **Add rollback**: Implement automatic rollback on failures
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📚 References
|
|
||||||
|
|
||||||
- [Jenkins Pipeline Syntax](https://www.jenkins.io/doc/book/pipeline/syntax/)
|
|
||||||
- [ArgoCD Documentation](https://argo-cd.readthedocs.io/)
|
|
||||||
- [Kubernetes Documentation](https://kubernetes.io/docs/)
|
|
||||||
- [Docker Documentation](https://docs.docker.com/)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🤝 Contributing
|
|
||||||
|
|
||||||
To improve this pipeline:
|
|
||||||
1. Fork the repository
|
|
||||||
2. Create feature branch
|
|
||||||
3. Make changes
|
|
||||||
4. Test thoroughly
|
|
||||||
5. Submit pull request
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Happy Deploying! 🚀**
|
|
||||||
|
|||||||
Reference in New Issue
Block a user