Add apps/demo-nginx/docs/rollback.md

2026-01-06 08:01:39 +00:00
parent edde8503d3
commit 1e373a4ea8
1 changed files with 471 additions and 0 deletions
--- a/apps/demo-nginx/docs/rollback.md
+++ b/apps/demo-nginx/docs/rollback.md
@@ -0,0 +1,471 @@
+# 🔄 Automatic Rollback Feature
+
+## ✅ Что добавлено
+
+Pipeline теперь автоматически откатывается к предыдущей версии при любой ошибке деплоя!
+
+---
+
+## 🎯 Как работает
+
+### 1. **Save Current State** (перед деплоем)
+```
+📸 Сохраняет:
+- Текущий Docker image tag
+- Количество реплик
+- Git commit SHA
+```
+
+### 2. **Deploy New Version**
+```
+🚀 Деплоит новую версию через:
+- Build Docker image
+- Push to registry
+- Update Git manifests
+- ArgoCD sync
+```
+
+### 3. **Health Checks**
+```
+🏥 Проверяет:
+- Rollout status (timeout: 300s)
+- Pod readiness (все поды Ready)
+- Image version (правильный tag)
+- Health endpoint (5 попыток)
+```
+
+### 4. **Auto Rollback** (при ошибке)
+```
+🔄 Если что-то пошло не так:
+- kubectl rollout undo
+- Revert Git commit
+- Restore previous state
+- Notify в logs
+```
+
+---
+
+## 📊 Pipeline Stages
+
+```
+┌─────────────────────────────┐
+│ 1. Save Current State       │ ← Сохраняет текущую версию
+└─────────────┬───────────────┘
+              ↓
+┌─────────────────────────────┐
+│ 2. Checkout Source          │
+└─────────────┬───────────────┘
+              ↓
+┌─────────────────────────────┐
+│ 3. Build Docker Image       │
+└─────────────┬───────────────┘
+              ↓
+┌─────────────────────────────┐
+│ 4. Push to Registry         │
+└─────────────┬───────────────┘
+              ↓
+┌─────────────────────────────┐
+│ 5. Update GitOps Manifests  │
+└─────────────┬───────────────┘
+              ↓
+┌─────────────────────────────┐
+│ 6. Wait for Deployment      │ ← 300s timeout
+└─────────────┬───────────────┘
+              ↓
+┌─────────────────────────────┐
+│ 7. Health Check             │ ← 5 retries
+└─────────────┬───────────────┘
+              ↓
+      ┌───────┴────────┐
+      ↓                ↓
+   SUCCESS          FAILURE
+      │                │
+      │                ↓
+      │         ┌──────────────┐
+      │         │   ROLLBACK   │ ← Автоматически!
+      │         └──────────────┘
+      ↓
+   ✅ DONE
+```
+
+---
+
+## 🔧 Configuration
+
+### Environment Variables:
+
+```groovy
+// Rollback configuration
+ROLLBACK_ENABLED = 'true'           // Включить/выключить rollback
+DEPLOYMENT_TIMEOUT = '300s'          // Timeout для rollout
+HEALTH_CHECK_RETRIES = '5'           // Количество попыток health check
+HEALTH_CHECK_DELAY = '10'            // Задержка между попытками (сек)
+```
+
+### Изменить настройки:
+
+```groovy
+environment {
+    ROLLBACK_ENABLED = 'false'       // Выключить rollback
+    DEPLOYMENT_TIMEOUT = '600s'       // Увеличить timeout
+    HEALTH_CHECK_RETRIES = '10'       // Больше попыток
+}
+```
+
+---
+
+## 🧪 Тестирование Rollback
+
+### Сценарий 1: Симуляция deployment failure
+
+Измени deployment.yaml чтобы вызвать ошибку:
+
+```yaml
+# apps/demo-nginx/deployment.yaml
+spec:
+  containers:
+  - name: nginx
+    image: nginx:nonexistent-tag  # Несуществующий tag
+```
+
+**Результат:**
+```
+❌ Deployment failed
+🔄 Rollback initiated automatically
+✅ Rolled back to previous version
+```
+
+---
+
+### Сценарий 2: Симуляция health check failure
+
+Измени nginx.conf чтобы сломать /health:
+
+```nginx
+location /health {
+    return 500 "broken";  # Вернет 500 error
+}
+```
+
+**Результат:**
+```
+❌ Health check failed after 5 attempts
+🔄 Rollback initiated automatically
+✅ Previous version restored
+```
+
+---
+
+### Сценарий 3: Симуляция timeout
+
+Установи очень короткий timeout:
+
+```groovy
+DEPLOYMENT_TIMEOUT = '10s'  // Слишком короткий
+```
+
+**Результат:**
+```
+❌ Deployment timeout exceeded
+🔄 Rollback initiated automatically
+✅ Rolled back successfully
+```
+
+---
+
+## 📋 Rollback Process Details
+
+### Что происходит при rollback:
+
+1. **Kubernetes Rollback:**
+   ```bash
+   kubectl rollout undo deployment/demo-nginx -n demo-app
+   ```
+
+2. **Git Revert:**
+   ```bash
+   git revert --no-edit HEAD
+   git push origin main
+   ```
+
+3. **ArgoCD Sync:**
+   ```
+   ArgoCD автоматически применит revert commit
+   ```
+
+4. **Verification:**
+   ```bash
+   kubectl rollout status deployment/demo-nginx -n demo-app
+   ```
+
+---
+
+## 🔍 Как проверить что rollback сработал
+
+### В Jenkins Console Output:
+
+```
+❌ DEPLOYMENT FAILED - INITIATING ROLLBACK!
+
+Rolling back to previous version...
+🔄 Rolling back to: docker.io/vladcrypto/demo-nginx:main-21
+
+✅ ROLLBACK COMPLETED!
+
+Rolled back to: docker.io/vladcrypto/demo-nginx:main-21
+Current build (#22) has been reverted.
+
+Please check logs and fix the issue before redeploying.
+```
+
+### В Kubernetes:
+
+```bash
+# Check deployment history
+kubectl rollout history deployment/demo-nginx -n demo-app
+
+# Вывод:
+REVISION  CHANGE-CAUSE
+21        Updated to main-21
+22        Updated to main-22
+23        Rollback to main-21  ← Rollback!
+```
+
+### В Git:
+
+```bash
+git log --oneline
+
+# Вывод:
+abc1234 Revert "chore(demo-nginx): Update image to main-22"
+def5678 chore(demo-nginx): Update image to main-22
+ghi9012 chore(demo-nginx): Update image to main-21
+```
+
+---
+
+## 💡 Best Practices
+
+### 1. **Всегда тестируй в staging**
+
+```groovy
+stage('Deploy to Staging') {
+    when { branch 'develop' }
+    steps {
+        // Deploy to staging namespace
+    }
+}
+```
+
+### 2. **Мониторинг после деплоя**
+
+```groovy
+stage('Post-Deploy Monitoring') {
+    steps {
+        sh """
+            # Monitor for 5 minutes
+            for i in {1..30}; do
+                kubectl top pods -n demo-app
+                sleep 10
+            done
+        """
+    }
+}
+```
+
+### 3. **Slack Notifications**
+
+```groovy
+post {
+    failure {
+        slackSend(
+            color: 'danger',
+            message: """
+                🚨 ROLLBACK EXECUTED!
+                Build: #${BUILD_NUMBER}
+                Rolled back to previous version
+            """
+        )
+    }
+}
+```
+
+### 4. **Сохранение artifacts**
+
+```groovy
+post {
+    always {
+        archiveArtifacts artifacts: '/tmp/previous_*.txt', allowEmptyArchive: true
+    }
+}
+```
+
+---
+
+## ⚠️ Important Notes
+
+### Rollback НЕ сработает если:
+
+1. **Нет предыдущей версии:**
+   ```
+   ⚠️ No previous version found - cannot rollback automatically
+   Manual intervention required!
+   ```
+
+2. **ROLLBACK_ENABLED = 'false':**
+   ```
+   ❌ Pipeline failed! (Rollback disabled)
+   ```
+
+3. **Не main branch:**
+   ```
+   Rollback only works on main branch
+   ```
+
+### Ручной rollback:
+
+Если автоматический rollback не сработал:
+
+```bash
+# Kubernetes rollback
+kubectl rollout undo deployment/demo-nginx -n demo-app
+
+# Git revert
+cd k3s-gitops
+git revert HEAD
+git push origin main
+
+# Force ArgoCD sync
+kubectl patch application demo-nginx -n argocd \
+  --type merge -p '{"operation":{"sync":{}}}'
+```
+
+---
+
+## 📊 Monitoring & Alerts
+
+### Grafana Dashboard
+
+Добавь панели для мониторинга rollbacks:
+
+```promql
+# Number of rollbacks
+sum(rate(deployment_rollback_total[5m])) by (deployment)
+
+# Rollback duration
+histogram_quantile(0.95, 
+  rate(deployment_rollback_duration_seconds_bucket[5m])
+)
+```
+
+### Alert Rules
+
+```yaml
+- alert: FrequentRollbacks
+  expr: rate(deployment_rollback_total[1h]) > 3
+  annotations:
+    summary: "Frequent rollbacks detected"
+    description: "More than 3 rollbacks in last hour"
+```
+
+---
+
+## 🎯 Advanced Features (Future)
+
+### 1. **Canary Deployments**
+
+```groovy
+stage('Canary Deploy') {
+    steps {
+        sh """
+            # Deploy 10% traffic to new version
+            kubectl set image deployment/${APP_NAME} 
+                ${APP_NAME}=${IMAGE_TAG} 
+                --record
+            kubectl scale deployment/${APP_NAME}-canary --replicas=1
+        """
+    }
+}
+```
+
+### 2. **Blue-Green Deployments**
+
+```groovy
+stage('Blue-Green Switch') {
+    steps {
+        sh """
+            # Switch service to new deployment
+            kubectl patch service ${APP_NAME} 
+                -p '{"spec":{"selector":{"version":"${IMAGE_TAG}"}}}'
+        """
+    }
+}
+```
+
+### 3. **Smoke Tests**
+
+```groovy
+stage('Smoke Tests') {
+    steps {
+        sh """
+            # Run automated tests
+            curl -f http://${APP_NAME}/api/health
+            curl -f http://${APP_NAME}/api/status
+        """
+    }
+}
+```
+
+---
+
+## ✅ Success Criteria
+
+Pipeline считается успешным когда:
+
+- ✅ Docker image built
+- ✅ Image pushed to registry
+- ✅ Git manifests updated
+- ✅ Deployment rolled out (300s timeout)
+- ✅ All pods ready
+- ✅ Image version matches
+- ✅ Health endpoint responds (5 retries)
+
+Pipeline откатывается если:
+
+- ❌ Deployment timeout
+- ❌ Pod not ready
+- ❌ Image version mismatch
+- ❌ Health check failed
+
+---
+
+## 🎉 Summary
+
+**Automatic Rollback добавляет:**
+
+✅ Безопасность деплоев  
+✅ Автоматическое восстановление  
+✅ Сохранение предыдущего состояния  
+✅ Git history revert  
+✅ Kubernetes rollback  
+✅ Health checks  
+✅ Timeout protection  
+
+**Zero manual intervention needed!** 🚀
+
+---
+
+## 📝 Testing Checklist
+
+- [ ] Normal deployment работает
+- [ ] Failed deployment triggers rollback
+- [ ] Previous version restored
+- [ ] Git commit reverted
+- [ ] Health checks work
+- [ ] Timeout works
+- [ ] Notifications sent
+- [ ] Logs clear and helpful
+
+---
+
+**Your pipeline is now production-ready with automatic rollback! 🎉**