diff --git a/apps/demo-nginx/docs/rollback.md b/apps/demo-nginx/docs/rollback.md new file mode 100644 index 0000000..bacc1b6 --- /dev/null +++ b/apps/demo-nginx/docs/rollback.md @@ -0,0 +1,471 @@ +# 🔄 Automatic Rollback Feature + +## ✅ Что добавлено + +Pipeline теперь автоматически откатывается к предыдущей версии при любой ошибке деплоя! + +--- + +## 🎯 Как работает + +### 1. **Save Current State** (перед деплоем) +``` +📸 Сохраняет: +- Текущий Docker image tag +- Количество реплик +- Git commit SHA +``` + +### 2. **Deploy New Version** +``` +🚀 Деплоит новую версию через: +- Build Docker image +- Push to registry +- Update Git manifests +- ArgoCD sync +``` + +### 3. **Health Checks** +``` +🏥 Проверяет: +- Rollout status (timeout: 300s) +- Pod readiness (все поды Ready) +- Image version (правильный tag) +- Health endpoint (5 попыток) +``` + +### 4. **Auto Rollback** (при ошибке) +``` +🔄 Если что-то пошло не так: +- kubectl rollout undo +- Revert Git commit +- Restore previous state +- Notify в logs +``` + +--- + +## 📊 Pipeline Stages + +``` +┌─────────────────────────────┐ +│ 1. Save Current State │ ← Сохраняет текущую версию +└─────────────┬───────────────┘ + ↓ +┌─────────────────────────────┐ +│ 2. Checkout Source │ +└─────────────┬───────────────┘ + ↓ +┌─────────────────────────────┐ +│ 3. Build Docker Image │ +└─────────────┬───────────────┘ + ↓ +┌─────────────────────────────┐ +│ 4. Push to Registry │ +└─────────────┬───────────────┘ + ↓ +┌─────────────────────────────┐ +│ 5. Update GitOps Manifests │ +└─────────────┬───────────────┘ + ↓ +┌─────────────────────────────┐ +│ 6. Wait for Deployment │ ← 300s timeout +└─────────────┬───────────────┘ + ↓ +┌─────────────────────────────┐ +│ 7. Health Check │ ← 5 retries +└─────────────┬───────────────┘ + ↓ + ┌───────┴────────┐ + ↓ ↓ + SUCCESS FAILURE + │ │ + │ ↓ + │ ┌──────────────┐ + │ │ ROLLBACK │ ← Автоматически! + │ └──────────────┘ + ↓ + ✅ DONE +``` + +--- + +## 🔧 Configuration + +### Environment Variables: + +```groovy +// Rollback configuration +ROLLBACK_ENABLED = 'true' // Включить/выключить rollback +DEPLOYMENT_TIMEOUT = '300s' // Timeout для rollout +HEALTH_CHECK_RETRIES = '5' // Количество попыток health check +HEALTH_CHECK_DELAY = '10' // Задержка между попытками (сек) +``` + +### Изменить настройки: + +```groovy +environment { + ROLLBACK_ENABLED = 'false' // Выключить rollback + DEPLOYMENT_TIMEOUT = '600s' // Увеличить timeout + HEALTH_CHECK_RETRIES = '10' // Больше попыток +} +``` + +--- + +## 🧪 Тестирование Rollback + +### Сценарий 1: Симуляция deployment failure + +Измени deployment.yaml чтобы вызвать ошибку: + +```yaml +# apps/demo-nginx/deployment.yaml +spec: + containers: + - name: nginx + image: nginx:nonexistent-tag # Несуществующий tag +``` + +**Результат:** +``` +❌ Deployment failed +🔄 Rollback initiated automatically +✅ Rolled back to previous version +``` + +--- + +### Сценарий 2: Симуляция health check failure + +Измени nginx.conf чтобы сломать /health: + +```nginx +location /health { + return 500 "broken"; # Вернет 500 error +} +``` + +**Результат:** +``` +❌ Health check failed after 5 attempts +🔄 Rollback initiated automatically +✅ Previous version restored +``` + +--- + +### Сценарий 3: Симуляция timeout + +Установи очень короткий timeout: + +```groovy +DEPLOYMENT_TIMEOUT = '10s' // Слишком короткий +``` + +**Результат:** +``` +❌ Deployment timeout exceeded +🔄 Rollback initiated automatically +✅ Rolled back successfully +``` + +--- + +## 📋 Rollback Process Details + +### Что происходит при rollback: + +1. **Kubernetes Rollback:** + ```bash + kubectl rollout undo deployment/demo-nginx -n demo-app + ``` + +2. **Git Revert:** + ```bash + git revert --no-edit HEAD + git push origin main + ``` + +3. **ArgoCD Sync:** + ``` + ArgoCD автоматически применит revert commit + ``` + +4. **Verification:** + ```bash + kubectl rollout status deployment/demo-nginx -n demo-app + ``` + +--- + +## 🔍 Как проверить что rollback сработал + +### В Jenkins Console Output: + +``` +❌ DEPLOYMENT FAILED - INITIATING ROLLBACK! + +Rolling back to previous version... +🔄 Rolling back to: docker.io/vladcrypto/demo-nginx:main-21 + +✅ ROLLBACK COMPLETED! + +Rolled back to: docker.io/vladcrypto/demo-nginx:main-21 +Current build (#22) has been reverted. + +Please check logs and fix the issue before redeploying. +``` + +### В Kubernetes: + +```bash +# Check deployment history +kubectl rollout history deployment/demo-nginx -n demo-app + +# Вывод: +REVISION CHANGE-CAUSE +21 Updated to main-21 +22 Updated to main-22 +23 Rollback to main-21 ← Rollback! +``` + +### В Git: + +```bash +git log --oneline + +# Вывод: +abc1234 Revert "chore(demo-nginx): Update image to main-22" +def5678 chore(demo-nginx): Update image to main-22 +ghi9012 chore(demo-nginx): Update image to main-21 +``` + +--- + +## 💡 Best Practices + +### 1. **Всегда тестируй в staging** + +```groovy +stage('Deploy to Staging') { + when { branch 'develop' } + steps { + // Deploy to staging namespace + } +} +``` + +### 2. **Мониторинг после деплоя** + +```groovy +stage('Post-Deploy Monitoring') { + steps { + sh """ + # Monitor for 5 minutes + for i in {1..30}; do + kubectl top pods -n demo-app + sleep 10 + done + """ + } +} +``` + +### 3. **Slack Notifications** + +```groovy +post { + failure { + slackSend( + color: 'danger', + message: """ + 🚨 ROLLBACK EXECUTED! + Build: #${BUILD_NUMBER} + Rolled back to previous version + """ + ) + } +} +``` + +### 4. **Сохранение artifacts** + +```groovy +post { + always { + archiveArtifacts artifacts: '/tmp/previous_*.txt', allowEmptyArchive: true + } +} +``` + +--- + +## ⚠️ Important Notes + +### Rollback НЕ сработает если: + +1. **Нет предыдущей версии:** + ``` + ⚠️ No previous version found - cannot rollback automatically + Manual intervention required! + ``` + +2. **ROLLBACK_ENABLED = 'false':** + ``` + ❌ Pipeline failed! (Rollback disabled) + ``` + +3. **Не main branch:** + ``` + Rollback only works on main branch + ``` + +### Ручной rollback: + +Если автоматический rollback не сработал: + +```bash +# Kubernetes rollback +kubectl rollout undo deployment/demo-nginx -n demo-app + +# Git revert +cd k3s-gitops +git revert HEAD +git push origin main + +# Force ArgoCD sync +kubectl patch application demo-nginx -n argocd \ + --type merge -p '{"operation":{"sync":{}}}' +``` + +--- + +## 📊 Monitoring & Alerts + +### Grafana Dashboard + +Добавь панели для мониторинга rollbacks: + +```promql +# Number of rollbacks +sum(rate(deployment_rollback_total[5m])) by (deployment) + +# Rollback duration +histogram_quantile(0.95, + rate(deployment_rollback_duration_seconds_bucket[5m]) +) +``` + +### Alert Rules + +```yaml +- alert: FrequentRollbacks + expr: rate(deployment_rollback_total[1h]) > 3 + annotations: + summary: "Frequent rollbacks detected" + description: "More than 3 rollbacks in last hour" +``` + +--- + +## 🎯 Advanced Features (Future) + +### 1. **Canary Deployments** + +```groovy +stage('Canary Deploy') { + steps { + sh """ + # Deploy 10% traffic to new version + kubectl set image deployment/${APP_NAME} + ${APP_NAME}=${IMAGE_TAG} + --record + kubectl scale deployment/${APP_NAME}-canary --replicas=1 + """ + } +} +``` + +### 2. **Blue-Green Deployments** + +```groovy +stage('Blue-Green Switch') { + steps { + sh """ + # Switch service to new deployment + kubectl patch service ${APP_NAME} + -p '{"spec":{"selector":{"version":"${IMAGE_TAG}"}}}' + """ + } +} +``` + +### 3. **Smoke Tests** + +```groovy +stage('Smoke Tests') { + steps { + sh """ + # Run automated tests + curl -f http://${APP_NAME}/api/health + curl -f http://${APP_NAME}/api/status + """ + } +} +``` + +--- + +## ✅ Success Criteria + +Pipeline считается успешным когда: + +- ✅ Docker image built +- ✅ Image pushed to registry +- ✅ Git manifests updated +- ✅ Deployment rolled out (300s timeout) +- ✅ All pods ready +- ✅ Image version matches +- ✅ Health endpoint responds (5 retries) + +Pipeline откатывается если: + +- ❌ Deployment timeout +- ❌ Pod not ready +- ❌ Image version mismatch +- ❌ Health check failed + +--- + +## 🎉 Summary + +**Automatic Rollback добавляет:** + +✅ Безопасность деплоев +✅ Автоматическое восстановление +✅ Сохранение предыдущего состояния +✅ Git history revert +✅ Kubernetes rollback +✅ Health checks +✅ Timeout protection + +**Zero manual intervention needed!** 🚀 + +--- + +## 📝 Testing Checklist + +- [ ] Normal deployment работает +- [ ] Failed deployment triggers rollback +- [ ] Previous version restored +- [ ] Git commit reverted +- [ ] Health checks work +- [ ] Timeout works +- [ ] Notifications sent +- [ ] Logs clear and helpful + +--- + +**Your pipeline is now production-ready with automatic rollback! 🎉** \ No newline at end of file