471 lines
11 KiB
Markdown
471 lines
11 KiB
Markdown
# 🔄 Automatic Rollback Feature
|
||
|
||
## ✅ Что добавлено
|
||
|
||
Pipeline теперь автоматически откатывается к предыдущей версии при любой ошибке деплоя!
|
||
|
||
---
|
||
|
||
## 🎯 Как работает
|
||
|
||
### 1. **Save Current State** (перед деплоем)
|
||
```
|
||
📸 Сохраняет:
|
||
- Текущий Docker image tag
|
||
- Количество реплик
|
||
- Git commit SHA
|
||
```
|
||
|
||
### 2. **Deploy New Version**
|
||
```
|
||
🚀 Деплоит новую версию через:
|
||
- Build Docker image
|
||
- Push to registry
|
||
- Update Git manifests
|
||
- ArgoCD sync
|
||
```
|
||
|
||
### 3. **Health Checks**
|
||
```
|
||
🏥 Проверяет:
|
||
- Rollout status (timeout: 300s)
|
||
- Pod readiness (все поды Ready)
|
||
- Image version (правильный tag)
|
||
- Health endpoint (5 попыток)
|
||
```
|
||
|
||
### 4. **Auto Rollback** (при ошибке)
|
||
```
|
||
🔄 Если что-то пошло не так:
|
||
- kubectl rollout undo
|
||
- Revert Git commit
|
||
- Restore previous state
|
||
- Notify в logs
|
||
```
|
||
|
||
---
|
||
|
||
## 📊 Pipeline Stages
|
||
|
||
```
|
||
┌─────────────────────────────┐
|
||
│ 1. Save Current State │ ← Сохраняет текущую версию
|
||
└─────────────┬───────────────┘
|
||
↓
|
||
┌─────────────────────────────┐
|
||
│ 2. Checkout Source │
|
||
└─────────────┬───────────────┘
|
||
↓
|
||
┌─────────────────────────────┐
|
||
│ 3. Build Docker Image │
|
||
└─────────────┬───────────────┘
|
||
↓
|
||
┌─────────────────────────────┐
|
||
│ 4. Push to Registry │
|
||
└─────────────┬───────────────┘
|
||
↓
|
||
┌─────────────────────────────┐
|
||
│ 5. Update GitOps Manifests │
|
||
└─────────────┬───────────────┘
|
||
↓
|
||
┌─────────────────────────────┐
|
||
│ 6. Wait for Deployment │ ← 300s timeout
|
||
└─────────────┬───────────────┘
|
||
↓
|
||
┌─────────────────────────────┐
|
||
│ 7. Health Check │ ← 5 retries
|
||
└─────────────┬───────────────┘
|
||
↓
|
||
┌───────┴────────┐
|
||
↓ ↓
|
||
SUCCESS FAILURE
|
||
│ │
|
||
│ ↓
|
||
│ ┌──────────────┐
|
||
│ │ ROLLBACK │ ← Автоматически!
|
||
│ └──────────────┘
|
||
↓
|
||
✅ DONE
|
||
```
|
||
|
||
---
|
||
|
||
## 🔧 Configuration
|
||
|
||
### Environment Variables:
|
||
|
||
```groovy
|
||
// Rollback configuration
|
||
ROLLBACK_ENABLED = 'true' // Включить/выключить rollback
|
||
DEPLOYMENT_TIMEOUT = '300s' // Timeout для rollout
|
||
HEALTH_CHECK_RETRIES = '5' // Количество попыток health check
|
||
HEALTH_CHECK_DELAY = '10' // Задержка между попытками (сек)
|
||
```
|
||
|
||
### Изменить настройки:
|
||
|
||
```groovy
|
||
environment {
|
||
ROLLBACK_ENABLED = 'false' // Выключить rollback
|
||
DEPLOYMENT_TIMEOUT = '600s' // Увеличить timeout
|
||
HEALTH_CHECK_RETRIES = '10' // Больше попыток
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## 🧪 Тестирование Rollback
|
||
|
||
### Сценарий 1: Симуляция deployment failure
|
||
|
||
Измени deployment.yaml чтобы вызвать ошибку:
|
||
|
||
```yaml
|
||
# apps/demo-nginx/deployment.yaml
|
||
spec:
|
||
containers:
|
||
- name: nginx
|
||
image: nginx:nonexistent-tag # Несуществующий tag
|
||
```
|
||
|
||
**Результат:**
|
||
```
|
||
❌ Deployment failed
|
||
🔄 Rollback initiated automatically
|
||
✅ Rolled back to previous version
|
||
```
|
||
|
||
---
|
||
|
||
### Сценарий 2: Симуляция health check failure
|
||
|
||
Измени nginx.conf чтобы сломать /health:
|
||
|
||
```nginx
|
||
location /health {
|
||
return 500 "broken"; # Вернет 500 error
|
||
}
|
||
```
|
||
|
||
**Результат:**
|
||
```
|
||
❌ Health check failed after 5 attempts
|
||
🔄 Rollback initiated automatically
|
||
✅ Previous version restored
|
||
```
|
||
|
||
---
|
||
|
||
### Сценарий 3: Симуляция timeout
|
||
|
||
Установи очень короткий timeout:
|
||
|
||
```groovy
|
||
DEPLOYMENT_TIMEOUT = '10s' // Слишком короткий
|
||
```
|
||
|
||
**Результат:**
|
||
```
|
||
❌ Deployment timeout exceeded
|
||
🔄 Rollback initiated automatically
|
||
✅ Rolled back successfully
|
||
```
|
||
|
||
---
|
||
|
||
## 📋 Rollback Process Details
|
||
|
||
### Что происходит при rollback:
|
||
|
||
1. **Kubernetes Rollback:**
|
||
```bash
|
||
kubectl rollout undo deployment/demo-nginx -n demo-app
|
||
```
|
||
|
||
2. **Git Revert:**
|
||
```bash
|
||
git revert --no-edit HEAD
|
||
git push origin main
|
||
```
|
||
|
||
3. **ArgoCD Sync:**
|
||
```
|
||
ArgoCD автоматически применит revert commit
|
||
```
|
||
|
||
4. **Verification:**
|
||
```bash
|
||
kubectl rollout status deployment/demo-nginx -n demo-app
|
||
```
|
||
|
||
---
|
||
|
||
## 🔍 Как проверить что rollback сработал
|
||
|
||
### В Jenkins Console Output:
|
||
|
||
```
|
||
❌ DEPLOYMENT FAILED - INITIATING ROLLBACK!
|
||
|
||
Rolling back to previous version...
|
||
🔄 Rolling back to: docker.io/vladcrypto/demo-nginx:main-21
|
||
|
||
✅ ROLLBACK COMPLETED!
|
||
|
||
Rolled back to: docker.io/vladcrypto/demo-nginx:main-21
|
||
Current build (#22) has been reverted.
|
||
|
||
Please check logs and fix the issue before redeploying.
|
||
```
|
||
|
||
### В Kubernetes:
|
||
|
||
```bash
|
||
# Check deployment history
|
||
kubectl rollout history deployment/demo-nginx -n demo-app
|
||
|
||
# Вывод:
|
||
REVISION CHANGE-CAUSE
|
||
21 Updated to main-21
|
||
22 Updated to main-22
|
||
23 Rollback to main-21 ← Rollback!
|
||
```
|
||
|
||
### В Git:
|
||
|
||
```bash
|
||
git log --oneline
|
||
|
||
# Вывод:
|
||
abc1234 Revert "chore(demo-nginx): Update image to main-22"
|
||
def5678 chore(demo-nginx): Update image to main-22
|
||
ghi9012 chore(demo-nginx): Update image to main-21
|
||
```
|
||
|
||
---
|
||
|
||
## 💡 Best Practices
|
||
|
||
### 1. **Всегда тестируй в staging**
|
||
|
||
```groovy
|
||
stage('Deploy to Staging') {
|
||
when { branch 'develop' }
|
||
steps {
|
||
// Deploy to staging namespace
|
||
}
|
||
}
|
||
```
|
||
|
||
### 2. **Мониторинг после деплоя**
|
||
|
||
```groovy
|
||
stage('Post-Deploy Monitoring') {
|
||
steps {
|
||
sh """
|
||
# Monitor for 5 minutes
|
||
for i in {1..30}; do
|
||
kubectl top pods -n demo-app
|
||
sleep 10
|
||
done
|
||
"""
|
||
}
|
||
}
|
||
```
|
||
|
||
### 3. **Slack Notifications**
|
||
|
||
```groovy
|
||
post {
|
||
failure {
|
||
slackSend(
|
||
color: 'danger',
|
||
message: """
|
||
🚨 ROLLBACK EXECUTED!
|
||
Build: #${BUILD_NUMBER}
|
||
Rolled back to previous version
|
||
"""
|
||
)
|
||
}
|
||
}
|
||
```
|
||
|
||
### 4. **Сохранение artifacts**
|
||
|
||
```groovy
|
||
post {
|
||
always {
|
||
archiveArtifacts artifacts: '/tmp/previous_*.txt', allowEmptyArchive: true
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## ⚠️ Important Notes
|
||
|
||
### Rollback НЕ сработает если:
|
||
|
||
1. **Нет предыдущей версии:**
|
||
```
|
||
⚠️ No previous version found - cannot rollback automatically
|
||
Manual intervention required!
|
||
```
|
||
|
||
2. **ROLLBACK_ENABLED = 'false':**
|
||
```
|
||
❌ Pipeline failed! (Rollback disabled)
|
||
```
|
||
|
||
3. **Не main branch:**
|
||
```
|
||
Rollback only works on main branch
|
||
```
|
||
|
||
### Ручной rollback:
|
||
|
||
Если автоматический rollback не сработал:
|
||
|
||
```bash
|
||
# Kubernetes rollback
|
||
kubectl rollout undo deployment/demo-nginx -n demo-app
|
||
|
||
# Git revert
|
||
cd k3s-gitops
|
||
git revert HEAD
|
||
git push origin main
|
||
|
||
# Force ArgoCD sync
|
||
kubectl patch application demo-nginx -n argocd \
|
||
--type merge -p '{"operation":{"sync":{}}}'
|
||
```
|
||
|
||
---
|
||
|
||
## 📊 Monitoring & Alerts
|
||
|
||
### Grafana Dashboard
|
||
|
||
Добавь панели для мониторинга rollbacks:
|
||
|
||
```promql
|
||
# Number of rollbacks
|
||
sum(rate(deployment_rollback_total[5m])) by (deployment)
|
||
|
||
# Rollback duration
|
||
histogram_quantile(0.95,
|
||
rate(deployment_rollback_duration_seconds_bucket[5m])
|
||
)
|
||
```
|
||
|
||
### Alert Rules
|
||
|
||
```yaml
|
||
- alert: FrequentRollbacks
|
||
expr: rate(deployment_rollback_total[1h]) > 3
|
||
annotations:
|
||
summary: "Frequent rollbacks detected"
|
||
description: "More than 3 rollbacks in last hour"
|
||
```
|
||
|
||
---
|
||
|
||
## 🎯 Advanced Features (Future)
|
||
|
||
### 1. **Canary Deployments**
|
||
|
||
```groovy
|
||
stage('Canary Deploy') {
|
||
steps {
|
||
sh """
|
||
# Deploy 10% traffic to new version
|
||
kubectl set image deployment/${APP_NAME}
|
||
${APP_NAME}=${IMAGE_TAG}
|
||
--record
|
||
kubectl scale deployment/${APP_NAME}-canary --replicas=1
|
||
"""
|
||
}
|
||
}
|
||
```
|
||
|
||
### 2. **Blue-Green Deployments**
|
||
|
||
```groovy
|
||
stage('Blue-Green Switch') {
|
||
steps {
|
||
sh """
|
||
# Switch service to new deployment
|
||
kubectl patch service ${APP_NAME}
|
||
-p '{"spec":{"selector":{"version":"${IMAGE_TAG}"}}}'
|
||
"""
|
||
}
|
||
}
|
||
```
|
||
|
||
### 3. **Smoke Tests**
|
||
|
||
```groovy
|
||
stage('Smoke Tests') {
|
||
steps {
|
||
sh """
|
||
# Run automated tests
|
||
curl -f http://${APP_NAME}/api/health
|
||
curl -f http://${APP_NAME}/api/status
|
||
"""
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## ✅ Success Criteria
|
||
|
||
Pipeline считается успешным когда:
|
||
|
||
- ✅ Docker image built
|
||
- ✅ Image pushed to registry
|
||
- ✅ Git manifests updated
|
||
- ✅ Deployment rolled out (300s timeout)
|
||
- ✅ All pods ready
|
||
- ✅ Image version matches
|
||
- ✅ Health endpoint responds (5 retries)
|
||
|
||
Pipeline откатывается если:
|
||
|
||
- ❌ Deployment timeout
|
||
- ❌ Pod not ready
|
||
- ❌ Image version mismatch
|
||
- ❌ Health check failed
|
||
|
||
---
|
||
|
||
## 🎉 Summary
|
||
|
||
**Automatic Rollback добавляет:**
|
||
|
||
✅ Безопасность деплоев
|
||
✅ Автоматическое восстановление
|
||
✅ Сохранение предыдущего состояния
|
||
✅ Git history revert
|
||
✅ Kubernetes rollback
|
||
✅ Health checks
|
||
✅ Timeout protection
|
||
|
||
**Zero manual intervention needed!** 🚀
|
||
|
||
---
|
||
|
||
## 📝 Testing Checklist
|
||
|
||
- [ ] Normal deployment работает
|
||
- [ ] Failed deployment triggers rollback
|
||
- [ ] Previous version restored
|
||
- [ ] Git commit reverted
|
||
- [ ] Health checks work
|
||
- [ ] Timeout works
|
||
- [ ] Notifications sent
|
||
- [ ] Logs clear and helpful
|
||
|
||
---
|
||
|
||
**Your pipeline is now production-ready with automatic rollback! 🎉** |