优化构建
This commit is contained in:
@@ -8,6 +8,14 @@ services:
|
||||
TZ: Asia/Shanghai
|
||||
volumes:
|
||||
- mysql_data:/var/lib/mysql
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 512M
|
||||
cpus: '1.0'
|
||||
reservations:
|
||||
memory: 256M
|
||||
cpus: '0.5'
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "mysqladmin ping -h 127.0.0.1 -uroot -p\"$${MYSQL_ROOT_PASSWORD}\""]
|
||||
interval: 10s
|
||||
@@ -19,6 +27,14 @@ services:
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 256M
|
||||
cpus: '0.5'
|
||||
reservations:
|
||||
memory: 128M
|
||||
cpus: '0.25'
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 10s
|
||||
@@ -35,6 +51,14 @@ services:
|
||||
TZ: Asia/Shanghai
|
||||
volumes:
|
||||
- minio_data:/data
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 512M
|
||||
cpus: '0.5'
|
||||
reservations:
|
||||
memory: 256M
|
||||
cpus: '0.25'
|
||||
healthcheck:
|
||||
test: ["CMD", "mc", "ready", "local"]
|
||||
interval: 10s
|
||||
@@ -49,6 +73,14 @@ services:
|
||||
- ../backend/.env
|
||||
volumes:
|
||||
- ../backend/logs:/app/logs
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 512M
|
||||
cpus: '1.0'
|
||||
reservations:
|
||||
memory: 256M
|
||||
cpus: '0.5'
|
||||
depends_on:
|
||||
mysql:
|
||||
condition: service_healthy
|
||||
@@ -64,6 +96,14 @@ services:
|
||||
context: ..
|
||||
dockerfile: frontend/Dockerfile
|
||||
restart: unless-stopped
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
memory: 256M
|
||||
cpus: '0.5'
|
||||
reservations:
|
||||
memory: 128M
|
||||
cpus: '0.25'
|
||||
depends_on:
|
||||
- backend
|
||||
ports:
|
||||
|
||||
@@ -0,0 +1,193 @@
|
||||
# 部署故障排查指南
|
||||
|
||||
## SSH 连接断开问题
|
||||
|
||||
### 问题现象
|
||||
执行 `scripts/deploy-prod.sh` 后:
|
||||
- SSH 连接立即断开
|
||||
- 无法重新连接服务器
|
||||
- 服务器可能完全无响应
|
||||
|
||||
### 根本原因
|
||||
**Docker 构建过程耗尽服务器内存,导致系统 OOM (Out of Memory) 崩溃**
|
||||
|
||||
#### 技术细节
|
||||
1. **同时构建多个镜像**:原脚本使用 `docker compose up -d --build` 会并行构建 backend 和 frontend
|
||||
2. **前端构建超高内存消耗**:Node.js/npm 构建过程可能消耗 1-2GB 内存
|
||||
3. **无资源限制**:所有容器没有 memory limit,可无限制消耗系统资源
|
||||
4. **连锁反应**:
|
||||
- Docker 构建吃满内存
|
||||
- Linux OOM Killer 开始杀进程
|
||||
- SSH daemon 被杀死 → 连接断开
|
||||
- 系统核心服务被杀 → 无法重连
|
||||
- 最严重时整个系统死机
|
||||
|
||||
### 立即恢复方法
|
||||
|
||||
#### 方法 1:物理/远程控制台重启
|
||||
```bash
|
||||
# 通过云服务商控制台重启服务器
|
||||
# 重启后立即禁用 Docker 自动启动
|
||||
sudo systemctl disable docker
|
||||
sudo systemctl stop docker
|
||||
|
||||
# 清理所有容器
|
||||
cd /path/to/hfb_sys
|
||||
docker compose -f deploy/docker-compose.prod.yml down
|
||||
```
|
||||
|
||||
#### 方法 2:强制重启前快速执行
|
||||
如果还能短暂连接,立即执行:
|
||||
```bash
|
||||
sudo systemctl stop docker
|
||||
sudo killall -9 dockerd containerd
|
||||
```
|
||||
|
||||
### 根本解决方案(已修复)
|
||||
|
||||
#### 1. 添加资源限制(docker-compose.prod.yml)
|
||||
所有服务现在都有明确的内存和 CPU 限制:
|
||||
- MySQL: 512M memory, 1 CPU
|
||||
- Redis: 256M memory, 0.5 CPU
|
||||
- MinIO: 512M memory, 0.5 CPU
|
||||
- Backend: 512M memory, 1 CPU
|
||||
- Frontend: 256M memory, 0.5 CPU
|
||||
|
||||
总计:~2GB 内存(适合 4GB 服务器)
|
||||
|
||||
#### 2. 串行构建(deploy-prod.sh)
|
||||
修改后的脚本:
|
||||
```bash
|
||||
# 先构建 backend
|
||||
compose build backend
|
||||
|
||||
# 再构建 frontend
|
||||
compose build frontend
|
||||
|
||||
# 最后启动(不再构建)
|
||||
compose up -d --no-build
|
||||
```
|
||||
|
||||
这样可以避免内存峰值,但会增加 1-2 分钟部署时间。
|
||||
|
||||
### 安全部署流程
|
||||
|
||||
#### 首次部署
|
||||
```bash
|
||||
# 1. 在本地或有充足资源的机器上预构建镜像
|
||||
docker compose -f deploy/docker-compose.prod.yml build
|
||||
|
||||
# 2. 保存镜像
|
||||
docker save -o backend.tar <image_name>:latest
|
||||
docker save -o frontend.tar <image_name>:latest
|
||||
|
||||
# 3. 上传到服务器
|
||||
scp *.tar server:/tmp/
|
||||
|
||||
# 4. 在服务器上加载
|
||||
ssh server
|
||||
docker load -i /tmp/backend.tar
|
||||
docker load -i /tmp/frontend.tar
|
||||
|
||||
# 5. 使用 --no-build 启动
|
||||
./scripts/deploy-prod.sh --no-build
|
||||
```
|
||||
|
||||
#### 日常更新(推荐)
|
||||
```bash
|
||||
# 直接使用修复后的脚本
|
||||
./scripts/deploy-prod.sh
|
||||
|
||||
# 或者只更新不重新构建
|
||||
./scripts/deploy-prod.sh --no-build
|
||||
```
|
||||
|
||||
### 监控服务器资源
|
||||
|
||||
#### 部署前检查
|
||||
```bash
|
||||
# 查看可用内存
|
||||
free -h
|
||||
|
||||
# 查看 CPU 负载
|
||||
uptime
|
||||
|
||||
# 建议:至少保留 1GB 可用内存
|
||||
```
|
||||
|
||||
#### 部署中监控
|
||||
```bash
|
||||
# 另一个终端实时监控
|
||||
watch -n 1 'free -h && docker stats --no-stream'
|
||||
```
|
||||
|
||||
### 服务器配置建议
|
||||
|
||||
#### 最低配置
|
||||
- 内存:4GB
|
||||
- CPU:2 核
|
||||
- 磁盘:20GB
|
||||
|
||||
#### 推荐配置
|
||||
- 内存:8GB
|
||||
- CPU:4 核
|
||||
- 磁盘:50GB
|
||||
- 启用 Swap:4GB
|
||||
|
||||
#### 启用 Swap(紧急缓冲)
|
||||
```bash
|
||||
# 创建 4GB swap
|
||||
sudo fallocate -l 4G /swapfile
|
||||
sudo chmod 600 /swapfile
|
||||
sudo mkswap /swapfile
|
||||
sudo swapon /swapfile
|
||||
|
||||
# 永久启用
|
||||
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
|
||||
```
|
||||
|
||||
### 其他可能原因
|
||||
|
||||
#### 1. 网络超时
|
||||
```bash
|
||||
# 增加 SSH 保活
|
||||
# 编辑 ~/.ssh/config
|
||||
Host your-server
|
||||
ServerAliveInterval 60
|
||||
ServerAliveCountMax 3
|
||||
```
|
||||
|
||||
#### 2. Docker daemon 崩溃
|
||||
```bash
|
||||
# 查看 Docker 日志
|
||||
sudo journalctl -u docker -n 100 --no-pager
|
||||
|
||||
# 重启 Docker
|
||||
sudo systemctl restart docker
|
||||
```
|
||||
|
||||
#### 3. 磁盘满
|
||||
```bash
|
||||
# 检查磁盘空间
|
||||
df -h
|
||||
|
||||
# 清理 Docker 垃圾
|
||||
docker system prune -a --volumes -f
|
||||
```
|
||||
|
||||
## 预防措施清单
|
||||
|
||||
- [x] 所有服务添加 memory/CPU 限制
|
||||
- [x] 串行构建避免内存峰值
|
||||
- [ ] 启用服务器 Swap
|
||||
- [ ] 部署前检查可用内存
|
||||
- [ ] 设置 SSH 保活
|
||||
- [ ] 定期清理 Docker 磁盘
|
||||
- [ ] 监控部署过程资源使用
|
||||
|
||||
## 紧急联系
|
||||
|
||||
如果服务器完全无响应:
|
||||
1. 联系云服务商客服强制重启
|
||||
2. 通过 VNC/控制台登录
|
||||
3. 考虑升级服务器配置
|
||||
@@ -316,8 +316,14 @@ main() {
|
||||
log "启动生产服务..."
|
||||
compose up -d
|
||||
else
|
||||
log "构建并启动生产服务..."
|
||||
compose up -d --build
|
||||
log "构建后端镜像..."
|
||||
compose build backend
|
||||
|
||||
log "构建前端镜像..."
|
||||
compose build frontend
|
||||
|
||||
log "启动生产服务..."
|
||||
compose up -d --no-build
|
||||
fi
|
||||
|
||||
wait_service_healthy mysql "MySQL"
|
||||
|
||||
Reference in New Issue
Block a user