DevOps?

由于花旗杯的契机,YDJSIR开始认真研究Jenkins。虽然说也不能研究得多么深入,但总归是有些尝试。下面的脚本实际上还不算完整的CICD流程,只是YDJSIR在南京大学软件学院《软件工程与计算Ⅲ》中使用的配置,仅供参考。

环境说明

该容器运行在一台腾讯轻量云服务器(上海,4C4G8M)的Docker容器中,通过Agent的方式,控制另一台腾讯轻量服务器(上海,4C4G8M)进行代码拉取、自动构建打包测试与发布等工作。

Jenkins通过南大Git上的镜像仓库获得WebHook推送++代码,相关操作均按照文档进行。前/后端和Python服务在生产环境中均运行在容器内。目前设置上以master分支为生产环境,仅该分支的推送会触发构建。构建状态会用GitLab Connection推回给南大Git。日常开发主分支是develop

部署图如下。

image-20220401202845923

仓库列表和说明如下。

课程仓库 南大Git镜像 备注
http://172.29.4.49/MXYZyyds/backend-mxyzyyds https://git.nju.edu.cn/YDJSIR/191250186_mxyzyyds_backend_mxyzyyds 后端项目
http://172.29.4.49/MXYZyyds/frontend-mxyzyyds https://git.nju.edu.cn/YDJSIR/191250186_mxyzyyds_frontend_mxyzyyds 前端项目
http://172.29.4.49/MXYZyyds/python-mxyzyyds https://git.nju.edu.cn/YDJSIR/191250186_mxyzyyds_python_mxyzyyds/ Python服务项目

搭建过程

此部分从略。总的来说,步骤如下:

1、安装基础环境、拉取镜像并配置镜像仓库;

2、启动Jenkins Docker,并确保它可以SSH走RSA密钥登录业务逻辑服务器;

3、配置业务逻辑服务器使其可以用SSH从南大Git拉取镜像;

4、编写流水线脚本、配置WebHook、GitLab访问令牌和GitLab Connection等,逐步调试使得整套CICD流程完善,并在日常开发中使用。

部署说明

触发方式

通过WebHook触发

通过主动发送或在master分支有新的提交以发起push event类型的WebHook以触发构建。

image-20220330193722422

主动在Jenkins上点击触发构建

登录到Jenkins后台后,进入Jenkins的流水线页面,点击Build Now手动触发构建。

image-20220330193606894

image-20220304112124935

目前在Java后端仓库已经能实现JUnit测试结果和基于JaCoCo的测试覆盖率报告收集并能够在Jenkins网页上展示。

此外,构建的结果已实测可以推回南大Git。image-20220304112045381

注意事项

前端

前端构建完基本是马上就可以用了,但强烈建议用隐私模式或者是Ctrl+F5强制刷新查看网页。虽然已改用cnpm,但是打包速度仍不是十分稳定,可能存在一定波动。

image-20220330213026354

后端

后端打包不是在容器里做的,只是部署在容器里,因而可以有效地利用cache,速度不错,基本都能在1分钟内完成(服务器升级4C4G后)

image-20220330213058741

Python

由于需要加载预训练模型,网络不一定稳定(要么快得秒过要么会莫名其妙卡死,大概20%概率?),所以构建成功后启动(docker后台启动不影响执行流)需要的时间波动大。直到四个模型文件加载完后Python服务才能正常启动已设置如果十分钟加载不完或者失败,gunicorn会重启worker来加载这四个文件。如有需要,可以多次尝试。

image-20220330213155071

pip源方面,docker构建时已换源。

1
2
3
4
5
6
7
8
9
10
11
[2022-03-30 11:28:53 +0000] [7] [INFO] Starting gunicorn 20.1.0
[2022-03-30 11:28:53 +0000] [7] [DEBUG] Arbiter booted
[2022-03-30 11:28:53 +0000] [7] [INFO] Listening at: http://0.0.0.0:5000 (7)
[2022-03-30 11:28:53 +0000] [7] [INFO] Using worker: sync
[2022-03-30 11:28:53 +0000] [8] [INFO] Booting worker with pid: 8
[2022-03-30 11:28:53 +0000] [7] [DEBUG] 1 workers
Downloading: 100%|██████████| 319/319 [00:00<00:00, 383kB/s]
Downloading: 100%|██████████| 107k/107k [00:01<00:00, 73.7kB/s]
Downloading: 100%|██████████| 112/112 [00:00<00:00, 133kB/s]
Downloading: 100%|██████████| 856/856 [00:00<00:00, 1.04MB/s]
Downloading: 100%|██████████| 390M/390M [02:51<00:00, 2.38MB/s]

部署脚本

目前Jenkinsfile改成放代码仓库里面了。还是代码即流水线好。

后端(Java Maven项目)流水线脚本

改版前

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
pipeline { 
agent any
stages {
stage('SCM from Mirror') {
steps {
git url: 'git@git.nju.edu.cn:YDJSIR/191250186_mxyzyyds_backend_mxyzyyds.git'
sh "ls -al"
}
}
stage('Build') {
steps {
sh "chmod +x mvnw"
sh "./mvnw install"
junit 'target/surefire-reports/*.xml'
step( [ $class: 'JacocoPublisher' ] )
}
post {
failure {
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success {
updateGitlabCommitStatus name:'build', state: 'success'
}
}
}
stage('Launch') {
steps {
sh "/usr/local/shell/start_collect_backend.sh"
}
}
}
}

脚本中引用的/usr/local/shell/start_collect_backend.sh内容如下。

1
2
3
4
5
6
7
8
9
cd /home/webroot/workspace/COLLECT-Backend
ls -al
docker stop collect-backend
docker rm collect-backend
docker system prune -f
docker build . -t collect-backend --no-cache
docker run -d --name collect-backend -p 8888:8888 --restart unless-stopped collect-backend:latest /bin/bash -c "java -jar target/collect*.jar; tail -f /dev/null"
exit
exit

Dockerfile如下:

1
2
FROM openjdk:8 as production-stage
COPY ./target/ /target/

改版后

只有Jenkinsfile改变,Dockerfile不变。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
pipeline {
agent {
label 'XY'
}
stages {
stage('SCM from Mirror') {
steps {
git url: 'git@git.nju.edu.cn:YDJSIR/191250186_mxyzyyds_backend_mxyzyyds.git'
sh "ls -al"
sh "pwd"
}
}
stage('Compile') {
steps {
sh "chmod +x mvnw"
sh "./mvnw install -Dmaven.test.skip=true"
}
post {
failure {
updateGitlabCommitStatus name: 'compile', state: 'failed'
}
success {
updateGitlabCommitStatus name:'compile', state: 'success'
}
}
}
stage('Test') {
steps {
sh "./mvnw test"
junit 'target/surefire-reports/*.xml'
step( [ $class: 'JacocoPublisher' ] )
}
post {
failure {
updateGitlabCommitStatus name: 'test', state: 'failed'
}
success {
updateGitlabCommitStatus name:'test', state: 'success'
}
}
}
stage('Build') {
steps {
sh "docker stop collect-backend | true"
sh "docker rm collect-backend | true"
sh "docker build . -t collect-backend --no-cache"
}
post {
failure {
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success {
updateGitlabCommitStatus name:'build', state: 'success'
}
}
}
stage('Start') {
steps {
sh 'docker run -d --name collect-backend -p 8888:8888 --restart unless-stopped collect-backend:latest /bin/bash -c "java -jar target/collect*.jar; tail -f /dev/null"'
sh "docker ps -a | grep collect-backend"
}
post {
failure {
updateGitlabCommitStatus name: 'start', state: 'failed'
}
success {
updateGitlabCommitStatus name:'start', state: 'success'
}
}
}
}
}

前端(基于Node.js的Vue项目)流水线脚本

改版前

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
pipeline { 
agent any
stages {
stage('SCM from Mirror') {
steps {
git url: 'git@git.nju.edu.cn:YDJSIR/191250186_mxyzyyds_frontend_mxyzyyds.git'
sh "ls -al"
sh "pwd"
}
}
stage('Build') {
steps {
sh "/usr/local/shell/start_collect_frontend.sh"
}
post {
failure {
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success {
updateGitlabCommitStatus name:'build', state: 'success'
}
}
}
stage('Post') {
steps {
sh "docker ps -a | grep collect-frontend"
}
}
}
}

脚本中引用的/usr/local/shell/start_collect_frontend.sh内容如下。

1
2
3
4
5
6
7
8
9
10
cd /home/webroot/workspace/COLLECT-Frontend
ls -al
pwd
docker stop collect-frontend
docker rm collect-frontend
docker build --no-cache . -t collect-frontend
docker run -d --name collect-frontend --restart unless-stopped -p 80:80 -p 443:443 collect-frontend:latest /bin/bash -c "nginx; tail -f /dev/null"
exit
exit

使用的Dockerfile如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
FROM node:14.18-stretch as build-stage
WORKDIR /app
COPY package*.json ./
RUN npm install -g cnpm --registry=https://registry.npm.taobao.org
RUN cnpm install
COPY ./ .
RUN cnpm run build

FROM nginx as production-stage
RUN mkdir /app
COPY --from=build-stage /app/dist /app
COPY nginx.conf /etc/nginx/nginx.conf
COPY cert/se3.ydjsir.com.cn.pem /etc/nginx/se3.ydjsir.com.cn.pem
COPY cert/se3.ydjsir.com.cn.key /etc/nginx/se3.ydjsir.com.cn.key

使用的nginx.conf如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
user  nginx;
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
keepalive_timeout 65;
client_max_body_size 100m;
server {
listen 80;
listen 443 ssl;
server_name se3.ydjsir.com.cn;
ssl_certificate /etc/nginx/se3.ydjsir.com.cn.pem;
ssl_certificate_key /etc/nginx/se3.ydjsir.com.cn.key;
if ($server_port !~ 443){
rewrite ^(/.*)$ https://$host$1 permanent;
}

location / {
root /app;
index index.html;
try_files $uri $uri/ /index.html;
}

# This is the proxy to backend, obviously the address will be modified accordingly after backend is also in the docker
location ^~ /api/ {
proxy_pass http://172.17.0.1:8888/;
}

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root /usr/share/nginx/html;
}
}
}

改版后

只有Jenkinsfile变了,Dockerfilenginx.conf不变。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
pipeline {
agent {
label 'XY'
}
stages {
stage('SCM from Mirror') {
steps {
git url: 'git@git.nju.edu.cn:YDJSIR/191250186_mxyzyyds_frontend_mxyzyyds.git'
sh "ls -al"
sh "pwd"
}
}
stage('Build') {
steps {
sh "docker stop collect-frontend | true"
sh "docker rm collect-frontend | true"
sh "docker build --no-cache . -t collect-frontend"

}
post {
failure {
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success {
updateGitlabCommitStatus name:'build', state: 'success'
}
}
}
stage('Start') {
steps {
sh 'docker run -d --name collect-frontend --restart unless-stopped -p 80:80 -p 443:443 collect-frontend:latest /bin/bash -c "nginx; tail -f /dev/null"'
sh "docker ps -a | grep collect-frontend"
}
post {
failure {
updateGitlabCommitStatus name: 'start', state: 'failed'
}
success {
updateGitlabCommitStatus name:'start', state: 'success'
}
}
}
}
}

Python服务(flask+gunicorn项目)流水线脚本

改版前

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
pipeline { 
agent any
stages {
stage('SCM from Mirror') {
steps {
git url: 'git@git.nju.edu.cn:YDJSIR/191250186_mxyzyyds_python_mxyzyyds.git'
sh "ls -al"
}
}
stage('Build') {
steps {
sh "docker stop sim"
sh "docker build -t se3python ."
}
post {
failure {
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success {
updateGitlabCommitStatus name:'build', state: 'success'
}
}
}
stage('Launch') {
steps {
sh "docker run --rm -d -p 5000:5000 --name sim se3python"
sh "docker system prune -f"
}
}
}
}

使用的Dockerfile如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
FROM python:3.9

WORKDIR /usr/src/app

COPY requirements.txt ./
RUN pip install -i https://mirrors.cloud.tencent.com/pypi/simple --no-cache-dir -r requirements.txt
RUN pip install -i https://mirrors.cloud.tencent.com/pypi/simple torch

COPY . .
RUN python3 setup.py install

CMD gunicorn -b 0.0.0.0:5000 app:app --timeout 600 --log-level debug
#CMD flask run -p 5000 --host=0.0.0.0

改版后

只有Jenkinsfile变了,Dockerfile不变。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
pipeline {
agent {
label 'XY'
}
stages {
stage('SCM from Mirror') {
steps {
git url: 'git@git.nju.edu.cn:YDJSIR/191250186_mxyzyyds_python_mxyzyyds.git'
sh "ls -al"
}
}
stage('Build') {
steps {
sh "docker stop sim | true"
sh "docker build -t se3python ."
}
post {
failure {
updateGitlabCommitStatus name: 'build', state: 'failed'
}
success {
updateGitlabCommitStatus name:'build', state: 'success'
}
}
}
stage('Start') {
steps {
sh "docker run --rm -d -p 5000:5000 --name sim se3python"
sh "docker system prune -f"
}
post {
failure {
updateGitlabCommitStatus name: 'start', state: 'failed'
}
success {
updateGitlabCommitStatus name:'start', state: 'success'
}
}
}
}
}