环境
Ubuntu 22.04 LTS
Docker 27.2.1
Jenkins 2.462.2
描述
本文未讲如何构建 Gitlab,Harbor,Docker Swarm,Jenkins 等应用。
本文讲述 Jenkins 对接 Gitlab Harbor 使用 Pipeline Jenkinsfile 持续构建部署 uwsgi + django + sqlite3 轻小型 CMS 中后台应用至 Docker Swarm。
完成如下配置后,将实现当 Gitlab 中创建新标签时,将会自动进行,代码推送,镜像构建,镜像推送私有库,通知运行主机拉取镜像并发布。
步骤
目录结构
以下是一个完整 Django Project 目录。
空目录使用 .gitkeep(空文件) 占位使得 Gitlab 中保持目录结构。
项目
—App目录
—data # 存放数据库 sqlite3
——.gitkeep
—项目目录 # 含有 Django setting.py 的目录
——settings.py
—docker # Docker 相关文件
——Dockerfile-uwsgi
——Entrypoint-uwsgi.sh
——stack-compose.yml
—script # 脚本目录
——deploy-swarm.sh
—static # 静态文件目录
——.gitkeep
.gitignore
Jenkinsfile-swarm
manage.py
requirements.txt
uwsgi.ini
图示
配置 Jenkins
安装插件
Pipeline Stage View Plugin
GitLab Plugin
Publish Over SSH
Pipeline
Credentials Plugin
创建并配置 Pipeline
New Item -> Pipeline -> 设置Jenkins 项目名称,如:django01
django01 -> Configure
Build Triggers:勾选Build when a change is pushed to GitLab.
并记录 Webhook 值,勾选子选项 Push Events
Pipeline:Pipeline script from SCM
- Repository URL:填写项目 URL,如:http://192.168.120.53:18001/t01/demo03.git
- Script Path:Jenkinsfile-swarm
创建凭证
Manage Jenkins -> Credentials -> Domain 下点击 global -> Add Credentials -> 添加 Harbor 用户帐号密码等信息:Kind:Username with password,帐号、密码、ID和描述。
配置 SSH Server
Manage Jenkins -> System
- 搜索 SSH Server,配置 Docker Swarm Node 节点信息,点开 Advanced 可设置帐号密码登录或密钥登录,Remote Directory 创建工作目录并授权用户访问,配置完成后,点击右下角 Test Configuration 测试。
- 搜索 Gitlab,取消勾选:Enable authentication for '/project' end-point
配置 Gitlab
Admin -> 设置 -> 网络 -> 出站请求 Outbound requests -> 一:勾选'允许系统钩子向本地网络发送请求',二(不执行也可):Webhook 和服务可以访问的本地 IP 地址和域名:Jenkins服务器IP地址。
项目 -> 设置 -> Webhook
- URL:前面记录的 Webhook 值
- 触发来源:标签推送事件
- 取消 SSL 验证。
相关文件
Dockerfile-uwsgi
注意修改 DJANGO_SUPERUSER_PASSWORD
,若不修改则默认密码为 admin
。
注意修改 Docker 加速echo "Acquire::http::Proxy \"http://192.168.120.2:10809\";" > /etc/apt/apt.conf.d/90proxy && \
;若无需 Docker 加速,请删除该行。
FROM python:3.10.15-slim-bookworm
LABEL org.opencontainers.image.authors="Shanks.Lee" \
org.opencontainers.image.blog="https://www.yudelei.com"
ENV DJANGO_SUPERUSER_PASSWORD="admin" DJANGO_SUPERUSER_EMAIL="root@localhost"
WORKDIR /usr/src/app
VOLUME /usr/src/app/data
COPY . .
RUN echo "Acquire::http::Proxy \"http://192.168.120.2:10809\";" > /etc/apt/apt.conf.d/90proxy && \
mv docker/Entrypoint-uwsgi.sh ./Entrypoint.sh && \
chmod +x ./Entrypoint.sh && apt update && apt install gcc -y && \
pip install --no-cache-dir -r requirements.txt && \
apt remove gcc -y && apt autoremove -y && \
rm -rf /var/cache/apt/archives /var/lib/apt/lists/* /etc/apt/apt.conf.d/90proxy
EXPOSE 8000
ENTRYPOINT ["./Entrypoint.sh"]
Entrypoint-uwsgi.sh
#!/bin/bash
echo "Flush the manage.py command it any"
while ! python manage.py flush --no-input 2>&1; do
echo "Flusing django manage command"
sleep 3
done
echo "Migrate the Database at startup of project"
# Wait for few minute and run db migraiton
while ! python manage.py migrate 2>&1; do
echo "Migration is in progress status"
sleep 3
done
yes yes | python manage.py collectstatic
if ! env |grep "DJANGO_SUPERUSER_PASSWORD" ; then
export DJANGO_SUPERUSER_PASSWORD=admin
fi
if ! env |grep "DJANGO_SUPERUSER_EMAIL" ; then
email=$DJANGO_SUPERUSER_EMAIL
else
email="root@localhost"
fi
check_do=$(echo "from django.contrib.auth.models import User; \
superuser_exists = User.objects.filter(is_superuser=True).exists(); \
print(superuser_exists)" | python manage.py shell)
if [ $check_do != "True" ] ; then
python manage.py createsuperuser --username admin --noinput
fi
unset DJANGO_SUPERUSER_PASSWORD
unset DJANGO_SUPERUSER_EMAIL
uwsgi --ini uwsgi.ini
deploy-swarm.sh
若主机端口被占用,会自动变更,若不需要,请注释下方:自动变更主机映射端口口
若本地镜像冲突(以项目名称判断),会删除已有镜像,若不需要,请注释下方:清理冲突镜像
将该文件放置到每台主机的/usr/bin/
目录下,添加执行权限chmod +x /usr/bin/deploy-swarm.sh
。
harbor_addr=$1
harbor_repo=$2
proj=$3
ver=$4
local_path=$5
harbor_user=$6
harbor_password=$7
port_zj=${8:-17002}
port_rq=${9:-8000}
mkdir -p $local_path
deploy_image=$harbor_addr/$harbor_repo/$proj:$ver
# 自动变更主机映射端口口
while nc -z -w2 127.0.0.1 $port_zj ; do
let port_zj++
sleep 0.1
done
export DEPLOY_IMAGE=$deploy_image
export DEPLOY_PORT_PUBLISH=$port_zj
export DEPLOY_DATA=$local_path
# 清理冲突镜像
old_images=`docker images |grep ${proj} |awk '{print $2}'`
if [[ "old_images_ids" =~ "$ver" ]] ; then
docker rmi -f $image_name
fi
docker login -u $harbor_user -p $harbor_password $harbor_addr
docker stack deploy --with-registry-auth -c stack-compose.yml ${proj}-stack2
stack-compose.yml
注意192.168.120.53:80/repo/demo03:v0.0.16
为默认私有仓库中的镜像或其他,-17002
为默认发布端口;
变量实际值由 Jenkins Pipeline 的 Jenkinfile 文件中设置,并传入 stack-compose.yml。
若 compose.yml 不需要默认值,则删除:-"192.168.120.53:80/repo/demo03:v0.0.16"
与:-17002
。volumes
为 Node 主机t01-ubuntu
本机目录,则需绑定运行t01-ubuntu
(注意修改为实际 node 名称);
理论上建议使用共享卷;中小型应用本地目录与共享目录均可满足性能要求;若为共享目录,则可注释块placement:
。
共享目录不可为 smb 或 nfs,会有资源锁 Lock,会导致 Sqlite 无法使用。
version: "3.8"
services:
django1:
image: ${DEPLOY_IMAGE:-"192.168.120.53:80/repo/demo03:v0.0.16"}
ports:
- published: ${DEPLOY_PORT_PUBLISH:-17002}
target: 8000
volumes:
- type: bind
source: ${DEPLOY_DATA}
target: /usr/src/app/data
deploy:
replicas: 1
placement:
constraints:
- node.hostname == t01-ubuntu
restart_policy:
condition: on-failure
max_attempts: 3
networks:
- net73
networks:
net73:
driver: overlay
Jenkinsfile-swarm
变量均为在全局 environment 中,请根据后方注释修改,并删除注释。
若更改了目录结构,有绝对路径的地方调整,请仔细检查。
pipeline {
agent any
environment {
harbor_addr = '192.168.120.53:80' # 私有仓库地址
harbor_repo = 'repo' # 私有仓库项目名称
proj_run_host = 't01-ubuntu' # 运行主机
proj_run_host_path = '/data/django/' # 将会挂载到容器中的目录
proj_host_port = 17002 # 主机端口
proj_container_port = 8000 # 容器端口
proj_name = '' # 由参数提取设置,若手动设置,请删除"参数提取"stage
proj_tag = '' # 由参数提取设置,若手动设置,请删除"参数提取"stage
tmp = '' # 临时变量
}
stages {
stage('参数提取') {
steps {
script {
proj_name = env.gitlabSourceRepoName
echo "$proj_name"
tmp = env.gitlabBranch
def tmp_tag = tmp.replace("refs/tags/", "")
proj_tag = tmp_tag
echo "$proj_tag"
}
}
}
stage('代码拉取') {
steps {
checkout scmGit(branches: [[name: "$proj_tag"]], extensions: [],
userRemoteConfigs: [[url: "$GIT_URL"]])
}
}
stage('镜像构建') {
environment {
HARBOR = credentials("$jks_harbor_user_password_credentials_id")
}
steps {
sh "docker build -t $proj_name:$proj_tag -f docker/Dockerfile-uwsgi . && \
docker login -u " + '$HARBOR_USR' + " -p " + '$HARBOR_PSW' + " $harbor_addr && \
docker tag $proj_name:$proj_tag $harbor_addr/$harbor_repo/$proj_name:$proj_tag && \
docker push $harbor_addr/$harbor_repo/$proj_name:$proj_tag"
}
}
stage('推送文件') {
steps {
sshPublisher(publishers: [sshPublisherDesc(configName: "$proj_run_host", #
transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: '',
execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false,
patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false,
removePrefix: 'docker/', sourceFiles: 'docker/stack-compose.yml')],
usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
}
}
stage('容器发布') {
environment {
HARBOR = credentials("$jks_harbor_user_password_credentials_id")
}
steps {
sshPublisher(publishers: [sshPublisherDesc(configName: "$proj_run_host", transfers:
[sshTransfer(cleanRemote: false, excludes: '',
execCommand: "cd $proj_work_dir && \
deploy-swarm.sh $harbor_addr $harbor_repo $proj_name $proj_tag $local_path $HARBOR_USR" + \
'$HARBOR_PSW' + " $proj_host_port $proj_container_port",
execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+',
remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '')],
usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
}
}
}
}
requirements.txt
若不需要 pip 加速,请删除行-i https://pypi.tuna.tsinghua.edu.cn/simple
。
-i https://pypi.tuna.tsinghua.edu.cn/simple
Django==4.2.16
shortuuid==1.0.13
django-simpleui==2024.8.28
uwsgi==2.0.27
uwsgi.ini
[uwsgi]
http-socket = 0.0.0.0:8000
chdir = /usr/src/app
wsgi-file = %(chdir)/项目名称/wsgi.py # 注意修改
static-map = /static=%(chdir)/static
module = serious_django.wsgi
master = True
processes = 2
enable-threads=true
threads = 2
workers=4
harakiri=30
vacuum = true
max-requests = 5000
settings.py
注意修改。
...
DEBUG = False
...
INSTALLED_APPS = [
'simpleui',
'django.contrib.admin',
'django.contrib.auth',
'django.contrib.contenttypes',
'django.contrib.sessions',
'django.contrib.messages',
'django.contrib.staticfiles',
'app'
]
...
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.sqlite3',
'NAME': BASE_DIR / 'data/db.sqlite3',
}
}
...
LANGUAGE_CODE = 'zh-hans'
TIME_ZONE = 'Asia/Shanghai'
USE_I18N = True
USE_TZ = True
...
STATIC_URL = 'static/'
STATIC_ROOT = os.path.join(BASE_DIR, "static")
...
# 以下非必要
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'
SIMPLEUI_DEFAULT_ICON = False
SIMPLEUI_LOGIN_PARTICLES = False
# SIMPLEUI_HOME_PAGE = '/'
SIMPLEUI_HOME_TITLE = 'Django01'
SIMPLEUI_INDEX = '/'
SIMPLEUI_HOME_INFO = False
SIMPLEUI_HOME_QUICK = True
SIMPLEUI_ANALYSIS = False
SIMPLEUI_STATIC_OFFLINE = True