Rancher + K3s + Harbor + GitLab + Jenkins CI/CD 实战部署笔记

本文完整记录一次 DevOps CI/CD 平台搭建全过程

整体规划

本实验共使用 3 台服务器

主机 IP 角色 系统 配置
node1 192.168.1.121 GitLab + Jenkins Debian12 2C8G
node2 192.168.1.122 Harbor 私有镜像仓库 Debian12 2C8G
node3 192.168.1.123 Rancher + K3s Kubernetes Debian12 2C8G

基础环境安装

每台主机都需要做

1
2
3
4
5
6
7
8
9
apt update
apt upgrade -y
# qemu-guest-agent
apt install -y qemu-guest-agent
systemctl enable qemu-guest-agent
systemctl start qemu-guest-agent

# tool
apt install -y curl wget vim sudo git net-tools docker.io tar

123主机

K3S安装

1
2
3
4
5
6
7
8
9
10
11
12
# 关闭 swap（K8S必须）
sudo swapoff -a
sudo sed -i '/ swap / s/^/#/' /etc/fstab
# 安装K3S
curl -sfL https://get.k3s.io | sh -
# 检查安装结果
kubectl get nodes
# 开放 kubeconfig （为了让 Jenkins 远程部署）
sudo chmod 644 /etc/rancher/k3s/k3s.yaml
# 创建一个测试Pod
kubectl run nginx --image=nginx
kubectl get pods

Rancher安装

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# /opt/rancher/docker-compose.yml
services:
  rancher:
    image: rancher/rancher:v2.7.9
    container_name: rancher
    restart: unless-stopped
    privileged: true
    ports:
      - "8088:80"
      - "8443:443"
    volumes:
      - rancher-data:/var/lib/rancher

volumes:
  rancher-data:

Rancher 配置

点击 “☰” → Cluster Management → Create Cluster

选择 “Import”（导入已有集群）

输入集群名称，比如：cicd-demo-cluster

点击 Create，Rancher 会生成一个 命令行脚本。

切换到 123 主机（已经安装 K3s 的主机）。
在终端执行 Rancher 提供的 Import 命令，一般是类似下面的：
kubectl apply -f https://<rancher-ip>/v3/import/<token>.yaml

Rancher 把自签证书加到系统信任

1
2
3
sudo cat /etc/rancher/k3s/k3s.yaml | grep certificate-authority-data
echo "<base64证书>" | base64 -d > /usr/local/share/ca-certificates/k3s-ca.crt
sudo update-ca-certificates

kubeconfig配置

1
2
3
4
5
# 123主机执行
cat /etc/rancher/k3s/k3s.yaml
cp /etc/rancher/k3s/k3s.yaml ~/kubeconfig.yaml
sz ~/kubeconfig.yaml
# 修改 server: https://192.168.1.123:6443

配置Jenkins中的Kube凭证kubeconfig凭证

122主机

Harbor

1
2
3
4
5
6
mkdir /opt/harbor
cd /opt/harbor
wget https://github.com/goharbor/harbor/releases/download/v2.10.0/harbor-offline-installer-v2.10.0.tgz
tar -xzf harbor-offline-installer-v2.10.0.tgz
cd harbor
cp harbor.yml.tmpl harbor.yml

修改harbor中的几个关键配置项

1
2
3
4
5
6
hostname: 192.168.1.122
http:
  port: 8080
harbor_admin_password: Harbor12345
data_volume: /data/harbor
# 注释掉https配置或者添加证书

运行./install.sh 自动安装

Harbor成功后配置与测试

创建镜像仓库项目

Name: cicd-demo
Access: Public

测试Harbor可用性

1
2
# 切换到121主机上执行（登录并输入Harbor账号密码）
docker login 192.168.1.122:8080

配置其他主机daemon

1
2
3
4
5
# /etc/docker/daemon.json
{
  "insecure-registries": ["192.168.1.122:8080"]
}
systemctl restart docker

121主机

GitLab

1
2
3
# 创建安装目录
mkdir -p /opt/gitlab
mkdir -p /opt/jenkins
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# /opt/gitlab/docker-compose.yml
services:
  gitlab:
    image: gitlab/gitlab-ce:16.2.1-ce.0
    container_name: gitlab
    restart: unless-stopped
    hostname: 192.168.1.121
    ports:
      - "80:80"
      - "443:443"
      - "2222:22"
    volumes:
      - gitlab-config:/etc/gitlab
      - gitlab-logs:/var/log/gitlab
      - gitlab-data:/var/opt/gitlab
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'http://192.168.1.121:80'
        gitlab_rails['gitlab_shell_ssh_port'] = 2222

volumes:
  gitlab-config:
  gitlab-logs:
  gitlab-data:

GitLab 创建 Access Token

User Avatar → Preferences → Access Tokens
Name： jenkins
Expiration：随意
Scopes： 勾选read_repository
将生成的Token设置到Jenkins里Jenkins凭证

Jenkins

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#  /opt/jenkins/docker-compose.yml 
services:
  jenkins:
    image: jenkins/jenkins:lts
    container_name: jenkins
    user: root
    privileged: true
    restart: unless-stopped
    ports:
      - "8080:8080"
      - "50000:50000"
    volumes:
      - jenkins-data:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro
      - /etc/timezone:/etc/timezone:ro

volumes:
  jenkins-data:

Jenkins插件安装

搜不到说明是自带的，不需要额外安装

  • Docker Pipeline
  • GitLab Plugin
  • Pipeline
  • Credentials Binding
  • Kubernetes CLI Plugin

Jenkins添加凭证

Manage Jenkins → Credentials → System → Global credentials → Add Credentials

Harbor凭证

Type: Username with password
Username: admin
Password: Harbor12345
Id: harbor-admin

Jenkins凭证

Type: Username with password
Username: root
Password: Gitlab生成的Access Token
Id: gitlab-token

kubeconfig凭证

Type: Secret file
Select File: kubeconfig.yamlkubeconfig配置

访问测试

  • 查看Jenkins默认密码(默认用户 admin) docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
  • 查看Gitlab默认密码(默认用户root) docker exec gitlab cat /etc/gitlab/initial_root_password
  • Jenkins 访问地址 http://192.168.1.121:8080

Gitlab 准备测试仓库

创建仓库cicd-demo

1
2
3
4
5
cicd-demo
 ├ Dockerfile
 ├ index.html
 └ k8s
     └ deployment.yaml
1
2
3
# Dockerfile
FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
deployment.yaml
# ==============================
# Deployment：用于管理 Pod 副本、滚动更新、自动恢复
# ==============================

apiVersion: apps/v1              # 使用的 Kubernetes API 版本
kind: Deployment                 # 资源类型：Deployment（部署控制器）

metadata:
  name: cicd-demo                # Deployment 名称
  namespace: default             # 所属命名空间（默认 default）
  labels:
    app: cicd-demo               # 标签，用于 Service 选择 Pod

spec:

  replicas: 2                    # Pod 副本数量（同时运行 2 个 Pod，提高可用性）
  selector:                      # Deployment 用来识别和管理 Pod 的选择器
    matchLabels:
      app: cicd-demo             # 只管理 label=app:cicd-demo 的 Pod
  template:                      # Pod 模板（Deployment 会按照这个模板创建 Pod）
    metadata:
      labels:
        app: cicd-demo           # Pod 的标签，必须与 selector.matchLabels 一致

    spec:
      imagePullSecrets:
        - name: harbor-secret  
      containers:
        - name: cicd-demo-container    # 容器名称（Pod 内唯一）
          image: 192.168.1.122:8080/cicd-demo/app:latest
          # 容器镜像地址
          # 格式：registry/project/image:tag
          # 这里表示从 Harbor 私有仓库拉取镜像
          imagePullPolicy: Always
          # 镜像拉取策略
          # Always：每次启动都重新拉取镜像（CI/CD 推荐）
          # IfNotPresent：本地存在则不拉取
          # Never：永远不拉取
          ports:
            - containerPort: 80
              # 容器内部监听端口
              # 这里 nginx 监听 80
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
              # Pod 最小资源需求
              # 100m CPU = 0.1 个 CPU
              # 128Mi = 128MB 内存
            limits:
              cpu: "500m"
              memory: "512Mi"
              # Pod 最大资源限制
              # 防止某个容器占满服务器资源
          # ==============================
          # 存活检测（Liveness Probe）
          # ==============================
          livenessProbe:
            httpGet:
              path: /           # 检测 URL 路径
              port: 80          # 检测端口
            initialDelaySeconds: 30
            # 容器启动后等待 30 秒再开始检测
            periodSeconds: 10
            # 每 10 秒执行一次检测
            timeoutSeconds: 2
            # HTTP 请求超时时间
            failureThreshold: 3
            # 连续失败 3 次后 Kubernetes 会重启容器
          # ==============================
          # 就绪检测（Readiness Probe）
          # ==============================
          readinessProbe:
            httpGet:
              path: /           # 检测 URL
              port: 80
            initialDelaySeconds: 5
            # 容器启动 5 秒后开始检测
            periodSeconds: 5
            # 每 5 秒检测一次
            timeoutSeconds: 2
            failureThreshold: 3
            # 连续失败 3 次则 Pod 不接收流量
            # Service 会自动把流量转发到其它 Pod

---
# ==============================
# Service：用于暴露 Pod 网络访问
# ==============================

apiVersion: v1
kind: Service                   # 资源类型：Service
metadata:
  name: cicd-demo-service      # Service 名称
  namespace: default

spec:
  type: NodePort               # 服务类型
  # ClusterIP  → 仅集群内部访问
  # NodePort   → 通过 NodeIP:Port 访问
  # LoadBalancer → 云厂商负载均衡

  selector:
    app: cicd-demo             # 匹配 label=app:cicd-demo 的 Pod

  ports:
    - name: http
      port: 80
      # Service 内部端口
      # 集群内部访问使用
      targetPort: 80
      # 转发到 Pod 的端口
      nodePort: 30080
      # 节点对外暴露端口
      # 访问地址：
      # http://192.168.1.123:30080
1
<h1>DevOps CI/CD Demo</h1>

Jenkins中添加Pipeline

Script
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
pipeline {
    agent any
    environment {
        REGISTRY = "192.168.1.122:8080"
        PROJECT = "cicd-demo"
        IMAGE = "app"
        TAG = "${BUILD_NUMBER}" // 使用 Jenkins Build Number 作为镜像 TAG
    }

    stages {

        stage('Checkout') {
            steps {
                git branch: 'main',
                credentialsId: 'gitlab-token',
                url: 'http://192.168.1.121/root/cicd-demo.git'
            }
        }

        stage('Build Image') {
            steps {
                sh '''
                echo "Building Docker image: $REGISTRY/$PROJECT/$IMAGE:$TAG"
                docker build -t $REGISTRY/$PROJECT/$IMAGE:$TAG .
                '''
            }
        }

        stage('Push Image') {
            steps {
                withCredentials([usernamePassword(
                    credentialsId: 'harbor-admin',
                    usernameVariable: 'USERNAME',
                    passwordVariable: 'PASSWORD'
                )]) {
                    sh '''
                    echo "Logging into Harbor"
                    docker login $REGISTRY -u $USERNAME -p $PASSWORD
                    echo "Pushing Docker image: $REGISTRY/$PROJECT/$IMAGE:$TAG"
                    docker push $REGISTRY/$PROJECT/$IMAGE:$TAG
                    '''
                }
            }
        }

        stage('Deploy to K3s') {
            steps {
                withCredentials([file(credentialsId: 'kubeconfig', variable: 'KUBECONFIG')]) {
                    sh '''
                    echo "Updating Deployment with new image tag: $TAG"
                    kubectl set image deployment/cicd-demo cicd-demo-container=$REGISTRY/$PROJECT/$IMAGE:$TAG --record
                    '''
                }
            }
        }
    }
}