Jenkins-Argocd CICD Rollouts金丝雀发布优化
# 一、Argo Rollouts金丝雀发布
# 介绍
- 这里是依赖前面的
Kubernetes + Jenkins + Gillab + Argocd的环境
# 功能
Argocd触发CD速度Argocd Rollouts金丝雀发布- 代码仓库打
Tag
# 二、Argocd触发CD速度
Argo CD每三分钟轮询一次Git存储库,检测清单的变化;消除轮询带来的延迟,将API服务器配置为接收Webhook事件,Argo CD支持Github、GitLab、Bitbucket、Bitbucket Server和Gogs的Git Webhook.官方参考文档: https://argo-cd.readthedocs.io/en/stable/operator-manual/webhook/#2-configure-argo-cd-with-the-webhook-secret-optional
# Argocd Secret创建
# 查看`Argocd`登录密码
$ kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d
FMgfgp1RLZ2HRLbz
# 创建`Argocd`的Secret
$ echo -n 'admin' | base64
YWRtaW4=
$ echo -n 'FMgfgp1RLZ2HRLbz' | base64
Rk1nZmdwMVJMWjJIUkxieg==
$ cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: Secret
metadata:
name: coolops
namespace: argocd
data:
username: YWRtaW4=
password: Rk1nZmdwMVJMWjJIUkxieg==
EOF
$ kubectl get secrets -n argocd
NAME TYPE DATA AGE
......
argocd-secret Opaque 6 31h
argocd-server-token-djvdg kubernetes.io/service-account-token 3 31h
argocd-tls kubernetes.io/tls 2 30h
coolops Opaque 2 13s
# 将创建好的`Secret`添加到`Argocd`的configmap中
$ kubectl edit secret argocd-secret -n argocd
apiVersion: v1
data:
......
webhook.gitlab.secret: coolops
kind: Secret
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
......
labels:
app.kubernetes.io/name: argocd-secret
app.kubernetes.io/part-of: argocd
name: argocd-secret
namespace: argocd
type: Opaque
$ kubectl describe secret argocd-secret -n argocd
Name: argocd-secret
Namespace: argocd
Labels: app.kubernetes.io/name=argocd-secret
app.kubernetes.io/part-of=argocd
Annotations: <none>
Type: Opaque
Data
====
admin.password: 60 bytes
admin.passwordMtime: 20 bytes
server.secretkey: 44 bytes
tls.crt: 1237 bytes
tls.key: 1679 bytes
webhook.gitlab.secret: 7 bytes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
# Gitla WebHook配置
项目仓库名: devops-cd
网址: https://argocddemo.chsaos.com/api/webhook
Secret 令牌: coolops
- 如果不是使用正式域名进行的部署的话;
Argocd集群内部证书是无效证书,要把Enabled SSL去掉
# 查看argocd server日志信息
$ kubectl logs -f argocd-server-5b8c45c484-828zw -n argocd
1
2
2
# 三、Argo Rollouts金丝雀发布配置
# 1.Kubernetes集群中部署argo-rollouts
官方文档:
# 指定本地文件安装
$ cd /home/kubernetes-software-yaml/argo-rollouts-canary
$ curl -OL https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/install.yaml
$ kubectl create namespace argo-rollouts
$ kubectl apply -n argo-rollouts -f install.yaml
# 指定在线安装文件
$ cd /home/kubernetes-software-yaml/argo-rollouts-canary
$ kubectl create namespace argo-rollouts
$ kubectl apply -n argo-rollouts -f https://raw.githubusercontent.com/argoproj/argo-rollouts/stable/manifests/install.yaml
# 查看服务状态
$ kubectl get pods,svc -n argo-rollouts Sat Nov 19 19:22:32 2022
NAME READY STATUS RESTARTS AGE
pod/argo-rollouts-84995876f-bf7s6 1/1 Running 0 21s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/argo-rollouts-metrics ClusterIP 10.99.222.41 <none> 8090/TCP 3h49m
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
# 2.安装argo-rollouts插件
$ curl -LO https://github.com/argoproj/argo-rollouts/releases/download/v1.3.1/kubectl-argo-rollouts-linux-amd64
$ chmod +x ./kubectl-argo-rollouts-linux-amd64
$ mv ./kubectl-argo-rollouts-linux-amd64 /usr/bin/kubectl-argo-rollouts
# `kubectl-argo-rollouts`命令使用
$ kubectl-argo-rollouts get rollout rollouts-demo
$ kubectl argo rollouts get rollout rollouts-demo --watch
1
2
3
4
5
6
7
8
2
3
4
5
6
7
8
3.重写deployment文件
主要配置清单如下:
rollout.yaml、services.yaml、ingress.yaml、kustomization.yaml;将编排文件保存到如下项目devops-cd/rollout-simple-java路径中
# 创建命名空间
$ kubectl create namespace myapp-dev
# 创建拉取镜像使用的`Secret`
$ kubectl create secret docker-registry gitlab-registry --docker-server=harbordemo.chsaos.com --docker-username=admin --docker-password=Harbor12345 --docker-email=admin@admin.com --namespace myapp-dev
# `host IP`版本`YAML`
$ vim rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollouts-simple-java
namespace: myapp-dev
spec:
replicas: 3
strategy:
canary:
canaryService: rollouts-simple-java-canary
stableService: rollouts-simple-java-stable
trafficRouting:
nginx:
stableIngress: rollouts-simple-java-stable
steps:
- setWeight: 20
- pause: {duration: 60}
- setWeight: 50
- pause: {duration: 10}
- setWeight: 80
- pause: {duration: 10}
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollouts-simple-java
template:
metadata:
labels:
app: rollouts-simple-java
spec:
containers:
- args:
- -jar
- /opt/myapp.jar
- --server.port=8080
command:
- java
env:
- name: HOST_IP
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: status.hostIP
image: registry.cn-hangzhou.aliyuncs.com/rookieops/myapp:latest
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- /bin/sleep 30
livenessProbe:
failureThreshold: 3
httpGet:
path: /hello
port: 8080
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 1
name: myapp
ports:
- containerPort: 8080
name: http
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /hello
port: 8080
scheme: HTTP
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 100m
memory: 1Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirstWithHostNet
imagePullSecrets:
- name: gitlab-registry
# `Deployment Service`版本`YAML`
$ vim rollout.yaml
---
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
name: rollouts-simple-java
namespace: myapp-dev
spec:
replicas: 3
strategy:
canary:
canaryService: rollouts-simple-java-canary
stableService: rollouts-simple-java-stable
trafficRouting:
nginx:
stableIngress: rollouts-simple-java-stable
steps:
- setWeight: 20
- pause: {duration: 60}
- setWeight: 50
- pause: {duration: 10}
- setWeight: 80
- pause: {duration: 10}
revisionHistoryLimit: 2
selector:
matchLabels:
app: rollouts-simple-java
template:
metadata:
labels:
app: rollouts-simple-java
spec:
containers:
- args:
- -jar
- /opt/myapp.jar
- --server.port=8080
command:
- java
image: harbordemo.chsaos.com/rollout-java-project/myapp:latest
imagePullPolicy: IfNotPresent
lifecycle:
preStop:
exec:
command:
- /bin/sh
- -c
- /bin/sleep 30
livenessProbe:
failureThreshold: 3
httpGet:
path: /hello
port: 8080
scheme: HTTP
initialDelaySeconds: 60
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 1
name: myapp
ports:
- containerPort: 8080
name: http
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /hello
port: 8080
scheme: HTTP
periodSeconds: 15
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "1"
memory: 2Gi
requests:
cpu: 100m
memory: 1Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullSecrets:
- name: gitlab-registry
# `Services.yaml`编排文件
$ vim services.yaml
---
apiVersion: v1
kind: Service
metadata:
name: rollouts-simple-java-canary
namespace: myapp-dev
spec:
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: rollouts-simple-java
# This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:
# rollouts-pod-template-hash: 7bf84f9696
---
apiVersion: v1
kind: Service
metadata:
name: rollouts-simple-java-stable
namespace: myapp-dev
spec:
ports:
- port: 8080
targetPort: http
protocol: TCP
name: http
selector:
app: rollouts-simple-java
# This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:
# rollouts-pod-template-hash: 789746c88d
# `ingress`编排文件
$ vim ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: rollouts-simple-java-stable
namespace: myapp-dev
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: testrolloutsdemo.chsaos.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: rollouts-simple-java-stable
port:
number: 8080
tls:
- hosts:
- testrolloutsdemo.chsaos.com
secretName: testcoolops-tls
# `kustomization.yaml`文件
$ vim kustomization.yaml
# Example configuration for the webserver
# at https://github.com/monopole/hello
commonLabels:
app: rollouts-simple-java
resources:
- rollout.yaml
- services.yaml
- ingress.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
images:
- name: harbordemo.chsaos.com/rollout-java-project/myapp
newTag: "latest"
namespace: myapp-dev
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
- 这里的金丝雀发布采是带有时间暂停的方式,才可以手动继续的方式;方便测试.
# 四、仓库代码Tag标签
- 当代码仓库经过长时间的迭代,针对不同的时期和需求,必定会有不同的版本.而借助
Git提供的标签功能,可以快捷方便地记录代码版本.无论什么时候,想取回某个版本,不再需要查找冗长的commit_id,只需要取出打标签的历史版本即可. - 标签是版本的一个快照.主流的
Git平台上,版本可以直接下载的,节省了开发者的时间精力. - 通过
gitlab的api对代码仓库打tag.API的具体操作: https://docs.gitlab.com/ee/api/tags.html
# shareLibrary新增gitlab.groovy文件
## 这里的`${gitServer}`需要在`rollout-java.Jenkinsfile`文件中的`environment`中进行定义
$ vim gitlab.groovy
package org.devops
//封装HTTP请求
def HttpReq(reqType,reqUrl,reqBody){
// def gitServer = "http://172.17.100.135:32080/api/v4"
withCredentials([string(credentialsId: 'gitlab-token', variable: 'gitlabToken')]) {
result = httpRequest customHeaders: [[maskValue: true, name: 'PRIVATE-TOKEN', value: "${gitlabToken}"]],
httpMode: reqType,
contentType: "APPLICATION_JSON",
consoleLogResponseBody: true,
ignoreSslErrors: true,
requestBody: reqBody,
url: "${gitServer}/${reqUrl}"
//quiet: true
}
return result
}
//获取项目ID
def GetProjectID(projectName){
projectApi = "projects?search=${projectName}"
response = HttpReq('GET',projectApi,'')
def result = readJSON text: """${response.content}"""
for (repo in result){
// println(repo['path_with_namespace'])
if (repo['path'] == "${projectName}"){
repoId = repo['id']
println(repoId)
}
}
return repoId
}
// 给仓库打tag
def TagGitlab(projectId,tag_name,tag_ref){
def apiUrl = "projects/${projectId}/repository/tags"
reqBody = """{"tag_name": "${tag_name}","ref": "${tag_ref}"}"""
HttpReq('POST',apiUrl,reqBody)
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
- 通过
GetProjectID获取到项目仓库的ID,然后再调用TagGitlab进行打Tag
# Jenkins创建gitlab-token
- 在
Jenkins上创建一个名叫gitlab-token的token凭据
# GitLab生成Token
Name: rollout-gitlab-token
Access Token:
yhMif-sy4dHegEr97d4U
Linux curl测试GitLab API接口GitLab官方文档https://docs.gitlab.com/ee/user/project/settings/project_access_tokens.html
https://docs.gitlab.com/ee/api/search.html
$ curl --header "PRIVATE-TOKEN: yhMif-sy4dHegEr97d4U" "https://gitlabdemo.chsaos.com/api/v4/projects?search=springboot-helloworld"|jq
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3675 0 3675 0 0 23099 0 --:--:-- --:--:-- --:--:-- 23259
[
{
"id": 2,
"description": "",
"name": "springboot-helloworld",
"name_with_namespace": "java / springboot-helloworld",
"path": "springboot-helloworld",
"path_with_namespace": "java/springboot-helloworld",
"created_at": "2022-11-16T21:41:52.667+08:00",
"default_branch": "main",
"tag_list": [],
"topics": [],
"ssh_url_to_repo": "ssh://git@gitlabdemo.chsaos.com:30022/java/springboot-helloworld.git",
"http_url_to_repo": "http://gitlabdemo.chsaos.com/java/springboot-helloworld.git",
"web_url": "http://gitlabdemo.chsaos.com/java/springboot-helloworld",
"readme_url": "http://gitlabdemo.chsaos.com/java/springboot-helloworld/-/blob/main/README.md",
"avatar_url": null,
"forks_count": 0,
"star_count": 0,
"last_activity_at": "2022-11-18T21:44:04.641+08:00",
"namespace": {
"id": 4,
"name": "java",
"path": "java",
"kind": "group",
"full_path": "java",
"parent_id": null,
"avatar_url": null,
"web_url": "http://gitlabdemo.chsaos.com/groups/java"
},
"_links": {
"self": "http://gitlabdemo.chsaos.com/api/v4/projects/2",
"issues": "http://gitlabdemo.chsaos.com/api/v4/projects/2/issues",
"merge_requests": "http://gitlabdemo.chsaos.com/api/v4/projects/2/merge_requests",
"repo_branches": "http://gitlabdemo.chsaos.com/api/v4/projects/2/repository/branches",
"labels": "http://gitlabdemo.chsaos.com/api/v4/projects/2/labels",
"events": "http://gitlabdemo.chsaos.com/api/v4/projects/2/events",
"members": "http://gitlabdemo.chsaos.com/api/v4/projects/2/members",
"cluster_agents": "http://gitlabdemo.chsaos.com/api/v4/projects/2/cluster_agents"
},
"packages_enabled": true,
"empty_repo": false,
"archived": false,
"visibility": "private",
"resolve_outdated_diff_discussions": false,
"container_expiration_policy": {
"cadence": "1d",
"enabled": false,
"keep_n": 10,
"older_than": "90d",
"name_regex": ".*",
"name_regex_keep": null,
"next_run_at": "2022-11-17T21:41:52.729+08:00"
},
"issues_enabled": true,
"merge_requests_enabled": true,
"wiki_enabled": true,
"jobs_enabled": true,
"snippets_enabled": true,
"container_registry_enabled": true,
"service_desk_enabled": false,
"service_desk_address": null,
"can_create_merge_request_in": true,
"issues_access_level": "enabled",
"repository_access_level": "enabled",
"merge_requests_access_level": "enabled",
"forking_access_level": "enabled",
"wiki_access_level": "enabled",
"builds_access_level": "enabled",
"snippets_access_level": "enabled",
"pages_access_level": "private",
"operations_access_level": "enabled",
"analytics_access_level": "enabled",
"container_registry_access_level": "enabled",
"security_and_compliance_access_level": "private",
"emails_disabled": null,
"shared_runners_enabled": true,
"lfs_enabled": true,
"creator_id": 1,
"import_url": null,
"import_type": null,
"import_status": "none",
"open_issues_count": 0,
"ci_default_git_depth": 20,
"ci_forward_deployment_enabled": true,
"ci_job_token_scope_enabled": false,
"ci_separated_caches": true,
"public_jobs": true,
"build_timeout": 3600,
"auto_cancel_pending_pipelines": "enabled",
"build_coverage_regex": null,
"ci_config_path": null,
"shared_with_groups": [],
"only_allow_merge_if_pipeline_succeeds": false,
"allow_merge_on_skipped_pipeline": null,
"restrict_user_defined_variables": false,
"request_access_enabled": true,
"only_allow_merge_if_all_discussions_are_resolved": false,
"remove_source_branch_after_merge": true,
"printing_merge_request_link_enabled": true,
"merge_method": "merge",
"squash_option": "default_off",
"enforce_auth_checks_on_uploads": true,
"suggestion_commit_message": null,
"merge_commit_template": null,
"squash_commit_template": null,
"auto_devops_enabled": false,
"auto_devops_deploy_strategy": "continuous",
"autoclose_referenced_issues": true,
"repository_storage": "default",
"keep_latest_artifact": true,
"runner_token_expiration_interval": null,
"permissions": {
"project_access": null,
"group_access": {
"access_level": 50,
"notification_level": 3
}
}
}
]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
# Jenkins添加Token
Jenkins上创建凭据 系统管理->凭据管理->全局凭据->添加凭据
这里的
ID要和gitlab.groovy中的ID一一对应
# 五、ArgoCD中创建项目
YAML配置模板
vim rollout-simple-java.yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: 'rollout-simple-java'
namespace: argocd
spec:
destination:
namespace: 'dev'
server: 'https://kubernetes.default.svc'
source:
path: 'rollout-simple-java/'
repoURL: 'https://gitlabdemo.chsaos.com/infra/devops-cd.git'
targetRevision: HEAD
project: 'default'
syncPolicy:
automated: {}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
UI界面配置
Application Name: rollout-simple-java
Project Name: default
SYNC POLICY: Automatic
SYNC OPTAIONS: PRUNE LAST
Repository URL: https://gitlabdemo.chsaos.com/infra/devops-cd.git
Revision: HEAD
Path: rollout-simple-java/
Cluster URL: https://kubernetes.default.svc
Namespace: myapp-dev
Kubernetes中查看服务部署状态信息
$ kubectl get pods,svc -n myapp-dev
NAME READY STATUS RESTARTS AGE
pod/rollouts-simple-java-66979f95b-45jpw 1/1 Running 0 4m3s
pod/rollouts-simple-java-66979f95b-b6cdj 1/1 Running 0 4m3s
pod/rollouts-simple-java-66979f95b-vjzzk 1/1 Running 0 4m3s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/rollouts-simple-java-canary ClusterIP 10.98.194.83 <none> 8080/TCP 4m3s
service/rollouts-simple-java-stable ClusterIP 10.97.5.114 <none> 8080/TCP 4m3s
$ kubectl get ingress -n myapp-dev
NAME CLASS HOSTS ADDRESS PORTS AGE
rollouts-simple-java-rollouts-simple-java-stable-canary <none> testrolloutsdemo.chsaos.com 80 4m23s
rollouts-simple-java-stable <none> testrolloutsdemo.chsaos.com 80, 443 4m23s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# 六、Jenkins配置项目
# 1.shareLibrary新增Jenkinsfile
rollout-java.Jenkinsfile
def labels = "slave-${UUID.randomUUID().toString()}"
// 引用共享库
@Library("jenkins_shareLibrary")
// 应用共享库中的方法
def tools = new org.devops.tools()
def sonarapi = new org.devops.sonarAPI()
def sendEmail = new org.devops.sendEmail()
def build = new org.devops.build()
def sonar = new org.devops.sonarqube()
def gitlab = new org.devops.gitlab()
def deploy = new org.devops.deploy()
// 前端传来的变量
def gitBranch = env.branch
def gitUrl = env.git_url
def buildShell = env.build_shell
def image = env.image
def dockerRegistryUrl = env.dockerRegistryUrl
def devops_cd_git = env.devops_cd_git
def repo_name = env.repo_name
// 固定变量
// def SonarServer = "http://sonar.devops.svc.cluster.local:9000/api"
// def dockerRegistryUrl = "https://harbordemo.chsaos.com"
def isUpdate = ''
pipeline {
agent {
kubernetes {
label labels
yaml """
apiVersion: v1
kind: Pod
metadata:
labels:
some-label: some-label-value
spec:
volumes:
- name: docker-sock
hostPath:
path: /var/run/docker.sock
type: ''
- name: maven-cache
persistentVolumeClaim:
claimName: maven-cache-pvc
containers:
- name: jnlp
image: jenkins/inbound-agent:alpine-jdk8
- name: maven
image: maven:3.6.1-jdk-8-alpine
command:
- cat
tty: true
volumeMounts:
- name: maven-cache
mountPath: /root/.m2
- name: docker
image: docker:20.10.21-alpine3.16
command:
- cat
tty: true
volumeMounts:
- name: docker-sock
mountPath: /var/run/docker.sock
- name: sonar-scanner
image: sonarsource/sonar-scanner-cli:4.7
command:
- cat
tty: true
- name: kustomize
image: banna1990/kubecl-kustomize:v1.23.10-v4.5.6
command:
- cat
tty: true
"""
}
}
environment{
auth = 'joker'
sonarUser = 'admin'
sonarPwd = 'admin321'
gitServer = 'https://gitlabdemo.chsaos.com/api/v4'
}
options {
timestamps() // 日志会有时间
skipDefaultCheckout() // 删除隐式checkout scm语句
disableConcurrentBuilds() //禁止并行
timeout(time:1,unit:'HOURS') //设置流水线超时时间
}
parameters {
string(name: 'git_url', defaultValue: 'http://gitlabdemo.chsaos.com/java/springboot-helloworld.git', description: '代码仓库地址')
string(name: 'branch', defaultValue: 'main', description: 'git分支')
string(name: 'build_shell', defaultValue: 'mvn clean package -Dmaven.test.skip=true && mv target/*-SNAPSHOT.jar ./myapp.jar', description: 'build执行命令')
string(name: 'image', defaultValue: 'harbordemo.chsaos.com/rollout-java-project/myapp', description: '镜像名称')
string(name: 'dockerRegistryUrl', defaultValue: 'harbordemo.chsaos.com', description: '镜像仓库地址')
string(name: 'toEmailUser', defaultValue: 'jiajia664878380@163.com', description: '邮件发送使用的用户')
string(name: 'devops_cd_git', defaultValue: 'gitlabdemo.chsaos.com/infra/devops-cd.git', description: 'kustomization编排文件仓库')
}
stages {
// 拉取代码
stage('GetCode') {
steps {
checkout([$class: 'GitSCM', branches: [[name: "${gitBranch}"]],
doGenerateSubmoduleConfigurations: false,
extensions: [],
submoduleCfg: [],
userRemoteConfigs: [[credentialsId: 'ci-devops', url: "${gitUrl}"]]])
}
}
// 单元测试和编译打包
stage('Build&Test') {
steps {
container('maven') {
script{
tools.PrintMes("编译打包","blue")
build.DockerBuild("${buildShell}")
}
}
}
}
// 代码扫描
stage('CodeScanner') {
steps {
container('sonar-scanner') {
script {
tools.PrintMes("代码扫描","green")
tools.PrintMes("搜索项目","green")
result = sonarapi.SearchProject("${JOB_NAME}")
println(result)
if (result == "false"){
println("${JOB_NAME}---项目不存在,准备创建项目---> ${JOB_NAME}！")
sonarapi.CreateProject("${JOB_NAME}")
} else {
println("${JOB_NAME}---项目已存在！")
}
tools.PrintMes("代码扫描","green")
sonar.SonarScan("${JOB_NAME}","${JOB_NAME}","src")
sleep 10
tools.PrintMes("获取扫描结果","green")
result = sonarapi.GetProjectStatus("${JOB_NAME}")
println(result)
if (result.toString() == "ERROR"){
toemail.Email("代码质量阈错误！请及时修复！",userEmail)
error " 代码质量阈错误！请及时修复！"
} else {
println(result)
}
}
}
}
}
// 构建镜像
stage('BuildImage') {
steps {
withCredentials([usernamePassword(credentialsId: 'dockerhub',
passwordVariable: 'password',
usernameVariable: 'username')]) {
container('docker') {
script{
tools.PrintMes("构建镜像","green")
imageTag = tools.createVersion()
sh """
docker login ${dockerRegistryUrl} -u ${username} -p ${password}
docker build -t ${image}:${imageTag} .
docker push ${image}:${imageTag}
docker rmi ${image}:${imageTag}
find ./ -name "*.jar" |xargs -i rm {} -rf
"""
}
}
}
}
}
// 部署
stage('Deploy') {
steps {
withCredentials([usernamePassword(credentialsId: 'ci-devops',
passwordVariable: 'password',
usernameVariable: 'username')]){
container('kustomize') {
script{
APP_DIR="rollout-simple-java"
UP_TIME=tools.createVersion()
sh """
rm /opt/devops-cd -rf
git clone https://${username}:${password}@${devops_cd_git} /opt/devops-cd
cd /opt/devops-cd/
git config --global user.name "Administrator"
git config --global user.email "jiajia664878380@163.com"
cd ${APP_DIR}
kustomize edit set image ${image}:${imageTag}
sleep 1
cd ../
git commit -am "image update ${UP_TIME}"
git push origin main
"""
}
}
}
}
}
// 接口测试
stage('InterfaceTest') {
steps{
sh 'echo "接口测试"'
}
}
// 继续更新或回滚
stage('UpdateOrRollBack') {
input {
message 'Should we continue?'
ok 'Yes, we should.'
submitter 'alice,bob'
parameters {
string(name: 'input', defaultValue: 'yes', description: 'continue update?')
}
}
steps {
script {
// 调用更新或者回滚函数
tools.PrintMes("更新或者回滚","green")
// 将input的值赋值给全局变量isUpdate，供下阶段使用
isUpdate = "${input}"
}
}
}
// 如果是继续更新服务，待验证通过后给gitlab代码仓库打tag
stage('TagGitlab') {
steps {
script {
if ("${isUpdate}" == 'yes' && "${gitBranch }" == 'main') {
tools.PrintMes('给仓库打TAG', 'green')
// 获取项目的projectId
repo_id = gitlab.GetProjectID("${repo_name}")
sh "echo ${repo_id}"
// 生产tag,以当前时间为tag
tag_name = "release"+"-"+tools.getTime()
gitlab.TagGitlab("${repo_id}", "${tag_name}", 'main')
}else {
tools.PrintMes('不打TAG', 'red')
}
}
}
}
}
// 构建后的操作
post {
success {
script{
println("success:只有构建成功才会执行")
currentBuild.description += "\n构建成功！"
// deploy.AnsibleDeploy("${deployHosts}","-m ping")
sendEmail.SendEmail("构建成功",toEmailUser)
// dingmes.SendDingTalk("构建成功 ✅")
}
}
failure {
script{
println("failure:只有构建失败才会执行")
currentBuild.description += "\n构建失败!"
sendEmail.SendEmail("构建失败",toEmailUser)
// dingmes.SendDingTalk("构建失败 ❌")
}
}
aborted {
script{
println("aborted:只有取消构建才会执行")
currentBuild.description += "\n构建取消!"
sendEmail.SendEmail("取消构建",toEmailUser)
// dingmes.SendDingTalk("构建失败 ❌","暂停或中断")
}
}
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
这里的Jenkinfile和前面的大同小异,只是增加了两个stage,其中UpdateOrRollBack这个stag只是占了一个位置,并没有具体实现,思路如下:
- 在部署新版本的时候第一次暂停,然后通过
Jenkins这里的输入决定是否继续- 如果继续则表示该版本上线没什么问题,继续后面的
TagGitlab - 如果不继续则表示该版本上线有问题,取消本次上线,并将应用回滚至上一版本
- 如果继续则表示该版本上线没什么问题,继续后面的
# 2.Jenkins配置项目
Jenkins上JOB名字的前缀和YAML清单所在的文件名一致
任务名称: rollout-simple-java_TEST
# 添加参数
名称: git_url
默认值: http://gitlabdemo.chsaos.com/java/springboot-helloworld.git
名称: branch
默认值: main
名称: build_shell
默认值:
mvn clean package -Dmaven.test.skip=true && mv target/*-SNAPSHOT.jar ./myapp.jar
名称: image
默认值:
harbordemo.chsaos.com/rollout-java-project/myapp
名称:
dockerRegistryUrl默认值:
harbordemo.chsaos.com
名称:
toEmailUser默认值:
jiajia664878380@163.com
名称:
devops_cd_git默认值:
gitlabdemo.chsaos.com/infra/devops-cd.git
名称:
repo_name默认值:
springboot-helloworld
Repository URL: https://gitlabdemo.chsaos.com/infra/sharelibrary.git
指定分支:
*/main脚本路径:
rollout-java.Jenkinsfile
# 七、发布应用
# 1.集成Gitlab
# 通过Webhook触发Jenkins
在
Jenkins中选择项目,在项目中配置gitlab触发
GitLab. GitLab webhook UR: http://jenkinsdemo.chsaos.com/project/rollout-simple-java_TEST
Secret token:2550164455ebd09fa2e346761de5260e
# Gitlab上配置集成
进入项目-->项目设置-->集成
项目名称:
springboot-helloworld
配置
Jenkins上生成的回调URL和TOKENGitLab. GitLab webhook UR: http://jenkinsdemo.chsaos.com/project/rollout-simple-java_TESTSecret token:2550164455ebd09fa2e346761de5260e
Gitlab手动发送test观察是否触发流水线
- 到
ArgocdWEB界面rollout-simple-java应用正在同步
这里暂时没有在
Jenkins上没有完成,在Argo Rollouts确认发布或者回滚的功能这里会暂停一段时间后,会自动恢复正常健康状态
Jenkins JOB上进行确认这里的
Jenkins的确认是虚假的展示;后端暂时没有完成这个功能
# 2.命令行终端使用curl测试
- 访问域名:
https://testrolloutsdemo.chsaos.com/hello
$ while true;do curl https://testrolloutsdemo.chsaos.com/hello;sleep 3;echo "\n";done
</html>\n
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8"></meta>
<title>Hello World</title>
</head>
<body>
<h1>Hello World</h1>
<body>
</html>\n
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8"></meta>
<title>Hello World</title>
</head>
<body>
<h1>Hello World</h1>
<body>
</html>
...
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
- 修改源代码将
Hello World!修改为Hello Bruce!如下:
Jenkins JOB进行确认
Arogcd上查看rollout-simple-java应用的状态变化
这里暂时暂停状态
这里已经在做最新版本的部署
最新版本部署完成
在Kubernetes集群上查看pod的部署状态
这里所有流量已经完全切换到新版本上
命令行端已经所有的流量都切换到最新发布的内容上
%5CCICD%5CCICD%5Cimages%5Cimage-20221120000504287.png)
WEB浏览器上展示内容也已经变化了
GitLab代码仓库查看新的Tag
查看更改的代码部分
# 八、Argo Rollouts Dashoard安装
$ git clone https://github.com/argoproj/argo-rollouts.git /root/argo-rollouts
$ cp -rp /root/argo-rollouts/manifests/dashboard-install /home/kubernetes-software-yaml/
$ cd /home/kubernetes-software-yaml/argo-rollouts-dashboard-install
# 提前创建好ssl的`Secret`
kubectl create secret tls argorollouts-tls --cert=/home/kubernetes-software-yaml/ssl/chsaos.com.crt --key=/home/kubernetes-software-yaml/ssl/chsaos.com.key -n argo-rollouts
# 新增加`ingress`编排文件
vim dashboard-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: argo-rollouts-dashboard
annotations:
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
rules:
- host: argorollouts.chsaos.com
http:
paths:
- pathType: Prefix
path: "/"
backend:
service:
name: argo-rollouts-dashboard
port:
number: 3100
tls:
- hosts:
- argorollouts.chsaos.com
secretName: argorollouts-tls
# 编辑`kustomization.yaml`增加ingress编排文件
vim kustomization.yaml
---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- dashboard-clusterrolebinding.yaml
- dashboard-clusterrole.yaml
- dashboard-deployment.yaml
- dashboard-service.yaml
- dashboard-serviceaccount.yaml
- dashboard-ingress.yaml
images:
- name: quay.io/argoproj/kubectl-argo-rollouts
newTag: latest
$ kubectl apply -k . --dry-run=server -n argo-rollouts
# 查看服务状态信息
$ kubectl get pods,svc -n argo-rollouts
$ kubectl get ingress -n argo-rollouts
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
# 九、总结
Argocd和Argo Rollouts,初步测试结果运行都比较稳定;但Argocd有如下几点需要注意:
- 备份
argocd上每个应用yaml文件,因为argocd本身是无状态的,保不齐什么时候就会被清空 argocd-cm的configMap每次修改过后会清空部署在的应用,不过对应用本身不影响,上面的备份是为了方便重建argo rollouts对ingress的支持有限,目前只支持ingress和alb
上次更新: 2024/04/09, 16:48:42
- 01
- AWS NAT-NetWork-Firwalld配置(一)04-09
- 02
- AWS NAT-NetWork-Firwalld配置(二)04-09
- 03
- kubernetes部署minio对象存储01-18