Bruce Blog
Bruce
2023-01-17
目录

Jenkins-Argocd CICD Rollouts金丝雀发布

# Argo Rollouts

Argo Rollout是一个Kubernetes Controller和对应一系列的CRD,提供更强大的Deployment能力.包括灰度发布、蓝绿部署、更新测试(experimentation)、渐进式交付(progressive delivery)等特性.

# 支持特性
  • 蓝绿发布更新策略
  • 金丝雀更新策略
  • 细粒度,加权流量转移
  • 自动回滚Rollback和promotion
  • 手动判断
  • 可定制的指标查询和业务KPI分析
  • 入口控制器集成: NGINX、ALB
  • 服务网格集成: Istio、Linked、SMI
  • Metric provider集成: Prometheus、Wavefront、Kayenta、Web、Kubernetes Jobs

Argo原理和Deployment类似,Argo是加强Rollout的策略和流量控制.当spec.template发送变化时,Argo-Rollout会根据spec.strategy进行rollout,通常会产生一个新的ReplicaSet,逐步scale down之前的ReplicaSet的pod数量.

# Argo Rollout安装

官方文档: https://argoproj.github.io/argo-rollouts/installation/#kubectl-plugin-installation

# 1.Kubernetes安装argo-rollouts
# 在线安装
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml

# 指定离线文档安装
curl -LO https://github.com/argoproj/argo-rollouts/releases/latest/download/install.yaml
kubectl create namespace argo-rollouts
kubectl apply -n argo-rollouts -f install.yaml

# 查看`argo-rollouts` pod信息
kubectl get pods -n argo-rollouts -owide
1
2
3
4
5
6
7
8
9
10
11
# 2.安装argo-rollouts插件
# 安装`argo-rollouts`插件
curl -LO https://github.com/argoproj/argo-rollouts/releases/download/v1.3.1/kubectl-argo-rollouts-linux-amd64
chmod +x kubectl-argo-rollouts-linux-amd64
cp -rp kubectl-argo-rollouts-linux-amd64  /usr/bin/kubectl-argo-rollouts

# 验证插件是否安装成功
$ kubectl argo rollouts version
kubectl-argo-rollouts: v1.3.1+b0b95ca
  BuildDate: 2022-09-29T15:17:17Z
  GitCommit: b0b95cade830d8a70624fcaaf8e92e1fe2ecd355
  GitTreeState: clean
  GoVersion: go1.18.6
  Compiler: gc
  Platform: linux/amd64
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# 金丝雀发布

  • 恢复发布包含如下:
    • Replica Shifting: 版本替换
    • Traffic Shifting: 流量控制

# 官方Demo

https://argoproj.github.io/argo-rollouts/getting-started/

# Replica Shifting

# 部署应用

应用部署项目路径: kubernetes-software-yaml/argo-rollouts/basic

$ git clone https://github.com/argoproj/argo-rollouts.git /root/argo-rollouts
$ cp -rp /root/argo-rollouts/docs/getting-started/basic /home/kubernetes-software-yaml/argo-rollouts/
$ cd /home/kubernetes-software-yaml/argo-rollouts/basic
$ tree  ./
./
├── rollout.yaml
└── service.yaml

0 directories, 2 files

# 查看`rollouts.yaml`文件内容
$ cat rollout.yaml 
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
spec:
  replicas: 5
  strategy:
    canary:
      steps:
      - setWeight: 20
      - pause: {}
      - setWeight: 40
      - pause: {duration: 10}
      - setWeight: 60
      - pause: {duration: 10}
      - setWeight: 80
      - pause: {duration: 10}
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
      - name: rollouts-demo
        image: argoproj/rollouts-demo:blue
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
  • apiVersion、Kind和Deployment相同;strategy字段定义的是发布策略:
    • setWeight: 设置流量的权重
    • pause: 暂停,如果没有跟duration: 10则表示需要手动更新,如果跟了表示等待多长时间会自动更新.

service.yaml和rollout-ingress.yaml文件内容

$ cat service.yaml
---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo
    
$ cat rollout-ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-rollout
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: rolloutdemo.chsaos.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: rollouts-demo
            port:
              number: 80
  tls:
  - hosts: 
    - rolloutdemo.chsaos.com
    secretName: jenkins-tls
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

部署argo rollouts

$ kubectl apply -f rollout.yaml  -f service.yaml  -f rollout-ingress.yaml  --dry-run=server
rollout.argoproj.io/rollouts-demo created (server dry run)
service/rollouts-demo created (server dry run)
ingress.networking.k8s.io/my-rollout created (server dry run)

$ kubectl apply -f rollout.yaml  -f service.yaml  -f rollout-ingress.yaml

$ kubectl get pods,svc
NAME                                          READY   STATUS    RESTARTS      AGE
pod/my-nginx-64cc7f8db7-xqfk7                 1/1     Running   1 (22h ago)   3d5h
pod/nfs-client-provisioner-66855d4df9-gt9jw   1/1     Running   1             22h
pod/rollouts-demo-687d76d795-648vw            1/1     Running   0             3m35s
pod/rollouts-demo-687d76d795-74b75            1/1     Running   0             3m35s
pod/rollouts-demo-687d76d795-r8hqr            1/1     Running   0             3m35s
pod/rollouts-demo-687d76d795-ww5n2            1/1     Running   0             3m35s
pod/rollouts-demo-687d76d795-z4vqj            1/1     Running   0             3m35s

NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/kubernetes	ClusterIP   10.96.0.1        <none>        443/TCP   3d5h
service/my-nginx        ClusterIP   10.111.141.0     <none>        80/TCP    3d5h
service/rollouts-demo   ClusterIP   10.105.242.183   <none>        80/TCP    3m35s

$ kubectl get ingress 
NAME         CLASS    HOSTS                    ADDRESS   PORTS     AGE
my-rollout   <none>   rolloutdemo.chsaos.com             80, 443   4m6s


# 查看部署状态
$ kubectl argo rollouts get rollout rollouts-demo --watch
$ kubectl argo rollouts get rollout rollouts-demo
Name:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          8/8
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       5
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS     AGE    INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy  5m23s  
└──# revision:1                                                          
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy  5m23s  stable
      ├──□ rollouts-demo-687d76d795-648vw  Pod         ✔ Running  5m23s  ready:1/1
      ├──□ rollouts-demo-687d76d795-74b75  Pod         ✔ Running  5m23s  ready:1/1
      ├──□ rollouts-demo-687d76d795-r8hqr  Pod         ✔ Running  5m23s  ready:1/1
      ├──□ rollouts-demo-687d76d795-ww5n2  Pod         ✔ Running  5m23s  ready:1/1
      └──□ rollouts-demo-687d76d795-z4vqj  Pod         ✔ Running  5m23s  ready:1/1

$ kubectl-argo-rollouts get rollout rollouts-demo
Name:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          8/8
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       5
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS     AGE   INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy  6m2s  
└──# revision:1                                                         
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy  6m2s  stable
      ├──□ rollouts-demo-687d76d795-648vw  Pod         ✔ Running  6m2s  ready:1/1
      ├──□ rollouts-demo-687d76d795-74b75  Pod         ✔ Running  6m2s  ready:1/1
      ├──□ rollouts-demo-687d76d795-r8hqr  Pod         ✔ Running  6m2s  ready:1/1
      ├──□ rollouts-demo-687d76d795-ww5n2  Pod         ✔ Running  6m2s  ready:1/1
      └──□ rollouts-demo-687d76d795-z4vqj  Pod         ✔ Running  6m2s  ready:1/1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
  • 看到版本被标记为stable,而且STATUS为healthy.可以在末尾添加--watch来实时监控服务状态
    • 完整命令: kubectl get rollouts get rollout rollouts-demo --watch
# 域名访问rollouts-demo服务
  • https://rolloutdemo.chsaos.com/

image-20221119163337862

# 更新应用
# argo rollouts 更新镜像`set image`
$ kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow
rollout "rollouts-demo" image updated

# 通过`--watch`实时观察服务状态
$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Name:            rollouts-demo
Namespace:       default
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/8
  SetWeight:     20
  ActualWeight:  0
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       5
  Current:       5
  Updated:       1
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS               AGE  INFO
⟳ rollouts-demo                            Rollout     ◌ Progressing        23m  
├──# revision:2                                                                  
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ◌ Progressing        6s   canary
│     └──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ◌ ContainerCreating  6s   ready:0/1
└──# revision:1                                                                  
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy            23m  stable
      ├──□ rollouts-demo-687d76d795-648vw  Pod         ✔ Running            23m  ready:1/1
      ├──□ rollouts-demo-687d76d795-74b75  Pod         ✔ Running            23m  ready:1/1
      ├──□ rollouts-demo-687d76d795-r8hqr  Pod         ✔ Running            23m  ready:1/1
      ├──□ rollouts-demo-687d76d795-ww5n2  Pod         ✔ Running            23m  ready:1/1
      └──□ rollouts-demo-687d76d795-z4vqj  Pod         ◌ Terminating        23m  ready:1/1


Name:            rollouts-demo
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/8
  SetWeight:     20
  ActualWeight:  20
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       5
  Current:       5
  Updated:       1
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS     AGE    INFO
⟳ rollouts-demo                            Rollout     ॥ Paused   27m    
├──# revision:2                                                          
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy  4m51s  canary
│     └──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running  4m51s  ready:1/1
└──# revision:1                                                          
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy  27m    stable
      ├──□ rollouts-demo-687d76d795-648vw  Pod         ✔ Running  27m    ready:1/1
      ├──□ rollouts-demo-687d76d795-74b75  Pod         ✔ Running  27m    ready:1/1
      ├──□ rollouts-demo-687d76d795-r8hqr  Pod         ✔ Running  27m    ready:1/1
      └──□ rollouts-demo-687d76d795-ww5n2  Pod         ✔ Running  27m    ready:1/1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

  • 此时多了一个version:2,而且版本标记为canary,状态是Status: Paused,canary计入流量为20%

  • 部署之所以处于Paused阶段,是因为在rollouts.yaml中定义了发布一个版本后会暂停,这时候需要手动接入接下来的更新.

  • argo rollouts提供了promote来进行后续的更新

    $ kubectl argo rollouts promote rollouts-demo
rollout 'rollouts-demo' promoted

# 再次查看`--watch`的状态
$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Name:            rollouts-demo
Namespace:       default
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          2/8
  SetWeight:     40
  ActualWeight:  25
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       5
  Current:       5
  Updated:       2
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS               AGE   INFO
⟳ rollouts-demo                            Rollout     ◌ Progressing        31m   
├──# revision:2                                                                   
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ◌ Progressing        8m1s  canary
│     ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running            8m1s  ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ◌ ContainerCreating  5s    ready:0/1
└──# revision:1                                                                   
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy            31m   stable
      ├──□ rollouts-demo-687d76d795-648vw  Pod         ✔ Running            31m   ready:1/1
      ├──□ rollouts-demo-687d76d795-74b75  Pod         ◌ Terminating        31m   ready:1/1
      ├──□ rollouts-demo-687d76d795-r8hqr  Pod         ✔ Running            31m   ready:1/1
      └──□ rollouts-demo-687d76d795-ww5n2  Pod         ✔ Running            31m   ready:1/1


Name:            rollouts-demo
Namespace:       default
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          4/8
  SetWeight:     60
  ActualWeight:  50
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       5
  Current:       5
  Updated:       3
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS               AGE    INFO
⟳ rollouts-demo                            Rollout     ◌ Progressing        31m    
├──# revision:2                                                                    
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ◌ Progressing        8m55s  canary
│     ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running            8m55s  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running            59s    ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ◌ ContainerCreating  32s    ready:0/1
└──# revision:1                                                                    
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy            31m    stable
      ├──□ rollouts-demo-687d76d795-648vw  Pod         ✔ Running            31m    ready:1/1
      └──□ rollouts-demo-687d76d795-r8hqr  Pod         ✔ Running            31m    ready:1/1


# 后续的更新在`pause`阶段只暂停`10s`,所以会依次自动更新完,不需要手动介入.
# 可以看到第一个版本已经下线,第二个版本的状态为`Healthy`,且镜像标记为`stable`
Name:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          8/8
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       5
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS        AGE    INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy     33m    
├──# revision:2                                                             
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy     10m    stable
│     ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running     10m    ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running     2m56s  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ✔ Running     2m29s  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-qqlbf  Pod         ✔ Running     91s    ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-ggqb7  Pod         ✔ Running     81s    ready:1/1
└──# revision:1                                                             
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  • ScaledDown  33m
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
# 终止更新
  • 在更新应用过程中,最新的应用有问题,终止更新操作
$ kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:red
rollout "rollouts-demo" image updated

$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          0/8
  SetWeight:     20
  ActualWeight:  0
Images:          argoproj/rolloutes-demo:red (canary)
                 argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       1
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS               AGE    INFO
⟳ rollouts-demo                            Rollout     ◌ Progressing        41m    
├──# revision:3                                                                    
│  └──⧉ rollouts-demo-6bd8dd6c77           ReplicaSet  ◌ Progressing        6s     canary
│     └──□ rollouts-demo-6bd8dd6c77-6726x  Pod         ◌ ContainerCreating  6s     ready:0/1
├──# revision:2                                                                    
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy            18m    stable
│     ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running            18m    ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running            10m    ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ✔ Running            9m39s  ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-qqlbf  Pod         ✔ Running            8m41s  ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-ggqb7  Pod         ◌ Terminating        8m31s  ready:1/1
└──# revision:1                                                                    
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  • ScaledDown         41m    

Name:            rollouts-demo
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/8
  SetWeight:     20
  ActualWeight:  20
Images:          argoproj/rollouts-demo:red (canary)
                 argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       1
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS        AGE   INFO
⟳ rollouts-demo                            Rollout     ॥ Paused      43m   
├──# revision:4                                                            
│  └──⧉ rollouts-demo-5747959bdb           ReplicaSet  ✔ Healthy     61s   canary
│     └──□ rollouts-demo-5747959bdb-5mbhp  Pod         ✔ Running     61s   ready:1/1
├──# revision:3                                                            
│  └──⧉ rollouts-demo-6bd8dd6c77           ReplicaSet  • ScaledDown  2m5s  
├──# revision:2                                                            
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy     20m   stable
│     ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running     20m   ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running     12m   ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ✔ Running     11m   ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-qqlbf  Pod         ✔ Running     10m   ready:1/1
└──# revision:1                                                            
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  • ScaledDown  43m 


# 更新动作会在第一次更新的时候处于`Paused`状态,可以用`abort`来终止发布
$ kubectl argo rollouts abort rollouts-demo
rollout 'rollouts-demo' aborted

# 再次通过`--watch`观察部署状态
$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Name:            rollouts-demo
Namespace:       default
Status:          ✖ Degraded
Message:         RolloutAborted: Rollout aborted update to revision 4
Strategy:        Canary
  Step:          0/8
  SetWeight:     0
  ActualWeight:  0
Images:          argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       0
  Ready:         5
  Available:     5


## 可以可以看到`Status`是`Degraded`状态而非`Healthy`状态,有必要将其变成`Healthy`状态.
NAME                                       KIND        STATUS         AGE   INFO
⟳ rollouts-demo                            Rollout     ✖ Degraded     44m   
├──# revision:4                                                             
│  └──⧉ rollouts-demo-5747959bdb           ReplicaSet  • ScaledDown   2m    canary
│     └──□ rollouts-demo-5747959bdb-5mbhp  Pod         ◌ Terminating  2m    ready:1/1
├──# revision:3                                                             
│  └──⧉ rollouts-demo-6bd8dd6c77           ReplicaSet  • ScaledDown   3m4s  
├──# revision:2                                                             
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy      21m   stable
│     ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running      21m   ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running      13m   ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ✔ Running      12m   ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-qqlbf  Pod         ✔ Running      11m   ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-wrz2v  Pod         ✔ Running      9s    ready:1/1
└──# revision:1                                                             
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  • ScaledDown   44m   


# 重新发布一下版本
$ kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow
rollout "rollouts-demo" image updated

## 可以看到其状态立即变成Healthy,并且没有创建新的副本、新的版本
$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Name:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          8/8
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       5
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS        AGE    INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy     48m    
├──# revision:5                                                             
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy     25m    stable
│     ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running     25m    ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running     17m    ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ✔ Running     17m    ready:1/1
│     ├──□ rollouts-demo-6cf78c66c5-qqlbf  Pod         ✔ Running     16m    ready:1/1
│     └──□ rollouts-demo-6cf78c66c5-wrz2v  Pod         ✔ Running     4m49s  ready:1/1
├──# revision:4                                                             
│  └──⧉ rollouts-demo-5747959bdb           ReplicaSet  • ScaledDown  6m40s  
└──# revision:3                                                             
   └──⧉ rollouts-demo-6bd8dd6c77           ReplicaSet  • ScaledDown  7m44s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
# 回退应用
  • 当应用上限后,有些BUG并没有发现,回退操作
    • 执行argo rollouts undo命令
# 这里没有处理好`argo rollouts`历史版本的保留记录.
$ kubectl-argo-rollouts undo rollouts-demo --to-reversion=4

# `--watch`观察部署状态
# 
$ kubectl argo rollouts get rollout rollouts-dmeo --watch
      └──□ rollouts-demo-6cf78c66c5-nxfql  Pod         ◌ Terminating  28s  ready:1/1
Name:            rollouts-demo
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/8
  SetWeight:     20
  ActualWeight:  20
Images:          argoproj/rollouts-demo:red (canary)
                 argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       1
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS         AGE  INFO
⟳ rollouts-demo                            Rollout     ॥ Paused       56m  
├──# revision:7                                                            
│  └──⧉ rollouts-demo-5747959bdb           ReplicaSet  ✔ Healthy      14m  canary
│     └──□ rollouts-demo-5747959bdb-dh974  Pod         ✔ Running      8s   ready:1/1
├──# revision:6                                                            
│  └──⧉ rollouts-demo-6bd8dd6c77           ReplicaSet  • ScaledDown   16m  
└──# revision:5                                                            
   └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy      33m  stable
      ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ✔ Running      33m  ready:1/1
      ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running      26m  ready:1/1
      ├──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ✔ Running      25m  ready:1/1
      ├──□ rollouts-demo-6cf78c66c5-qqlbf  Pod         ✔ Running      24m  ready:1/1
      └──□ rollouts-demo-6cf78c66c5-nxfql  Pod         ◌ Terminating  29s  ready:1/1

# 于暂停状态,执行`promote`命令继续后续的更新
$ kubectl argo rollouts promote rollouts-demo


$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Name:            rollouts-demo
Namespace:       default
Status:          ◌ Progressing
Message:         more replicas need to be updated
Strategy:        Canary
  Step:          2/8
  SetWeight:     40
  ActualWeight:  25
Images:          argoproj/rollouts-demo:blue (canary)
                 argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       2
  Ready:         4
  Available:     4

NAME                                       KIND        STATUS               AGE    INFO
⟳ rollouts-demo                            Rollout     ◌ Progressing        59m    
├──# revision:7                                                                    
│  └──⧉ rollouts-demo-5747959bdb           ReplicaSet  ◌ Progressing        17m    canary
│     ├──□ rollouts-demo-5747959bdb-dh974  Pod         ✔ Running            2m33s  ready:1/1
│     └──□ rollouts-demo-5747959bdb-f4z2n  Pod         ◌ ContainerCreating  5s     ready:0/1
├──# revision:6                                                                    
│  └──⧉ rollouts-demo-6bd8dd6c77           ReplicaSet  • ScaledDown         18m    
└──# revision:5                                                                    
   └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy            36m    stable
      ├──□ rollouts-demo-6cf78c66c5-hs5rd  Pod         ◌ Terminating        36m    ready:1/1
      ├──□ rollouts-demo-6cf78c66c5-zbr84  Pod         ✔ Running            28m    ready:1/1
      ├──□ rollouts-demo-6cf78c66c5-6xmzp  Pod         ✔ Running            27m    ready:1/1
      └──□ rollouts-demo-6cf78c66c5-qqlbf  Pod         ✔ Running            27m    ready:1/1


# 从`Images`可以看到应用被回退到最初版本为`blue`镜像了
Name:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          8/8
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue (stable)
Replicas:
  Desired:       5
  Current:       5
  Updated:       5
  Ready:         5
  Available:     5

NAME                                       KIND        STATUS        AGE    INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy     62m    
├──# revision:7                                                             
│  └──⧉ rollouts-demo-5747959bdb           ReplicaSet  ✔ Healthy     20m    stable
│     ├──□ rollouts-demo-5747959bdb-dh974  Pod         ✔ Running     5m20s  ready:1/1
│     ├──□ rollouts-demo-5747959bdb-f4z2n  Pod         ✔ Running     2m52s  ready:1/1
│     ├──□ rollouts-demo-5747959bdb-kjvn8  Pod         ✔ Running     2m13s  ready:1/1
│     ├──□ rollouts-demo-5747959bdb-2qtns  Pod         ✔ Running     99s    ready:1/1
│     └──□ rollouts-demo-5747959bdb-t87ld  Pod         ✔ Running     88s    ready:1/1
├──# revision:6                                                             
│  └──⧉ rollouts-demo-6bd8dd6c77           ReplicaSet  • ScaledDown  21m    
└──# revision:5                                                             
   └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  • ScaledDown  39m
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106

# Traffic Shifting

  • 上面没有接入外部流量,仅在内部使用了金丝雀部署过程.这里来接入外部流量进行测试
  • Argo-Rollout主要集成了Ingress和ServiceMesh两种流量控制方法.
  • 目前Ingress支持ALB和NGINX Ingress(这里使用的是nginx ingress)
# 应用部署

应用部署项目路径: kubernetes-software-yaml/argo-rollouts/nginx

# 编排文件内容
# 重新部署官方`nginx`的示例
$ cp -rp /root/argo-rollouts/docs/getting-started/nginx /home/kubernetes-software-yaml/argo-rollouts/
$ cd /home/kubernetes-software-yaml/argo-rollouts/nginx/
$ tree  ./
./
├── index.md
├── ingress.yaml
├── paused-rollout-nginx.png
├── rollout-nginx.png
├── rollout.yaml
└── services.yaml

0 directories, 6 files


# 这个例子包含1个rollout,2个service,1个ingress
## 内容如下
$ cat rollout.yaml
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: rollouts-demo
spec:
  replicas: 1
  strategy:
    canary:
      canaryService: rollouts-demo-canary
      stableService: rollouts-demo-stable
      trafficRouting:
        nginx:
          stableIngress: rollouts-demo-stable
      steps:
      - setWeight: 5
      - pause: {}
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: rollouts-demo
  template:
    metadata:
      labels:
        app: rollouts-demo
    spec:
      containers:
      - name: rollouts-demo
        image: argoproj/rollouts-demo:blue
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP
        resources:
          requests:
            memory: 32Mi
            cpu: 5m


$ cat services.yaml 
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-canary
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo
    # This selector will be updated with the pod-template-hash of the canary ReplicaSet. e.g.:
    # rollouts-pod-template-hash: 7bf84f9696

---
apiVersion: v1
kind: Service
metadata:
  name: rollouts-demo-stable
spec:
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    app: rollouts-demo
    # This selector will be updated with the pod-template-hash of the stable ReplicaSet. e.g.:
    # rollouts-pod-template-hash: 789746c88d

# 这里的`argocd-rollouts-tls`需要提前创建好,详情查看上级目录中的`README.md`文件
$ cat ingress.yaml
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: rollouts-demo-stable
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  rules:
  - host: rolloutdemo.chsaos.com
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: rollouts-demo-stable
            port:
              number: 80
  tls:
  - hosts: 
    - rolloutdemo.chsaos.com
    secretName: argocd-rollouts-tls
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
  • services.yaml文件中Rollout分别用canaryService和stableService来分别定义了应用灰度的Service Name(rollouts-demo-canary)和当前版本的Service Name(rollouts-demo-stable).且rollouts-demo-canary和rollots-demo-stable的service的内容是一样的.selector中暂时没有配置上pod-template-hash,Argo-Rollout Controller会根据实际的ReplicaSet hash来修改该值.
  • 当创建完ingress后,Rollout Controller会根据ingress rollouts-demo-stable内容,自动创建一个ingress用了灰度的流量,名字为--canary,所以就多了一个ingress rollouts-demo-rollouts-demo-stable-canary,将流量导向Canary Service(rollouts-demo-canary)
# 部署操作
# 先删除`Replica Shifting`示例部署的内容
$ cd /home/kubernetes-software-yaml/argo-rollouts/basic
$ kubectl delete -f rollout.yaml -f service.yaml 
rollout.argoproj.io "rollouts-demo" deleted
service "rollouts-demo" deleted

# 重新部署官方`nginx`的示例
$ cp -rp /root/argo-rollouts/docs/getting-started/nginx /home/kubernetes-software-yaml/argo-rollouts/
$ cd /home/kubernetes-software-yaml/argo-rollouts/nginx/
$ tree  ./
./
├── index.md
├── ingress.yaml
├── paused-rollout-nginx.png
├── rollout-nginx.png
├── rollout.yaml
└── services.yaml

0 directories, 6 files

$ kubectl apply -f rollout.yaml  -f services.yaml -f ingress.yaml 
rollout.argoproj.io/rollouts-demo created
service/rollouts-demo-canary created
service/rollouts-demo-stable created
ingress.networking.k8s.io/rollouts-demo-stable created

# 这个例子包含1个rollout,2个service,1个ingress
$ kubectl  get pods,svc 
NAME                                          READY   STATUS    RESTARTS      AGE
pod/rollouts-demo-687d76d795-mf9hk            1/1     Running   0             20s

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/kubernetes             ClusterIP   10.96.0.1       <none>        443/TCP   3d6h
service/rollouts-demo-canary   ClusterIP   10.110.101.55   <none>        80/TCP    20s
service/rollouts-demo-stable   ClusterIP   10.99.105.92    <none>        80/TCP    20s

$ kubectl  get ingress
NAME                                        CLASS    HOSTS                    ADDRESS   PORTS     AGE
rollouts-demo-rollouts-demo-stable-canary   <none>   rolloutdemo.chsaos.com             80        94s
rollouts-demo-stable                        <none>   rolloutdemo.chsaos.com             80, 443   94s
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
# 访问域名
  • https://rolloutdemo.chsaos.com/

image-20221119175950643

# 更新应用
# 更新应用
$ kubectl argo rollouts set image rollouts-demo rollouts-demo=argoproj/rollouts-demo:yellow
rollout "rollouts-demo" image updated

# 观察更新状态
$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Name:            rollouts-demo
Namespace:       default
Status:          ॥ Paused
Message:         CanaryPauseStep
Strategy:        Canary
  Step:          1/2
  SetWeight:     5
  ActualWeight:  5
Images:          argoproj/rollouts-demo:blue (stable)
                 argoproj/rollouts-demo:yellow (canary)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                                       KIND        STATUS     AGE  INFO
⟳ rollouts-demo                            Rollout     ॥ Paused   27m  
├──# revision:2                                                        
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy  46s  canary
│     └──□ rollouts-demo-6cf78c66c5-nhh6c  Pod         ✔ Running  46s  ready:1/1
└──# revision:1                                                        
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy  27m  stable
      └──□ rollouts-demo-687d76d795-mf9hk  Pod         ✔ Running  27m  ready:1/1


# 查看`canary`ingress中新增参数信息
$ kubectl get ingress
NAME                                        CLASS    HOSTS                    ADDRESS   PORTS     AGE
rollouts-demo-rollouts-demo-stable-canary   <none>   rolloutdemo.chsaos.com             80        29m
rollouts-demo-stable                        <none>   rolloutdemo.chsaos.com             80, 443   29m

$ kubectl get ingress rollouts-demo-rollouts-demo-stable-canary -oyaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/canary: "true"      # 新增参数
    nginx.ingress.kubernetes.io/canary-weight: "5"	# 新增参数
  creationTimestamp: "2022-11-19T09:35:53Z"
  generation: 1
  name: rollouts-demo-rollouts-demo-stable-canary
  namespace: default
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Rollout
    name: rollouts-demo
    uid: 4584a2c6-79bf-41cb-a68c-87a2a4eabe61
  resourceVersion: "597087"
  uid: f627c2de-ba67-456d-9601-1e72bfa87c96
spec:
  rules:
  - host: rolloutdemo.chsaos.com
    http:
      paths:
      - backend:
          service:
            name: rollouts-demo-canary
            port:
              number: 80
        path: /
        pathType: Prefix
status:
  loadBalancer: {}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
  • 通过页面发现此时多出了yellow的图标

image-20221119180842948

# 继续发布
$ kubectl argo rollouts promote rollouts-demo

# 终止发布
$ kubectl argo rollouts abort rollouts-demo


$ kubectl argo rollouts get rollout rollouts-dmeo --watch
Name:            rollouts-demo
Namespace:       default
Status:          ✔ Healthy
Strategy:        Canary
  Step:          2/2
  SetWeight:     100
  ActualWeight:  100
Images:          argoproj/rollouts-demo:blue
                 argoproj/rollouts-demo:yellow (stable)
Replicas:
  Desired:       1
  Current:       2
  Updated:       1
  Ready:         2
  Available:     2

NAME                                       KIND        STATUS     AGE    INFO
⟳ rollouts-demo                            Rollout     ✔ Healthy  34m    
├──# revision:2                                                          
│  └──⧉ rollouts-demo-6cf78c66c5           ReplicaSet  ✔ Healthy  7m40s  stable
│     └──□ rollouts-demo-6cf78c66c5-nhh6c  Pod         ✔ Running  7m40s  ready:1/1
└──# revision:1                                                          
   └──⧉ rollouts-demo-687d76d795           ReplicaSet  ✔ Healthy  34m    delay:17s
      └──□ rollouts-demo-687d76d795-mf9hk  Pod         ✔ Running  34m    ready:1/1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

# 总结

  • Argo-Rollout提供更加强大的Deplyment,包含比较适合运维的金丝雀发布和蓝绿发布
  • 例如使用蓝绿发布,仅需要配置rollout如下:
apiVersion: argoproj.io/v1alpha1
kind: Rollout  # 部署完rollout后就有了这个kind资源,这个资源和deployment类似也是管理你的副本集的,所以不能像deployment那样在k8s界面看见,只能通过kubectl命令行
metadata:
  name: rollout-bluegreen
  namespace: rollout-test
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 30
      containers:
      - resources: # {}
          requests:
            cpu: "1"
            memory: "2Gi"
          limits:
            cpu: "2"
            memory: "2Gi"
        terminationMessagePolicy: File
        imagePullPolicy: Always
        name: rollout-bluegreen
        image: argoproj/rollouts-demo:green # nginx:1.17.1
      schedulerName: default-scheduler
      securityContext: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
    metadata:
      labels:
        app: rollout-bluegreen
  selector:
    matchLabels:
      app: rollout-bluegreen
  replicas: 2
  strategy:
    blueGreen:  # 蓝绿启用配置
      activeService: rollout-bluegreen-active   # 生效的服务,需要自己创建建本代码最下面service资源
      previewService: rollout-bluegreen-preview  # 配置预览服务,同理需要自己创建
      autoPromotionEnabled: true  # 是否直接切换,如为true,会在新版本变绿后直接切换到对外服务
      scaleDownDelayRevisionLimit: 0
      previewReplicaCount: 1  # 新版本的pod数量,设为一个从而控制资源消耗
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 25%
    type: RollingUpdate
  revisionHistoryLimit: 2
  progressDeadlineSeconds: 600
---
apiVersion: v1
kind: Service
metadata:
  name: rollout-bluegreen-active
  namespace: rollout-test
spec:
  sessionAffinity: None
  selector:
    app: rollout-bluegreen
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: LoadBalancer
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
上次更新: 2024/04/09, 16:48:42
Jenkins-Argocd CICD-下
Jenkins-Argocd CICD Rollouts金丝雀发布优化

← Jenkins-Argocd CICD Rollouts金丝雀发布优化→

最近更新
01
AWS NAT-NetWork-Firwalld配置(一)
04-09
02
AWS NAT-NetWork-Firwalld配置(二)
04-09
03
kubernetes部署minio对象存储
01-18
更多文章>
Theme by Vdoing | Copyright © 2019-2024 Bruce Tao Blog Space | MIT License