• ConfigMap的热更新
    • 测试示例
    • 代码
    • 测试
      • 更新使用ConfigMap挂载的Env
      • 更新使用ConfigMap挂载的Volume
    • ConfigMap 更新后滚动更新 Pod
    • 总结
    • 参考

    ConfigMap的热更新

    ConfigMap是用来存储配置文件的kubernetes资源对象,所有的配置内容都存储在etcd中,下文主要是探究 ConfigMap 的创建和更新流程,以及对 ConfigMap 更新后容器内挂载的内容是否同步更新的测试。

    测试示例

    假设我们在 default namespace 下有一个名为 nginx-config 的 ConfigMap,可以使用 kubectl命令来获取:

    1. $ kubectl get configmap nginx-config
    2. NAME DATA AGE
    3. nginx-config 1 99d

    获取该ConfigMap的内容。

    1. kubectl get configmap nginx-config -o yaml
    1. apiVersion: v1
    2. data:
    3. nginx.conf: |-
    4. worker_processes 1;
    5. events { worker_connections 1024; }
    6. http {
    7. sendfile on;
    8. server {
    9. listen 80;
    10. # a test endpoint that returns http 200s
    11. location / {
    12. proxy_pass http://httpstat.us/200;
    13. proxy_set_header X-Real-IP $remote_addr;
    14. }
    15. }
    16. server {
    17. listen 80;
    18. server_name api.hello.world;
    19. location / {
    20. proxy_pass http://l5d.default.svc.cluster.local;
    21. proxy_set_header Host $host;
    22. proxy_set_header Connection "";
    23. proxy_http_version 1.1;
    24. more_clear_input_headers 'l5d-ctx-*' 'l5d-dtab' 'l5d-sample';
    25. }
    26. }
    27. server {
    28. listen 80;
    29. server_name www.hello.world;
    30. location / {
    31. # allow 'employees' to perform dtab overrides
    32. if ($cookie_special_employee_cookie != "letmein") {
    33. more_clear_input_headers 'l5d-ctx-*' 'l5d-dtab' 'l5d-sample';
    34. }
    35. # add a dtab override to get people to our beta, world-v2
    36. set $xheader "";
    37. if ($cookie_special_employee_cookie ~* "dogfood") {
    38. set $xheader "/host/world => /srv/world-v2;";
    39. }
    40. proxy_set_header 'l5d-dtab' $xheader;
    41. proxy_pass http://l5d.default.svc.cluster.local;
    42. proxy_set_header Host $host;
    43. proxy_set_header Connection "";
    44. proxy_http_version 1.1;
    45. }
    46. }
    47. }
    48. kind: ConfigMap
    49. metadata:
    50. creationTimestamp: 2017-08-01T06:53:17Z
    51. name: nginx-config
    52. namespace: default
    53. resourceVersion: "14925806"
    54. selfLink: /api/v1/namespaces/default/configmaps/nginx-config
    55. uid: 18d70527-7686-11e7-bfbd-8af1e3a7c5bd

    ConfigMap中的内容是存储到etcd中的,然后查询etcd:

    1. ETCDCTL_API=3 etcdctl get /registry/configmaps/default/nginx-config -w json|python -m json.tool

    注意使用 v3 版本的 etcdctl API,下面是输出结果:

    1. {
    2. "count": 1,
    3. "header": {
    4. "cluster_id": 12091028579527406772,
    5. "member_id": 16557816780141026208,
    6. "raft_term": 36,
    7. "revision": 29258723
    8. },
    9. "kvs": [
    10. {
    11. "create_revision": 14925806,
    12. "key": "L3JlZ2lzdHJ5L2NvbmZpZ21hcHMvZGVmYXVsdC9uZ2lueC1jb25maWc=",
    13. "mod_revision": 14925806,
    14. "value": "azhzAAoPCgJ2MRIJQ29uZmlnTWFwEqQMClQKDG5naW54LWNvbmZpZxIAGgdkZWZhdWx0IgAqJDE4ZDcwNTI3LTc2ODYtMTFlNy1iZmJkLThhZjFlM2E3YzViZDIAOABCCwjdyoDMBRC5ss54egASywsKCm5naW54LmNvbmYSvAt3b3JrZXJfcHJvY2Vzc2VzIDE7CgpldmVudHMgeyB3b3JrZXJfY29ubmVjdGlvbnMgMTAyNDsgfQoKaHR0cCB7CiAgICBzZW5kZmlsZSBvbjsKCiAgICBzZXJ2ZXIgewogICAgICAgIGxpc3RlbiA4MDsKCiAgICAgICAgIyBhIHRlc3QgZW5kcG9pbnQgdGhhdCByZXR1cm5zIGh0dHAgMjAwcwogICAgICAgIGxvY2F0aW9uIC8gewogICAgICAgICAgICBwcm94eV9wYXNzIGh0dHA6Ly9odHRwc3RhdC51cy8yMDA7CiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgIFgtUmVhbC1JUCAgJHJlbW90ZV9hZGRyOwogICAgICAgIH0KICAgIH0KCiAgICBzZXJ2ZXIgewoKICAgICAgICBsaXN0ZW4gODA7CiAgICAgICAgc2VydmVyX25hbWUgYXBpLmhlbGxvLndvcmxkOwoKICAgICAgICBsb2NhdGlvbiAvIHsKICAgICAgICAgICAgcHJveHlfcGFzcyBodHRwOi8vbDVkLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWw7CiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgSG9zdCAkaG9zdDsKICAgICAgICAgICAgcHJveHlfc2V0X2hlYWRlciBDb25uZWN0aW9uICIiOwogICAgICAgICAgICBwcm94eV9odHRwX3ZlcnNpb24gMS4xOwoKICAgICAgICAgICAgbW9yZV9jbGVhcl9pbnB1dF9oZWFkZXJzICdsNWQtY3R4LSonICdsNWQtZHRhYicgJ2w1ZC1zYW1wbGUnOwogICAgICAgIH0KICAgIH0KCiAgICBzZXJ2ZXIgewoKICAgICAgICBsaXN0ZW4gODA7CiAgICAgICAgc2VydmVyX25hbWUgd3d3LmhlbGxvLndvcmxkOwoKICAgICAgICBsb2NhdGlvbiAvIHsKCgogICAgICAgICAgICAjIGFsbG93ICdlbXBsb3llZXMnIHRvIHBlcmZvcm0gZHRhYiBvdmVycmlkZXMKICAgICAgICAgICAgaWYgKCRjb29raWVfc3BlY2lhbF9lbXBsb3llZV9jb29raWUgIT0gImxldG1laW4iKSB7CiAgICAgICAgICAgICAgbW9yZV9jbGVhcl9pbnB1dF9oZWFkZXJzICdsNWQtY3R4LSonICdsNWQtZHRhYicgJ2w1ZC1zYW1wbGUnOwogICAgICAgICAgICB9CgogICAgICAgICAgICAjIGFkZCBhIGR0YWIgb3ZlcnJpZGUgdG8gZ2V0IHBlb3BsZSB0byBvdXIgYmV0YSwgd29ybGQtdjIKICAgICAgICAgICAgc2V0ICR4aGVhZGVyICIiOwoKICAgICAgICAgICAgaWYgKCRjb29raWVfc3BlY2lhbF9lbXBsb3llZV9jb29raWUgfiogImRvZ2Zvb2QiKSB7CiAgICAgICAgICAgICAgc2V0ICR4aGVhZGVyICIvaG9zdC93b3JsZCA9PiAvc3J2L3dvcmxkLXYyOyI7CiAgICAgICAgICAgIH0KCiAgICAgICAgICAgIHByb3h5X3NldF9oZWFkZXIgJ2w1ZC1kdGFiJyAkeGhlYWRlcjsKCgogICAgICAgICAgICBwcm94eV9wYXNzIGh0dHA6Ly9sNWQuZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbDsKICAgICAgICAgICAgcHJveHlfc2V0X2hlYWRlciBIb3N0ICRob3N0OwogICAgICAgICAgICBwcm94eV9zZXRfaGVhZGVyIENvbm5lY3Rpb24gIiI7CiAgICAgICAgICAgIHByb3h5X2h0dHBfdmVyc2lvbiAxLjE7CiAgICAgICAgfQogICAgfQp9GgAiAA==",
    15. "version": 1
    16. }
    17. ]
    18. }

    其中的value就是 nginx.conf 配置文件的内容。

    可以使用base64解码查看具体值,关于etcdctl的使用请参考使用etcdctl访问kuberentes数据。

    代码

    ConfigMap 结构体的定义:

    1. // ConfigMap holds configuration data for pods to consume.
    2. type ConfigMap struct {
    3. metav1.TypeMeta `json:",inline"`
    4. // Standard object's metadata.
    5. // More info: http://releases.k8s.io/HEAD/docs/devel/api-conventions.md#metadata
    6. // +optional
    7. metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
    8. // Data contains the configuration data.
    9. // Each key must be a valid DNS_SUBDOMAIN with an optional leading dot.
    10. // +optional
    11. Data map[string]string `json:"data,omitempty" protobuf:"bytes,2,rep,name=data"`
    12. }

    staging/src/k8s.io/client-go/kubernetes/typed/core/v1/configmap.go 中ConfigMap 的接口定义:

    1. // ConfigMapInterface has methods to work with ConfigMap resources.
    2. type ConfigMapInterface interface {
    3. Create(*v1.ConfigMap) (*v1.ConfigMap, error)
    4. Update(*v1.ConfigMap) (*v1.ConfigMap, error)
    5. Delete(name string, options *meta_v1.DeleteOptions) error
    6. DeleteCollection(options *meta_v1.DeleteOptions, listOptions meta_v1.ListOptions) error
    7. Get(name string, options meta_v1.GetOptions) (*v1.ConfigMap, error)
    8. List(opts meta_v1.ListOptions) (*v1.ConfigMapList, error)
    9. Watch(opts meta_v1.ListOptions) (watch.Interface, error)
    10. Patch(name string, pt types.PatchType, data []byte, subresources ...string) (result *v1.ConfigMap, err error)
    11. ConfigMapExpansion
    12. }

    staging/src/k8s.io/client-go/kubernetes/typed/core/v1/configmap.go 中创建 ConfigMap 的方法如下:

    1. // Create takes the representation of a configMap and creates it. Returns the server's representation of the configMap, and an error, if there is any.
    2. func (c *configMaps) Create(configMap *v1.ConfigMap) (result *v1.ConfigMap, err error) {
    3. result = &v1.ConfigMap{}
    4. err = c.client.Post().
    5. Namespace(c.ns).
    6. Resource("configmaps").
    7. Body(configMap).
    8. Do().
    9. Into(result)
    10. return
    11. }

    通过 RESTful 请求在 etcd 中存储 ConfigMap 的配置,该方法中设置了资源对象的 namespace 和 HTTP 请求中的 body,执行后将请求结果保存到 result 中返回给调用者。

    注意 Body 的结构

    1. // Body makes the request use obj as the body. Optional.
    2. // If obj is a string, try to read a file of that name.
    3. // If obj is a []byte, send it directly.
    4. // If obj is an io.Reader, use it directly.
    5. // If obj is a runtime.Object, marshal it correctly, and set Content-Type header.
    6. // If obj is a runtime.Object and nil, do nothing.
    7. // Otherwise, set an error.

    创建 ConfigMap RESTful 请求中的的 Body 中包含 ObjectMetanamespace

    HTTP 请求中的结构体:

    1. // Request allows for building up a request to a server in a chained fashion.
    2. // Any errors are stored until the end of your call, so you only have to
    3. // check once.
    4. type Request struct {
    5. // required
    6. client HTTPClient
    7. verb string
    8. baseURL *url.URL
    9. content ContentConfig
    10. serializers Serializers
    11. // generic components accessible via method setters
    12. pathPrefix string
    13. subpath string
    14. params url.Values
    15. headers http.Header
    16. // structural elements of the request that are part of the Kubernetes API conventions
    17. namespace string
    18. namespaceSet bool
    19. resource string
    20. resourceName string
    21. subresource string
    22. timeout time.Duration
    23. // output
    24. err error
    25. body io.Reader
    26. // This is only used for per-request timeouts, deadlines, and cancellations.
    27. ctx context.Context
    28. backoffMgr BackoffManager
    29. throttle flowcontrol.RateLimiter
    30. }

    测试

    分别测试使用 ConfigMap 挂载 Env 和 Volume 的情况。

    更新使用ConfigMap挂载的Env

    使用下面的配置创建 nginx 容器测试更新 ConfigMap 后容器内的环境变量是否也跟着更新。

    1. apiVersion: extensions/v1beta1
    2. kind: Deployment
    3. metadata:
    4. name: my-nginx
    5. spec:
    6. replicas: 1
    7. template:
    8. metadata:
    9. labels:
    10. run: my-nginx
    11. spec:
    12. containers:
    13. - name: my-nginx
    14. image: harbor-001.jimmysong.io/library/nginx:1.9
    15. ports:
    16. - containerPort: 80
    17. envFrom:
    18. - configMapRef:
    19. name: env-config
    20. ---
    21. apiVersion: v1
    22. kind: ConfigMap
    23. metadata:
    24. name: env-config
    25. namespace: default
    26. data:
    27. log_level: INFO

    获取环境变量的值

    1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` env|grep log_level
    2. log_level=INFO

    修改 ConfigMap

    1. $ kubectl edit configmap env-config

    修改 log_level 的值为 DEBUG

    再次查看环境变量的值。

    1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` env|grep log_level
    2. log_level=INFO

    实践证明修改 ConfigMap 无法更新容器中已注入的环境变量信息。

    更新使用ConfigMap挂载的Volume

    使用下面的配置创建 nginx 容器测试更新 ConfigMap 后容器内挂载的文件是否也跟着更新。

    1. apiVersion: extensions/v1beta1
    2. kind: Deployment
    3. metadata:
    4. name: my-nginx
    5. spec:
    6. replicas: 1
    7. template:
    8. metadata:
    9. labels:
    10. run: my-nginx
    11. spec:
    12. containers:
    13. - name: my-nginx
    14. image: harbor-001.jimmysong.io/library/nginx:1.9
    15. ports:
    16. - containerPort: 80
    17. volumeMounts:
    18. - name: config-volume
    19. mountPath: /etc/config
    20. volumes:
    21. - name: config-volume
    22. configMap:
    23. name: special-config
    24. ---
    25. apiVersion: v1
    26. kind: ConfigMap
    27. metadata:
    28. name: special-config
    29. namespace: default
    30. data:
    31. log_level: INFO
    1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` cat /etc/config/log_level
    2. INFO

    修改 ConfigMap

    1. $ kubectl edit configmap special-config

    修改 log_level 的值为 DEBUG

    等待大概10秒钟时间,再次查看环境变量的值。

    1. $ kubectl exec `kubectl get pods -l run=my-nginx -o=name|cut -d "/" -f2` cat /tmp/log_level
    2. DEBUG

    我们可以看到使用 ConfigMap 方式挂载的 Volume 的文件中的内容已经变成了 DEBUG

    Known Issue:如果使用ConfigMap的subPath挂载为Container的Volume,Kubernetes不会做自动热更新:https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically

    ConfigMap 更新后滚动更新 Pod

    更新 ConfigMap 目前并不会触发相关 Pod 的滚动更新,可以通过修改 pod annotations 的方式强制触发滚动更新。

    1. $ kubectl patch deployment my-nginx --patch '{"spec": {"template": {"metadata": {"annotations": {"version/config": "20180411" }}}}}'

    这个例子里我们在 .spec.template.metadata.annotations 中添加 version/config,每次通过修改 version/config 来触发滚动更新。

    总结

    更新 ConfigMap 后:

    • 使用该 ConfigMap 挂载的 Env 不会同步更新
    • 使用该 ConfigMap 挂载的 Volume 中的数据需要一段时间(实测大概10秒)才能同步更新

    ENV 是在容器启动的时候注入的,启动之后 kubernetes 就不会再改变环境变量的值,且同一个 namespace 中的 pod 的环境变量是不断累加的,参考 Kubernetes中的服务发现与docker容器间的环境变量传递源码探究。为了更新容器中使用 ConfigMap 挂载的配置,需要通过滚动更新 pod 的方式来强制重新挂载 ConfigMap。

    参考

    • Kubernetes 1.7 security in practice
    • ConfigMap | kubernetes handbook - jimmysong.io
    • 创建高可用ectd集群 | Kubernetes handbook - jimmysong.io
    • Kubernetes中的服务发现与docker容器间的环境变量传递源码探究
    • Automatically Roll Deployments When ConfigMaps or Secrets change