使用cri工具运行容器组

文章目录

前言

刚开始用cri工具时,一直被配置文件绕得云里雾里,因为一直找不到相关教程,就搁置下来了。

结果在调了几个月CRI的接口后,昨天才反应过来,crictl工具要求配置文件参数都在API里:

https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1alpha2/api.pb.go

照着源码中的json字段编写json配置文件就可以运行容器组了,操作逻辑和调接口是一样的,下面用containerd-cri代替docker/docker-ce来运行wordpress。

测试环境

系统配置

  • 系统:CentOS 7.9.2009 64位
  • CRI:containerd-cri v1.4.3

软件配置

原有的docker/docker-ce/containerd等,需要全部删除,避免和接下来安装的containerd-cri文件夹、数据产生冲突。

1yum autoremove docker docker-ce docker-ce-cli containerd.io docker-compose 

containerd-cri可以直接在release页面下载到:

https://github.com/containerd/containerd/tags

这里使用整合了cri工具和cni插件的v1.4.3版本:

1curl -LO https://github.com/containerd/containerd/releases/download/v1.4.3/cri-containerd-cni-1.4.3-linux-amd64.tar.gz

解压到根目录直接安装

1tar -zxvf cri-containerd-cni-1.4.3-linux-amd64.tar.gz -C /

启动服务

1systemctl enable --now containerd

压缩包内文件占用的目录如下,若需要卸载时,关闭containerd服务后,删除相关文件即可:

 1.
 2├── etc
 3│   ├── cni
 4│   │   └── net.d
 5│   │       └── 10-containerd-net.conflist
 6│   ├── crictl.yaml
 7│   └── systemd
 8│       └── system
 9│           └── containerd.service
10├── opt
11│   ├── cni
12│   │   └── bin
13│   │       ├── bandwidth
14│   │       ├── bridge
15│   │       ├── dhcp
16│   │       ├── firewall
17│   │       ├── flannel
18│   │       ├── host-device
19│   │       ├── host-local
20│   │       ├── ipvlan
21│   │       ├── loopback
22│   │       ├── macvlan
23│   │       ├── portmap
24│   │       ├── ptp
25│   │       ├── sbr
26│   │       ├── static
27│   │       ├── tuning
28│   │       └── vlan
29│   └── containerd
30│       └── cluster
31│           ├── gce
32│           │   ├── cloud-init
33│           │   │   ├── master.yaml
34│           │   │   └── node.yaml
35│           │   ├── cni.template
36│           │   ├── configure.sh
37│           │   └── env
38│           └── version
39└── usr
40    └── local
41        ├── bin
42        │   ├── containerd
43        │   ├── containerd-shim
44        │   ├── containerd-shim-runc-v1
45        │   ├── containerd-shim-runc-v2
46        │   ├── crictl
47        │   ├── critest
48        │   └── ctr
49        └── sbin
50            └── runc

crictl命令

 1➜  ~ crictl --help
 2NAME:
 3   crictl - client for CRI
 4
 5USAGE:
 6   crictl [global options] command [command options] [arguments...]
 7
 8VERSION:
 9   1.18.0-100-g2bf7674
10
11COMMANDS:
12   attach              Attach to a running container
13   create              Create a new container
14   exec                Run a command in a running container
15   version             Display runtime version information
16   images, image, img  List images
17   inspect             Display the status of one or more containers
18   inspecti            Return the status of one or more images
19   imagefsinfo         Return image filesystem info
20   inspectp            Display the status of one or more pods
21   logs                Fetch the logs of a container
22   port-forward        Forward local port to a pod
23   ps                  List containers
24   pull                Pull an image from a registry
25   run                 Run a new container inside a sandbox
26   runp                Run a new pod
27   rm                  Remove one or more containers
28   rmi                 Remove one or more images
29   rmp                 Remove one or more pods
30   pods                List pods
31   start               Start one or more created containers
32   info                Display information of the container runtime
33   stop                Stop one or more running containers
34   stopp               Stop one or more running pods
35   update              Update one or more running containers
36   config              Get and set crictl client configuration options
37   stats               List container(s) resource usage statistics
38   completion          Output shell completion code
39   help, h             Shows a list of commands or help for one command
40
41GLOBAL OPTIONS:
42   --config value, -c value            Location of the client config file. If not specified and the default does not exist, the program's directory is searched as well (default: "/etc/crictl.yaml") [$CRI_CONFIG_FILE]
43   --debug, -D                         Enable debug mode (default: false)
44   --image-endpoint value, -i value    Endpoint of CRI image manager service (default: uses 'runtime-endpoint' setting) [$IMAGE_SERVICE_ENDPOINT]
45   --runtime-endpoint value, -r value  Endpoint of CRI container runtime service (default: uses in order the first successful one of [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]). Default is now deprecated and the endpoint should be set instead. [$CONTAINER_RUNTIME_ENDPOINT]
46   --timeout value, -t value           Timeout of connecting to the server in seconds (e.g. 2s, 20s.). 0 or less is set to default (default: 2s)
47   --help, -h                          show help (default: false)
48   --version, -v                       print the version (default: false)

crictl命令代替了常用docker命令,以容器组的形式运行容器,默认读取 /etc/crictl.yaml 配置文件,可以添加debug参数输出详细信息:

1➜  ~ cat /etc/crictl.yaml
2runtime-endpoint: unix:///run/containerd/containerd.sock
3debug: true

运行包含一个容器的容器组

 1➜  ~ crictl run --help
 2NAME:
 3   crictl run - Run a new container inside a sandbox
 4
 5USAGE:
 6   crictl run [command options] container-config.[json|yaml] pod-config.[json|yaml]
 7
 8OPTIONS:
 9   --auth AUTH_STRING           Use AUTH_STRING for accessing the registry. AUTH_STRING is a base64 encoded 'USERNAME[:PASSWORD]'
10   --creds USERNAME[:PASSWORD]  Use USERNAME[:PASSWORD] for accessing the registry
11   --no-pull                    Do not pull the image (overrides disable-pull-on-run=false in config) (default: false)
12   --runtime value, -r value    Runtime handler to use. Available options are defined by the container runtime.
13   --with-pull                  Pull the image (overrides disable-pull-on-run=true in config) (default: false)
14   --help, -h                   show help (default: false)

假设有两个编写好容器配置文件 container-config.json 和沙箱配置文件 pod-config.json,执行命令:

1crictl run container-config.json pod-config.json

crictl工具将顺序执行:

  • 创建沙箱
  • 拉取容器镜像
  • 创建容器
  • 启动容器

若镜像需要用户密码拉取,可以在命令行中使用auth或creds标志位传入。

运行包含多个容器的容器组

这里需要拆解步骤,逐一手动执行:

  • 创建沙箱:crictl runp pod-config.json,在输出信息中,取得沙箱ID,假设为64e44d802c1c5
  • 拉取容器镜像:crictl pull 镜像名,拉取所需镜像
  • 创建容器:crictl create 沙箱ID container-config.json pod-config.json,在输出信息中,取得容器ID,假设为c677fe6a45e20
  • 启动容器:crictl start 容器ID

重复2~4步骤,启动所有容器

容器及沙箱配置文件

这里以配置wordpress容器组为例,由于编写配置文件时,需要从源码参考字段,最好先配代码跳转方便查看。

默认配置下,使用runc作为runtime,使用bridge和portmap插件提供内网地址和端口映射功能。

沙箱配置文件

沙箱配置文件的json字段对应结构体 PodSandboxConfig 的tag,如下:

 1// PodSandboxConfig holds all the required and optional fields for creating a
 2// sandbox.
 3type PodSandboxConfig struct {
 4	// Metadata of the sandbox. This information will uniquely identify the
 5	// sandbox, and the runtime should leverage this to ensure correct
 6	// operation. The runtime may also use this information to improve UX, such
 7	// as by constructing a readable name.
 8	Metadata *PodSandboxMetadata `protobuf:"bytes,1,opt,name=metadata,proto3" json:"metadata,omitempty"`
 9	// Hostname of the sandbox. Hostname could only be empty when the pod
10	// network namespace is NODE.
11	Hostname string `protobuf:"bytes,2,opt,name=hostname,proto3" json:"hostname,omitempty"`
12	// Path to the directory on the host in which container log files are
13	// stored.
14	// By default the log of a container going into the LogDirectory will be
15	// hooked up to STDOUT and STDERR. However, the LogDirectory may contain
16	// binary log files with structured logging data from the individual
17	// containers. For example, the files might be newline separated JSON
18	// structured logs, systemd-journald journal files, gRPC trace files, etc.
19	// E.g.,
20	//     PodSandboxConfig.LogDirectory = `/var/log/pods/<podUID>/`
21	//     ContainerConfig.LogPath = `containerName/Instance#.log`
22	//
23	// WARNING: Log management and how kubelet should interface with the
24	// container logs are under active discussion in
25	// https://issues.k8s.io/24677. There *may* be future change of direction
26	// for logging as the discussion carries on.
27	LogDirectory string `protobuf:"bytes,3,opt,name=log_directory,json=logDirectory,proto3" json:"log_directory,omitempty"`
28	// DNS config for the sandbox.
29	DnsConfig *DNSConfig `protobuf:"bytes,4,opt,name=dns_config,json=dnsConfig,proto3" json:"dns_config,omitempty"`
30	// Port mappings for the sandbox.
31	PortMappings []*PortMapping `protobuf:"bytes,5,rep,name=port_mappings,json=portMappings,proto3" json:"port_mappings,omitempty"`
32	// Key-value pairs that may be used to scope and select individual resources.
33	Labels map[string]string `protobuf:"bytes,6,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
34	// Unstructured key-value map that may be set by the kubelet to store and
35	// retrieve arbitrary metadata. This will include any annotations set on a
36	// pod through the Kubernetes API.
37	//
38	// Annotations MUST NOT be altered by the runtime; the annotations stored
39	// here MUST be returned in the PodSandboxStatus associated with the pod
40	// this PodSandboxConfig creates.
41	//
42	// In general, in order to preserve a well-defined interface between the
43	// kubelet and the container runtime, annotations SHOULD NOT influence
44	// runtime behaviour.
45	//
46	// Annotations can also be useful for runtime authors to experiment with
47	// new features that are opaque to the Kubernetes APIs (both user-facing
48	// and the CRI). Whenever possible, however, runtime authors SHOULD
49	// consider proposing new typed fields for any new features instead.
50	Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
51	// Optional configurations specific to Linux hosts.
52	Linux                *LinuxPodSandboxConfig `protobuf:"bytes,8,opt,name=linux,proto3" json:"linux,omitempty"`
53	XXX_NoUnkeyedLiteral struct{}               `json:"-"`
54	XXX_sizecache        int32                  `json:"-"`
55}

参考字段编写一个用于wordpress的沙箱配置文件

 1{
 2    "metadata": {
 3        "name": "blog-sandbox",
 4        "uid": "cdgbvafsbdvafbvfb",
 5        "namespace": "default",
 6        "attempt": 3
 7    },
 8    "hostname": "blog-sandbox",
 9    "log_directory": "/root/blog/log",
10    "dns_config": {
11        "servers": ["8.8.8.8","8.8.4.4"]
12    },
13    "port_mappings":[
14        {
15            "protocol":0,
16            "container_port": 80,
17            "host_port": 80,
18            "host_ip": "public_ip"
19        }
20    ]
21}

字段含义如下:

  • metadata:包含沙箱基本信息,需要确保name、uid在同一个namespace下唯一
  • hostname:沙箱主机名称
  • log_directory:日志目录,所有容器日志都会存放在该目录下,这里使用/root/blog/log作为根目录
  • dns_config:DNS配置,这里只设置了DNS服务器
  • port_mappings:端口映射,根据需求开启宿主机到容器的端口映射,每个配置最终对应一套iptables规则实现流量转发

容器配置文件

容器配置文件的json字段对应结构体 ContainerConfig 的tag,如下:

 1// ContainerConfig holds all the required and optional fields for creating a
 2// container.
 3type ContainerConfig struct {
 4	// Metadata of the container. This information will uniquely identify the
 5	// container, and the runtime should leverage this to ensure correct
 6	// operation. The runtime may also use this information to improve UX, such
 7	// as by constructing a readable name.
 8	Metadata *ContainerMetadata `protobuf:"bytes,1,opt,name=metadata,proto3" json:"metadata,omitempty"`
 9	// Image to use.
10	Image *ImageSpec `protobuf:"bytes,2,opt,name=image,proto3" json:"image,omitempty"`
11	// Command to execute (i.e., entrypoint for docker)
12	Command []string `protobuf:"bytes,3,rep,name=command,proto3" json:"command,omitempty"`
13	// Args for the Command (i.e., command for docker)
14	Args []string `protobuf:"bytes,4,rep,name=args,proto3" json:"args,omitempty"`
15	// Current working directory of the command.
16	WorkingDir string `protobuf:"bytes,5,opt,name=working_dir,json=workingDir,proto3" json:"working_dir,omitempty"`
17	// List of environment variable to set in the container.
18	Envs []*KeyValue `protobuf:"bytes,6,rep,name=envs,proto3" json:"envs,omitempty"`
19	// Mounts for the container.
20	Mounts []*Mount `protobuf:"bytes,7,rep,name=mounts,proto3" json:"mounts,omitempty"`
21	// Devices for the container.
22	Devices []*Device `protobuf:"bytes,8,rep,name=devices,proto3" json:"devices,omitempty"`
23	// Key-value pairs that may be used to scope and select individual resources.
24	// Label keys are of the form:
25	//     label-key ::= prefixed-name | name
26	//     prefixed-name ::= prefix '/' name
27	//     prefix ::= DNS_SUBDOMAIN
28	//     name ::= DNS_LABEL
29	Labels map[string]string `protobuf:"bytes,9,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
30	// Unstructured key-value map that may be used by the kubelet to store and
31	// retrieve arbitrary metadata.
32	//
33	// Annotations MUST NOT be altered by the runtime; the annotations stored
34	// here MUST be returned in the ContainerStatus associated with the container
35	// this ContainerConfig creates.
36	//
37	// In general, in order to preserve a well-defined interface between the
38	// kubelet and the container runtime, annotations SHOULD NOT influence
39	// runtime behaviour.
40	Annotations map[string]string `protobuf:"bytes,10,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
41	// Path relative to PodSandboxConfig.LogDirectory for container to store
42	// the log (STDOUT and STDERR) on the host.
43	// E.g.,
44	//     PodSandboxConfig.LogDirectory = `/var/log/pods/<podUID>/`
45	//     ContainerConfig.LogPath = `containerName/Instance#.log`
46	//
47	// WARNING: Log management and how kubelet should interface with the
48	// container logs are under active discussion in
49	// https://issues.k8s.io/24677. There *may* be future change of direction
50	// for logging as the discussion carries on.
51	LogPath string `protobuf:"bytes,11,opt,name=log_path,json=logPath,proto3" json:"log_path,omitempty"`
52	// Variables for interactive containers, these have very specialized
53	// use-cases (e.g. debugging).
54	// TODO: Determine if we need to continue supporting these fields that are
55	// part of Kubernetes's Container Spec.
56	Stdin     bool `protobuf:"varint,12,opt,name=stdin,proto3" json:"stdin,omitempty"`
57	StdinOnce bool `protobuf:"varint,13,opt,name=stdin_once,json=stdinOnce,proto3" json:"stdin_once,omitempty"`
58	Tty       bool `protobuf:"varint,14,opt,name=tty,proto3" json:"tty,omitempty"`
59	// Configuration specific to Linux containers.
60	Linux *LinuxContainerConfig `protobuf:"bytes,15,opt,name=linux,proto3" json:"linux,omitempty"`
61	// Configuration specific to Windows containers.
62	Windows              *WindowsContainerConfig `protobuf:"bytes,16,opt,name=windows,proto3" json:"windows,omitempty"`
63	XXX_NoUnkeyedLiteral struct{}                `json:"-"`
64	XXX_sizecache        int32                   `json:"-"`
65}

参考字段分别编写wordpress和mysql容器配置文件

container-wordpress.json

 1{
 2    "metadata":{
 3        "name":"wordpress",
 4        "attempt": 3
 5    },
 6    "image": {
 7        "image": "wordpress:latest"
 8    },
 9    "envs": [
10        {
11            "key": "WORDPRESS_DB_HOST",
12            "value": "127.0.0.1"
13        },
14        {
15            "key": "WORDPRESS_DB_USER",
16            "value": "wordpress"
17        },
18        {
19            "key": "WORDPRESS_DB_PASSWORD",
20            "value": "wordpress"
21        },
22        {
23            "key": "WORDPRESS_DB_NAME",
24            "value": "wordpress"
25        }
26    ],
27    "mounts": [
28        {
29            "container_path": "/var/www/html",
30            "host_path": "/root/blog/wordpress"
31        }
32    ],
33    "log_path": "wp.log"
34}

container-mysql.json

 1{
 2    "metadata":{
 3        "name":"mysql",
 4        "attempt": 3
 5    },
 6    "image": {
 7        "image": "mysql:5.7"
 8    },
 9    "envs": [
10        {
11            "key": "MYSQL_DATABASE",
12            "value": "wordpress"
13        },
14        {
15            "key": "MYSQL_USER",
16            "value": "wordpress"
17        },
18        {
19            "key": "MYSQL_PASSWORD",
20            "value": "wordpress"
21        },
22        {
23            "key": "MYSQL_RANDOM_ROOT_PASSWORD",
24            "value": "1"
25        }
26    ],
27    "mounts": [
28        {
29            "container_path": "/var/lib/mysql",
30            "host_path": "/root/blog/mysql"
31        }
32    ],
33    "log_path": "db.log"
34}

字段含义如下:

  • metadata:包含容器基本信息
  • image:镜像配置
  • envs:环境变量
  • mounts:挂载路径,这里在/root/blog下分别创建wordpress和mysql目录存放容器数据
  • log_path:相对日志路径,和沙箱中的日志目录/root/blog/log组合起来为容器日志文件绝对路径

安装上一个章节的流程,就可以运行wordpress了:

1CONTAINER           IMAGE                   CREATED             STATE               NAME                ATTEMPT             POD ID
2c677fe6a45e20       wordpress:latest        13 hours ago        Running             wordpress                  3                   c3aa934fd0cb4
319d7625c26a6c       mysql:5.7               14 hours ago        Running             mysql                  3                   c3aa934fd0cb4

cri与docker

  • cri是k8s的接口设计,并不是面向docker用户的,因此缺乏也很多预期的docker功能,如镜像管理、日志切割、自动重启、resetful接口、docker-compose等等
  • cri工具只能使用配置文件运行容器组(运行单一容器使用ctr命令),与常用的docker-compose运行多个容器不同,所有容器运行在同一个网络命名空间下,因此使用环回地址即可访问同一沙箱下的容器
  • 在k8s外,使用cri来代替docker,目前看来不是可行的方案