使用cri工具运行容器组

前言

刚开始用cri工具时，一直被配置文件绕得云里雾里，因为一直找不到相关教程，就搁置下来了。

结果在调了几个月CRI的接口后，昨天才反应过来，crictl工具要求配置文件参数都在API里：

https://github.com/kubernetes/cri-api/blob/master/pkg/apis/runtime/v1alpha2/api.pb.go

照着源码中的json字段编写json配置文件就可以运行容器组了，操作逻辑和调接口是一样的，下面用containerd-cri代替docker/docker-ce来运行wordpress。

测试环境

系统配置

系统：CentOS 7.9.2009 64位
CRI：containerd-cri v1.4.3

软件配置

原有的docker/docker-ce/containerd等，需要全部删除，避免和接下来安装的containerd-cri文件夹、数据产生冲突。

1

yum autoremove docker docker-ce docker-ce-cli containerd.io docker-compose

containerd-cri可以直接在release页面下载到：

https://github.com/containerd/containerd/tags

这里使用整合了cri工具和cni插件的v1.4.3版本：

1

curl -LO https://github.com/containerd/containerd/releases/download/v1.4.3/cri-containerd-cni-1.4.3-linux-amd64.tar.gz

解压到根目录直接安装

1

tar -zxvf cri-containerd-cni-1.4.3-linux-amd64.tar.gz -C /

启动服务

1

systemctl enable --now containerd

压缩包内文件占用的目录如下，若需要卸载时，关闭containerd服务后，删除相关文件即可：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50


.
├── etc
│   ├── cni
│   │   └── net.d
│   │       └── 10-containerd-net.conflist
│   ├── crictl.yaml
│   └── systemd
│       └── system
│           └── containerd.service
├── opt
│   ├── cni
│   │   └── bin
│   │       ├── bandwidth
│   │       ├── bridge
│   │       ├── dhcp
│   │       ├── firewall
│   │       ├── flannel
│   │       ├── host-device
│   │       ├── host-local
│   │       ├── ipvlan
│   │       ├── loopback
│   │       ├── macvlan
│   │       ├── portmap
│   │       ├── ptp
│   │       ├── sbr
│   │       ├── static
│   │       ├── tuning
│   │       └── vlan
│   └── containerd
│       └── cluster
│           ├── gce
│           │   ├── cloud-init
│           │   │   ├── master.yaml
│           │   │   └── node.yaml
│           │   ├── cni.template
│           │   ├── configure.sh
│           │   └── env
│           └── version
└── usr
    └── local
        ├── bin
        │   ├── containerd
        │   ├── containerd-shim
        │   ├── containerd-shim-runc-v1
        │   ├── containerd-shim-runc-v2
        │   ├── crictl
        │   ├── critest
        │   └── ctr
        └── sbin
            └── runc

crictl命令

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


➜  ~ crictl --help
NAME:
   crictl - client for CRI

USAGE:
   crictl [global options] command [command options] [arguments...]

VERSION:
   1.18.0-100-g2bf7674

COMMANDS:
   attach              Attach to a running container
   create              Create a new container
   exec                Run a command in a running container
   version             Display runtime version information
   images, image, img  List images
   inspect             Display the status of one or more containers
   inspecti            Return the status of one or more images
   imagefsinfo         Return image filesystem info
   inspectp            Display the status of one or more pods
   logs                Fetch the logs of a container
   port-forward        Forward local port to a pod
   ps                  List containers
   pull                Pull an image from a registry
   run                 Run a new container inside a sandbox
   runp                Run a new pod
   rm                  Remove one or more containers
   rmi                 Remove one or more images
   rmp                 Remove one or more pods
   pods                List pods
   start               Start one or more created containers
   info                Display information of the container runtime
   stop                Stop one or more running containers
   stopp               Stop one or more running pods
   update              Update one or more running containers
   config              Get and set crictl client configuration options
   stats               List container(s) resource usage statistics
   completion          Output shell completion code
   help, h             Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --config value, -c value            Location of the client config file. If not specified and the default does not exist, the program's directory is searched as well (default: "/etc/crictl.yaml") [$CRI_CONFIG_FILE]
   --debug, -D                         Enable debug mode (default: false)
   --image-endpoint value, -i value    Endpoint of CRI image manager service (default: uses 'runtime-endpoint' setting) [$IMAGE_SERVICE_ENDPOINT]
   --runtime-endpoint value, -r value  Endpoint of CRI container runtime service (default: uses in order the first successful one of [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]). Default is now deprecated and the endpoint should be set instead. [$CONTAINER_RUNTIME_ENDPOINT]
   --timeout value, -t value           Timeout of connecting to the server in seconds (e.g. 2s, 20s.). 0 or less is set to default (default: 2s)
   --help, -h                          show help (default: false)
   --version, -v                       print the version (default: false)

crictl命令代替了常用docker命令，以容器组的形式运行容器，默认读取 /etc/crictl.yaml 配置文件，可以添加debug参数输出详细信息：

1
2
3


➜  ~ cat /etc/crictl.yaml
runtime-endpoint: unix:///run/containerd/containerd.sock
debug: true

运行包含一个容器的容器组

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14


➜  ~ crictl run --help
NAME:
   crictl run - Run a new container inside a sandbox

USAGE:
   crictl run [command options] container-config.[json|yaml] pod-config.[json|yaml]

OPTIONS:
   --auth AUTH_STRING           Use AUTH_STRING for accessing the registry. AUTH_STRING is a base64 encoded 'USERNAME[:PASSWORD]'
   --creds USERNAME[:PASSWORD]  Use USERNAME[:PASSWORD] for accessing the registry
   --no-pull                    Do not pull the image (overrides disable-pull-on-run=false in config) (default: false)
   --runtime value, -r value    Runtime handler to use. Available options are defined by the container runtime.
   --with-pull                  Pull the image (overrides disable-pull-on-run=true in config) (default: false)
   --help, -h                   show help (default: false)

假设有两个编写好容器配置文件 container-config.json 和沙箱配置文件 pod-config.json，执行命令：

1

crictl run container-config.json pod-config.json

crictl工具将顺序执行：

创建沙箱
拉取容器镜像
创建容器
启动容器

若镜像需要用户密码拉取，可以在命令行中使用auth或creds标志位传入。

运行包含多个容器的容器组

这里需要拆解步骤，逐一手动执行：

创建沙箱：crictl runp pod-config.json，在输出信息中，取得沙箱ID，假设为64e44d802c1c5
拉取容器镜像：crictl pull 镜像名，拉取所需镜像
创建容器：crictl create 沙箱ID container-config.json pod-config.json，在输出信息中，取得容器ID，假设为c677fe6a45e20
启动容器：crictl start 容器ID

重复2~4步骤，启动所有容器

容器及沙箱配置文件

这里以配置wordpress容器组为例，由于编写配置文件时，需要从源码参考字段，最好先配代码跳转方便查看。

默认配置下，使用runc作为runtime，使用bridge和portmap插件提供内网地址和端口映射功能。

沙箱配置文件

沙箱配置文件的json字段对应结构体 PodSandboxConfig 的tag，如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55


// PodSandboxConfig holds all the required and optional fields for creating a
// sandbox.
type PodSandboxConfig struct {
	// Metadata of the sandbox. This information will uniquely identify the
	// sandbox, and the runtime should leverage this to ensure correct
	// operation. The runtime may also use this information to improve UX, such
	// as by constructing a readable name.
	Metadata *PodSandboxMetadata `protobuf:"bytes,1,opt,name=metadata,proto3" json:"metadata,omitempty"`
	// Hostname of the sandbox. Hostname could only be empty when the pod
	// network namespace is NODE.
	Hostname string `protobuf:"bytes,2,opt,name=hostname,proto3" json:"hostname,omitempty"`
	// Path to the directory on the host in which container log files are
	// stored.
	// By default the log of a container going into the LogDirectory will be
	// hooked up to STDOUT and STDERR. However, the LogDirectory may contain
	// binary log files with structured logging data from the individual
	// containers. For example, the files might be newline separated JSON
	// structured logs, systemd-journald journal files, gRPC trace files, etc.
	// E.g.,
	//     PodSandboxConfig.LogDirectory = `/var/log/pods/<podUID>/`
	//     ContainerConfig.LogPath = `containerName/Instance#.log`
	//
	// WARNING: Log management and how kubelet should interface with the
	// container logs are under active discussion in
	// https://issues.k8s.io/24677. There *may* be future change of direction
	// for logging as the discussion carries on.
	LogDirectory string `protobuf:"bytes,3,opt,name=log_directory,json=logDirectory,proto3" json:"log_directory,omitempty"`
	// DNS config for the sandbox.
	DnsConfig *DNSConfig `protobuf:"bytes,4,opt,name=dns_config,json=dnsConfig,proto3" json:"dns_config,omitempty"`
	// Port mappings for the sandbox.
	PortMappings []*PortMapping `protobuf:"bytes,5,rep,name=port_mappings,json=portMappings,proto3" json:"port_mappings,omitempty"`
	// Key-value pairs that may be used to scope and select individual resources.
	Labels map[string]string `protobuf:"bytes,6,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Unstructured key-value map that may be set by the kubelet to store and
	// retrieve arbitrary metadata. This will include any annotations set on a
	// pod through the Kubernetes API.
	//
	// Annotations MUST NOT be altered by the runtime; the annotations stored
	// here MUST be returned in the PodSandboxStatus associated with the pod
	// this PodSandboxConfig creates.
	//
	// In general, in order to preserve a well-defined interface between the
	// kubelet and the container runtime, annotations SHOULD NOT influence
	// runtime behaviour.
	//
	// Annotations can also be useful for runtime authors to experiment with
	// new features that are opaque to the Kubernetes APIs (both user-facing
	// and the CRI). Whenever possible, however, runtime authors SHOULD
	// consider proposing new typed fields for any new features instead.
	Annotations map[string]string `protobuf:"bytes,7,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Optional configurations specific to Linux hosts.
	Linux                *LinuxPodSandboxConfig `protobuf:"bytes,8,opt,name=linux,proto3" json:"linux,omitempty"`
	XXX_NoUnkeyedLiteral struct{}               `json:"-"`
	XXX_sizecache        int32                  `json:"-"`
}

参考字段编写一个用于wordpress的沙箱配置文件

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21


{
    "metadata": {
        "name": "blog-sandbox",
        "uid": "cdgbvafsbdvafbvfb",
        "namespace": "default",
        "attempt": 3
    },
    "hostname": "blog-sandbox",
    "log_directory": "/root/blog/log",
    "dns_config": {
        "servers": ["8.8.8.8","8.8.4.4"]
    },
    "port_mappings":[
        {
            "protocol":0,
            "container_port": 80,
            "host_port": 80,
            "host_ip": "public_ip"
        }
    ]
}

字段含义如下：

metadata：包含沙箱基本信息，需要确保name、uid在同一个namespace下唯一
hostname：沙箱主机名称
log_directory：日志目录，所有容器日志都会存放在该目录下，这里使用/root/blog/log作为根目录
dns_config：DNS配置，这里只设置了DNS服务器
port_mappings：端口映射，根据需求开启宿主机到容器的端口映射，每个配置最终对应一套iptables规则实现流量转发

容器配置文件

容器配置文件的json字段对应结构体 ContainerConfig 的tag，如下：

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65


// ContainerConfig holds all the required and optional fields for creating a
// container.
type ContainerConfig struct {
	// Metadata of the container. This information will uniquely identify the
	// container, and the runtime should leverage this to ensure correct
	// operation. The runtime may also use this information to improve UX, such
	// as by constructing a readable name.
	Metadata *ContainerMetadata `protobuf:"bytes,1,opt,name=metadata,proto3" json:"metadata,omitempty"`
	// Image to use.
	Image *ImageSpec `protobuf:"bytes,2,opt,name=image,proto3" json:"image,omitempty"`
	// Command to execute (i.e., entrypoint for docker)
	Command []string `protobuf:"bytes,3,rep,name=command,proto3" json:"command,omitempty"`
	// Args for the Command (i.e., command for docker)
	Args []string `protobuf:"bytes,4,rep,name=args,proto3" json:"args,omitempty"`
	// Current working directory of the command.
	WorkingDir string `protobuf:"bytes,5,opt,name=working_dir,json=workingDir,proto3" json:"working_dir,omitempty"`
	// List of environment variable to set in the container.
	Envs []*KeyValue `protobuf:"bytes,6,rep,name=envs,proto3" json:"envs,omitempty"`
	// Mounts for the container.
	Mounts []*Mount `protobuf:"bytes,7,rep,name=mounts,proto3" json:"mounts,omitempty"`
	// Devices for the container.
	Devices []*Device `protobuf:"bytes,8,rep,name=devices,proto3" json:"devices,omitempty"`
	// Key-value pairs that may be used to scope and select individual resources.
	// Label keys are of the form:
	//     label-key ::= prefixed-name | name
	//     prefixed-name ::= prefix '/' name
	//     prefix ::= DNS_SUBDOMAIN
	//     name ::= DNS_LABEL
	Labels map[string]string `protobuf:"bytes,9,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Unstructured key-value map that may be used by the kubelet to store and
	// retrieve arbitrary metadata.
	//
	// Annotations MUST NOT be altered by the runtime; the annotations stored
	// here MUST be returned in the ContainerStatus associated with the container
	// this ContainerConfig creates.
	//
	// In general, in order to preserve a well-defined interface between the
	// kubelet and the container runtime, annotations SHOULD NOT influence
	// runtime behaviour.
	Annotations map[string]string `protobuf:"bytes,10,rep,name=annotations,proto3" json:"annotations,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"`
	// Path relative to PodSandboxConfig.LogDirectory for container to store
	// the log (STDOUT and STDERR) on the host.
	// E.g.,
	//     PodSandboxConfig.LogDirectory = `/var/log/pods/<podUID>/`
	//     ContainerConfig.LogPath = `containerName/Instance#.log`
	//
	// WARNING: Log management and how kubelet should interface with the
	// container logs are under active discussion in
	// https://issues.k8s.io/24677. There *may* be future change of direction
	// for logging as the discussion carries on.
	LogPath string `protobuf:"bytes,11,opt,name=log_path,json=logPath,proto3" json:"log_path,omitempty"`
	// Variables for interactive containers, these have very specialized
	// use-cases (e.g. debugging).
	// TODO: Determine if we need to continue supporting these fields that are
	// part of Kubernetes's Container Spec.
	Stdin     bool `protobuf:"varint,12,opt,name=stdin,proto3" json:"stdin,omitempty"`
	StdinOnce bool `protobuf:"varint,13,opt,name=stdin_once,json=stdinOnce,proto3" json:"stdin_once,omitempty"`
	Tty       bool `protobuf:"varint,14,opt,name=tty,proto3" json:"tty,omitempty"`
	// Configuration specific to Linux containers.
	Linux *LinuxContainerConfig `protobuf:"bytes,15,opt,name=linux,proto3" json:"linux,omitempty"`
	// Configuration specific to Windows containers.
	Windows              *WindowsContainerConfig `protobuf:"bytes,16,opt,name=windows,proto3" json:"windows,omitempty"`
	XXX_NoUnkeyedLiteral struct{}                `json:"-"`
	XXX_sizecache        int32                   `json:"-"`
}

参考字段分别编写wordpress和mysql容器配置文件

container-wordpress.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


{
    "metadata":{
        "name":"wordpress",
        "attempt": 3
    },
    "image": {
        "image": "wordpress:latest"
    },
    "envs": [
        {
            "key": "WORDPRESS_DB_HOST",
            "value": "127.0.0.1"
        },
        {
            "key": "WORDPRESS_DB_USER",
            "value": "wordpress"
        },
        {
            "key": "WORDPRESS_DB_PASSWORD",
            "value": "wordpress"
        },
        {
            "key": "WORDPRESS_DB_NAME",
            "value": "wordpress"
        }
    ],
    "mounts": [
        {
            "container_path": "/var/www/html",
            "host_path": "/root/blog/wordpress"
        }
    ],
    "log_path": "wp.log"
}

container-mysql.json

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34


{
    "metadata":{
        "name":"mysql",
        "attempt": 3
    },
    "image": {
        "image": "mysql:5.7"
    },
    "envs": [
        {
            "key": "MYSQL_DATABASE",
            "value": "wordpress"
        },
        {
            "key": "MYSQL_USER",
            "value": "wordpress"
        },
        {
            "key": "MYSQL_PASSWORD",
            "value": "wordpress"
        },
        {
            "key": "MYSQL_RANDOM_ROOT_PASSWORD",
            "value": "1"
        }
    ],
    "mounts": [
        {
            "container_path": "/var/lib/mysql",
            "host_path": "/root/blog/mysql"
        }
    ],
    "log_path": "db.log"
}

字段含义如下：

metadata：包含容器基本信息
image：镜像配置
envs：环境变量
mounts：挂载路径，这里在/root/blog下分别创建wordpress和mysql目录存放容器数据
log_path：相对日志路径，和沙箱中的日志目录/root/blog/log组合起来为容器日志文件绝对路径

安装上一个章节的流程，就可以运行wordpress了：

1
2
3


CONTAINER           IMAGE                   CREATED             STATE               NAME                ATTEMPT             POD ID
c677fe6a45e20       wordpress:latest        13 hours ago        Running             wordpress                  3                   c3aa934fd0cb4
19d7625c26a6c       mysql:5.7               14 hours ago        Running             mysql                  3                   c3aa934fd0cb4

cri与docker

cri是k8s的接口设计，并不是面向docker用户的，因此缺乏也很多预期的docker功能，如镜像管理、日志切割、自动重启、resetful接口、docker-compose等等
cri工具只能使用配置文件运行容器组（运行单一容器使用ctr命令），与常用的docker-compose运行多个容器不同，所有容器运行在同一个网络命名空间下，因此使用环回地址即可访问同一沙箱下的容器
在k8s外，使用cri来代替docker，目前看来不是可行的方案

文章目录

前言

测试环境

crictl命令

容器及沙箱配置文件

cri与docker