Etcd容器部署集群并启用TLS

发表于 2021-04-27 | 分类于技术

在实际生产环境中，etcd集群都是使用TLS加密通信的，那么应该如何部署一套使用TLS证书加密的etcd集群呢？

1. TLS证书制作

证书制作建议使用cfssl工具，当然openssl工具也行，不过相对比较繁琐，且由于Etcd对证书的检查比较严格，使用openssl工具时，CA配置容易遗漏关键项，我经过尝试还是放弃了。

以下证书配置的相关内容主要参考网络上的资料，并对错误的地方做了一些纠正。

1.1 下载cfssl

mkdir ~/bin
curl -s -L -o ~/bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
curl -s -L -o ~/bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
chmod +x ~/bin/{cfssl,cfssljson}
export PATH=$PATH:~/bin

上述两个工具的地址国内访问可能比较慢，有条件的可以翻墙解决。

1.2 签发CA

1.2.1 生成cfssl默认配置和CA的签名请求配置

mkdir ~/cfss
lcd ~/cfssl
cfssl print-defaults config > ca-config.json
cfssl print-defaults csr > ca-csr.json

1.2.2 更新CA配置选项

修改生成的默认配置ca-config.json，修改成如下所示：

{
    "signing": {
        "default": {
            "expiry": "43800h"
        },
        "profiles": {
            "server": {
                "expiry": "43800h",
                "usages": ["signing", "key encipherment", "server auth", "client auth"]
            },
            "client": {
                "expiry": "43800h",
                "usages": ["signing", "key encipherment", "client auth"]
            },
            "peer": {
                "expiry": "43800h",
                "usages": ["signing", "key encipherment", "server auth", "client auth"]
            }
        }
    }
}

profiles属性下有三个字段，分别是server, client, peer，分别对应需要创建的三套证书：服务端证书，客户端证书，点对点证书。

注意：
在集群部署的场景下，server中需要包含client auth用途。否则集群启动时会报出错误：certificate specifies an incompatible key usage

1.2.3 更新CA CSR配置选项

修改生成的默认配置ca-csr.json，修改成如下所示：

{
    "CN": "My CA",
    "key": {
        "algo": "rsa",
        "size": 2048
    },
    "names": [
        {
            "C": "CN",
            "L": "SZ",
            "ST": "JS",
            "O": "Hillstone",
            "OU": "Cloud"
        }
    ]
}

1.2.4 使用自签名的方式签发CA

cfssl gencert -initca ca-csr.json | cfssljson -bare ca -

执行上述命令后，将会生成以下文件：

ca-key.pem
ca.csr
ca.pem

1.3 创建服务端证书

1.3.1 生成服务端证书的签名请求配置

cfssl print-defaults csr > server.json

修改server.json的内容如下所示：

{
    "CN": "etcd-ssl-cluster-server",
    "hosts": [
        "10.182.51.82",
        "127.0.0.1"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SZ",
            "ST": "JS",
            "O": "Hillstone",
            "OU": "Cloud"
        }
    ]
}

这里需要填写hosts字段，hosts列表中需要包含etcd服务端的访问IP，如果多个etcd节点都共用这个证书，则需要把所有节点相关的访问IP都填进来。因为我是用的docker部署，容器对外提供服务都是使用了宿主机的IP，只不过映射到了不同的端口号，所以这里只填了1个IP。

注意：
hosts下的127.0.0.1是必须的

1.3.2 生成服务端证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=server server.json | cfssljson -bare server

执行上述命令后，即可得到以下文件：

server-key.pem
server.csr
server.pem

1.4 生成点对点证书

点对点证书也就是etcd节点之间通信用的证书，每个节点都应该创建一套该证书。

1.4.1 生成证书签名请求配置

cfssl print-defaults csr > node1.json

修改server.json的内容如下所示：

{
    "CN": "etcd-ssl-cluster-node1",
    "hosts": [
        "10.182.51.82",
        "172.28.0.1",
        "127.0.0.1",
        "node1"
    ],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SZ",
            "ST": "JS",
            "O": "Hillstone",
            "OU": "Cloud"
        }
    ]
}

在同一个宿主机上的容器间通信会通过一个网桥，这个网桥上的IP需要添加到hosts列表中，否则节点之间TLS通信会被拒绝。

注意：
我这里有三个节点，所以还创建了node2.json、node3.json

1.4.2 生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=peer node1.json | cfssljson -bare node1

执行上述命令后将得到：

node1-key.pem
node1.csr
node1.pem

注意：
同理，还需要生成node2、node3的证书

1.5 生成客户端证书

客户端证书是etcd客户端访问etcd集群所使用的的证书。

1.5.1 创建签名请求

cfssl print-defaults csr > client.json

修改client.json为如下内容：

{
    "CN": "etcd-ssl-cluster-client",
    "hosts": [""],
    "key": {
        "algo": "ecdsa",
        "size": 256
    },
    "names": [
        {
            "C": "CN",
            "L": "SZ",
            "ST": "JS",
            "O": "Hillstone",
            "OU": "Cloud"
        }
    ]
}

hosts列表留空即可。

1.5.2 生成证书

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=client client.json | cfssljson -bare client

2. 容器部署

通过上述步骤，证书已经准备完成，接下来开始创建etcd容器。

2.1 下载容器镜像

这里使用的是bitnami的etcd容器。

docker image pull bitnami/etcd:latest

2.2 创建docker-compose配置文件

首先需要构造一个目录结构，参考我的：

.
├── docker-compose.yml
├── node1
│   ├── ca-config.json
│   ├── ca.csr
│   ├── ca-csr.json
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── client.csr
│   ├── client.json
│   ├── client-key.pem
│   ├── client.pem
│   ├── etcd.conf.yml
│   ├── node1.csr
│   ├── node1.json
│   ├── node1-key.pem
│   ├── node1.pem
│   ├── server.csr
│   ├── server.json
│   ├── server-key.pem
│   └── server.pem
├── node2
│   ├── ca-config.json
│   ├── ca.csr
│   ├── ca-csr.json
│   ├── ca-key.pem
│   ├── ca.pem
│   ├── client.csr
│   ├── client.json
│   ├── client-key.pem
│   ├── client.pem
│   ├── etcd.conf.yml
│   ├── node2.csr
│   ├── node2.json
│   ├── node2-key.pem
│   ├── node2.pem
│   ├── server.csr
│   ├── server.json
│   ├── server-key.pem
│   └── server.pem
└── node3
    ├── ca-config.json
    ├── ca.csr
    ├── ca-csr.json
    ├── ca-key.pem
    ├── ca.pem
    ├── client.csr
    ├── client.json
    ├── client-key.pem
    ├── client.pem
    ├── etcd.conf.yml
    ├── node3.csr
    ├── node3.json
    ├── node3-key.pem
    ├── node3.pem
    ├── server.csr
    ├── server.json
    ├── server-key.pem
    └── server.pem

创建docker-compose.yml文件：

version: '2'

services:
  node1:
    image: 'bitnami/etcd:latest'
    environment:
      - "ETCD_NAME=node1"
      - "ETCD_ROOT_PASSWORD=hillstone"
      - "ETCD_CLIENT_CERT_AUTH=true"
      - "ETCD_PEER_CLIENT_CERT_AUTH=true"
      - "ETCD_ADVERTISE_CLIENT_URLS=https://10.182.51.82:42379"
      - "ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.182.51.82:42380"
      - "ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
      - "ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
      - "ETCD_INITIAL_CLUSTER_TOKEN=etcd_cluster"
      - "ETCD_INITIAL_CLUSTER=node1=https://10.182.51.82:42380,node2=https://10.182.51.82:52380,node3=https://10.182.51.82:62380"
      - "ETCD_INITIAL_CLUSTER_STATE=new"
      - "ETCD_DATA_DIR=/opt/bitnami/etcd/data"
      - "ETCD_TRUSTED_CA_FILE=/opt/bitnami/etcd/conf/ca.pem"
      - "ETCD_KEY_FILE=/opt/bitnami/etcd/conf/server-key.pem"
      - "ETCD_CERT_FILE=/opt/bitnami/etcd/conf/server.pem"
      - "ETCD_PEER_TRUSTED_CA_FILE=/opt/bitnami/etcd/conf/ca.pem"
      - "ETCD_PEER_KEY_FILE=/opt/bitnami/etcd/conf/peer-key.pem"
      - "ETCD_PEER_CERT_FILE=/opt/bitnami/etcd/conf/peer.pem"
    volumes:
      - ./node1/ca.pem:/opt/bitnami/etcd/conf/ca.pem
      - ./node1/node1.pem:/opt/bitnami/etcd/conf/peer.pem
      - ./node1/node1-key.pem:/opt/bitnami/etcd/conf/peer-key.pem
      - ./node1/server.pem:/opt/bitnami/etcd/conf/server.pem
      - ./node1/server-key.pem:/opt/bitnami/etcd/conf/server-key.pem
      - ./node1/client-key.pem:/opt/bitnami/etcd/client-key.pem
      - ./node1/client.pem:/opt/bitnami/etcd/client.pem
    ports:
      - 42379:2379
      - 42380:2380

  node2:
    image: 'bitnami/etcd:latest'
    environment:
      - "ETCD_NAME=node2"
      - "ETCD_ROOT_PASSWORD=hillstone"
      - "ETCD_CLIENT_CERT_AUTH=true"
      - "ETCD_PEER_CLIENT_CERT_AUTH=true"
      - "ETCD_ADVERTISE_CLIENT_URLS=https://10.182.51.82:52379"
      - "ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.182.51.82:52380"
      - "ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
      - "ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
      - "ETCD_INITIAL_CLUSTER_TOKEN=etcd_cluster"
      - "ETCD_INITIAL_CLUSTER=node1=https://10.182.51.82:42380,node2=https://10.182.51.82:52380,node3=https://10.182.51.82:62380"
      - "ETCD_INITIAL_CLUSTER_STATE=new"
      - "ETCD_DATA_DIR=/opt/bitnami/etcd/data"
      - "ETCD_TRUSTED_CA_FILE=/opt/bitnami/etcd/conf/ca.pem"
      - "ETCD_KEY_FILE=/opt/bitnami/etcd/conf/server-key.pem"
      - "ETCD_CERT_FILE=/opt/bitnami/etcd/conf/server.pem"
      - "ETCD_PEER_TRUSTED_CA_FILE=/opt/bitnami/etcd/conf/ca.pem"
      - "ETCD_PEER_KEY_FILE=/opt/bitnami/etcd/conf/peer-key.pem"
      - "ETCD_PEER_CERT_FILE=/opt/bitnami/etcd/conf/peer.pem"
    volumes:
      - ./node2/ca.pem:/opt/bitnami/etcd/conf/ca.pem
      - ./node2/node2.pem:/opt/bitnami/etcd/conf/peer.pem
      - ./node2/node2-key.pem:/opt/bitnami/etcd/conf/peer-key.pem
      - ./node2/server.pem:/opt/bitnami/etcd/conf/server.pem
      - ./node2/server-key.pem:/opt/bitnami/etcd/conf/server-key.pem
      - ./node2/client-key.pem:/opt/bitnami/etcd/client-key.pem
      - ./node2/client.pem:/opt/bitnami/etcd/client.pem
    ports:
      - 52379:2379
      - 52380:2380

  node3:
    image: 'bitnami/etcd:latest'
    environment:
      - "ETCD_NAME=node3"
      - "ETCD_ROOT_PASSWORD=hillstone"
      - "ETCD_CLIENT_CERT_AUTH=true"
      - "ETCD_PEER_CLIENT_CERT_AUTH=true"
      - "ETCD_ADVERTISE_CLIENT_URLS=https://10.182.51.82:62379"
      - "ETCD_INITIAL_ADVERTISE_PEER_URLS=https://10.182.51.82:62380"
      - "ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379"
      - "ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380"
      - "ETCD_INITIAL_CLUSTER_TOKEN=etcd_cluster"
      - "ETCD_INITIAL_CLUSTER=node1=https://10.182.51.82:42380,node2=https://10.182.51.82:52380,node3=https://10.182.51.82:62380"
      - "ETCD_INITIAL_CLUSTER_STATE=new"
      - "ETCD_DATA_DIR=/opt/bitnami/etcd/data"
      - "ETCD_TRUSTED_CA_FILE=/opt/bitnami/etcd/conf/ca.pem"
      - "ETCD_KEY_FILE=/opt/bitnami/etcd/conf/server-key.pem"
      - "ETCD_CERT_FILE=/opt/bitnami/etcd/conf/server.pem"
      - "ETCD_PEER_TRUSTED_CA_FILE=/opt/bitnami/etcd/conf/ca.pem"
      - "ETCD_PEER_KEY_FILE=/opt/bitnami/etcd/conf/peer-key.pem"
      - "ETCD_PEER_CERT_FILE=/opt/bitnami/etcd/conf/peer.pem"
    volumes:
      - ./node3/ca.pem:/opt/bitnami/etcd/conf/ca.pem
      - ./node3/node3.pem:/opt/bitnami/etcd/conf/peer.pem
      - ./node3/node3-key.pem:/opt/bitnami/etcd/conf/peer-key.pem
      - ./node3/server.pem:/opt/bitnami/etcd/conf/server.pem
      - ./node3/server-key.pem:/opt/bitnami/etcd/conf/server-key.pem
      - ./node3/client-key.pem:/opt/bitnami/etcd/client-key.pem
      - ./node3/client.pem:/opt/bitnami/etcd/client.pem
    ports:
      - 62379:2379
      - 62380:2380

上述文件中涉及到的文件目录等自行构建，IP10.182.51.82也需要根据实际情况修改为容器宿主机的IP。

2.3 创建并启动容器

docker-compose up -d

容器启动后，使用member list判断etcd集群是否正常：

docker exec -it --user root <容器名> etcdctl --user root --password hillstone --cacert /opt/bitnami/etcd/conf/ca.pem --key /opt/bitnami/etcd/client-key.pem --cert /opt/bitnami/etcd/client.pem member list

容器名根据实际情况填写。
正常会返回如下内容：

9927dd915256a772, started, node2, https://10.182.51.82:52380, https://10.182.51.82:52379, false
e4d7547554d0cafc, started, node1, https://10.182.51.82:42380, https://10.182.51.82:42379, false
eceffcc18f614185, started, node3, https://10.182.51.82:62380, https://10.182.51.82:62379, false

https etcd docker 容器 TLS etcd cluster

< etcd入门教程

CentOS7编译安装lxd >