最近大半年一直在参与公司针对OpenStack的一些定制开发，主要是与Neutron组件相关。大大小小的各个项目中，与华为、中兴、EasyStack等都有过合作，每个项目基于的OpenStack版本都不一样，所以经常来回的安装部署OpenStack环境，比较繁琐，工作之余一直在寻求一种部署环境的便捷方式。目前比较主流的部署方式是devstack，也非常方便，但观察OpenStack的发展趋势，已经有越来越多的公司在做OpenStack的容器化部署，OpenStack社区也有对应的项目：kolla，虽然现在用的不是特别多，但利用容器产生的一些优势，相信在将来会有更多的公司采用容器化部署，本人在空闲时间尝试用kolla部署了一套mitaka版本的OpenStack环境（ALL IN ONE），以下是详细步骤，如果有任何问题也欢迎底下留言。
操作系统： Ubuntu 14.04
Docker： Docker version 1.12.5, build 7392c3b注意： Docker版本不能使用最新版本，推荐1.12.x。使用最新版本部署的时候会有问题，主要是python的docker api库版本不匹配，网络上其他教程基本都有这个问题。
apt update
apt upgrade
apt-get install linux-image-generic-lts-wily
rebootsudo add-apt-repository ppa:fkrull/deadsnakes-python2.7
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 5BB92C09DB82666C
apt update
apt upgrade
apt-get install python-dev libffi-dev libssl-dev gcc这里推荐手动安装pip工具，而不要使用apt来安装，原因是用apt安装会同时安装几个python库，这几个python库是无法被pip管理的，后面安装过程中可能会有冲突。
wget https://files.pythonhosted.org/packages/c8/89/ad7f27938e59db1f0f55ce214087460f65048626e2226531ba6cb6da15f0/pip-19.0.1.tar.gz
tar xvf pip-19.0.1.tar.gz
cd pip-19.0.1/
python setup.py install
pip install -U pip安装完了pip以后，使用dpgk命令查看下系统当前安装了哪些python库，用apt remove将它们都删除，用pip再安装一遍。
dpkg -l | grep python-使用最新版本的docker部署mitaka会报错，本文使用1.12.5版本的docker。
wget https://apt.dockerproject.org/repo/pool/main/d/docker-engine/docker-engine_1.12.5-0~ubuntu-trusty_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/main/s/systemd/libsystemd-journal0_204-5ubuntu20_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/main/libt/libtool/libltdl7_2.4.2-1.7ubuntu1_amd64.deb
wget http://archive.ubuntu.com/ubuntu/pool/universe/libg/libgcrypt20/libgcrypt20_1.6.1-2ubuntu1.14.04.1_amd64.deb
dpkg -i libltdl7_2.4.2-1.7ubuntu1_amd64.deb
dpkg -i libgcrypt20_1.6.1-2ubuntu1.14.04.1_amd64.deb
dpkg -i libsystemd-journal0_204-5ubuntu20_amd64.deb
dpkg -i docker-engine_1.12.5-0~ubuntu-trusty_amd64.deb
reboot安装完了以后，需要对docker配置shared mount flag，否则部分容器部署会失败。
mount --make-shared /run
service docker restart推荐将上述两条命令加入到rc.local脚本中，否则每次重启都需要执行。
网上也有教程说安装docker库，本人没有试验过。
pip install -U docker-pymitaka版本需要使用1.9.4，不能使用最新版本。
pip install -U ansible==1.9.4OpenStack官方的仓库中已经废弃了mitaka版本的分支，所以现在无法从官方下载，我从github中查找到了一个包含mitaka版本的仓库，但是有些小问题，复制到自己的github仓库：https://github.com/zpzhoudev/kolla-deprecate.git 进行了修复。
apt install git
git clone -b stable/mitaka https://github.com/zpzhoudev/kolla-deprecate.git
mv kolla-deprecate kolla
pip install -r kolla/requirements.txt -r kolla/test-requirements.txt
pip install kolla/利用tox生成配置文件
pip install -U tox
cd kolla/
tox -e genconfig
cp -rv etc/kolla /etc/pip install -U python-openstackclient python-neutronclient因为本次部署的all in one，所以这步跳过了，多节点的话是必须的。
docker pull registry:latest
docker run -d -p 4000:5000 -e REGISTRY_STORAGE_DELETE_ENABLED="true" --restart=always --name registry registry:latestvi /etc/default/docker
DOCKER_OPTS="--insecure-registry 10.180.52.80:4000"
service docker restart本次只搭建基础环境，所以需要的镜像也不多，没必要全部编译。
kolla-build -b ubuntu -t binary horizon cinder heat nova neutron glance keystone rabbitmq keepalived haproxy heka kolla-toolbox mariadb memcached cron openvswitch
# 如果使用docker本地仓库，需要添加参数： --registry {registry-ip}:4000 --push以上命令是基于ubuntu系统来编译，现在貌似只有ubuntu才有mitaka版本的软件包源。
编译完成后镜像如下所示：
root@ubuntu:~# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
kollaglue/ubuntu-binary-kolla-toolbox 2.0.1 04ad51ce7150 26 hours ago 824.1 MB
kollaglue/ubuntu-binary-nova-compute 2.0.1 260bd29ab0ac 2 days ago 800.8 MB
kollaglue/ubuntu-binary-nova-libvirt 2.0.1 9cb64d949a52 2 days ago 822.9 MB
kollaglue/ubuntu-binary-nova-compute-ironic 2.0.1 47a68f4d4c88 2 days ago 724.2 MB
kollaglue/ubuntu-binary-nova-novncproxy 2.0.1 38f6cd8037b1 2 days ago 471.1 MB
kollaglue/ubuntu-binary-nova-spicehtml5proxy 2.0.1 c77f67bcae77 2 days ago 448.9 MB
kollaglue/ubuntu-binary-nova-api 2.0.1 645b11126977 2 days ago 450.3 MB
kollaglue/ubuntu-binary-nova-conductor 2.0.1 f005defaba9a 2 days ago 448.6 MB
kollaglue/ubuntu-binary-nova-consoleauth 2.0.1 5ca76067f86a 2 days ago 448.6 MB
kollaglue/ubuntu-binary-nova-scheduler 2.0.1 49577fddeac3 2 days ago 448.6 MB
kollaglue/ubuntu-binary-horizon 2.0.1 638e97eb8575 2 days ago 490.3 MB
kollaglue/ubuntu-binary-nova-ssh 2.0.1 4f89ffb93a8b 2 days ago 452.5 MB
kollaglue/ubuntu-binary-nova-network 2.0.1 8cf2779ac36c 2 days ago 451.4 MB
kollaglue/ubuntu-binary-cinder-volume 2.0.1 4a3486e3e3b0 2 days ago 513.7 MB
kollaglue/ubuntu-binary-cinder-rpcbind 2.0.1 ec6088366536 2 days ago 506.8 MB
kollaglue/ubuntu-binary-cinder-backup 2.0.1 392b20fbe1e4 2 days ago 506.3 MB
kollaglue/ubuntu-binary-cinder-scheduler 2.0.1 d9faab863d82 2 days ago 506.3 MB
kollaglue/ubuntu-binary-cinder-api 2.0.1 6c50baa045b6 2 days ago 506.3 MB
kollaglue/ubuntu-binary-nova-base 2.0.1 fda510c8876b 2 days ago 446.6 MB
kollaglue/ubuntu-binary-cinder-base 2.0.1 dce5291bdbca 2 days ago 504.2 MB
kollaglue/ubuntu-binary-neutron-metadata-agent 2.0.1 1988f1ca228e 2 days ago 420.3 MB
kollaglue/ubuntu-binary-neutron-l3-agent 2.0.1 d5cd6f437820 2 days ago 425.9 MB
kollaglue/ubuntu-binary-neutron-openvswitch-agent 2.0.1 2babf6b4b944 2 days ago 420.5 MB
kollaglue/ubuntu-binary-neutron-linuxbridge-agent 2.0.1 5043fcb29f0b 2 days ago 420.9 MB
kollaglue/ubuntu-binary-neutron-dhcp-agent 2.0.1 dea8dee1cf79 2 days ago 421 MB
kollaglue/ubuntu-binary-keystone 2.0.1 2672e9a5612d 2 days ago 412.4 MB
kollaglue/ubuntu-binary-heat-api-cfn 2.0.1 ce00524c2649 2 days ago 403.6 MB
kollaglue/ubuntu-binary-neutron-server 2.0.1 aed7e7adc997 2 days ago 418.2 MB
kollaglue/ubuntu-binary-neutron-base 2.0.1 4d0957a67168 2 days ago 418.2 MB
kollaglue/ubuntu-binary-glance-api 2.0.1 9014d2e097fa 2 days ago 428.7 MB
kollaglue/ubuntu-binary-glance-registry 2.0.1 5a0fdf9afefb 2 days ago 428.7 MB
kollaglue/ubuntu-binary-heat-engine 2.0.1 0ecbcfa285fb 2 days ago 403.6 MB
kollaglue/ubuntu-binary-heat-api 2.0.1 ff613b152586 2 days ago 403.6 MB
kollaglue/ubuntu-binary-glance-base 2.0.1 d605ff7f7a54 2 days ago 428.7 MB
kollaglue/ubuntu-binary-heat-base 2.0.1 7a404658b38f 2 days ago 401.7 MB
kollaglue/ubuntu-binary-openstack-base 2.0.1 e73bbe926a69 2 days ago 348.7 MB
kollaglue/ubuntu-binary-mariadb 2.0.1 bfc07c71e3a8 2 days ago 445.6 MB
kollaglue/ubuntu-binary-rabbitmq 2.0.1 be7794769c8f 2 days ago 276.7 MB
kollaglue/ubuntu-binary-heka 2.0.1 91757ed86dbc 2 days ago 283.1 MB
kollaglue/ubuntu-binary-openvswitch-vswitchd 2.0.1 99ccfb2278a8 2 days ago 246.6 MB
kollaglue/ubuntu-binary-openvswitch-db-server 2.0.1 0b50b1151baf 2 days ago 246.6 MB
kollaglue/ubuntu-binary-openvswitch-base 2.0.1 8e05dc556547 2 days ago 246.6 MB
kollaglue/ubuntu-binary-haproxy 2.0.1 a2ab0c4c6e72 2 days ago 240.6 MB
kollaglue/ubuntu-binary-keepalived 2.0.1 1c9c4cc82e86 2 days ago 246.3 MB
kollaglue/ubuntu-binary-memcached 2.0.1 a4f0cb710477 2 days ago 239.2 MB
kollaglue/ubuntu-binary-cron 2.0.1 530021fed789 2 days ago 237.6 MB
kollaglue/ubuntu-binary-base 2.0.1 23a9cb3b1b0e 2 days ago 237.6 MB
ubuntu 14.04 5dbc3f318ea5 12 days ago 188.1 MB使用kolla-genpwd命令生成密码文件。
kolla-genpwd生成的文件保存在/etc/kolla/passwords.yml，为方便后面的使用，可以将password结尾的配置项的值都改为固定的易识别的字符串。
编辑文件/etc/kolla/globals.yml，根据实际情况修改，以下是我的配置：
kolla_base_distro: "ubuntu"
kolla_install_type: "binary"
openstack_release: "2.0.1"
# 当不是高可用的情况下，使用network_interface接口上的ip，prechecks可能不通过，可忽略
kolla_internal_address: "10.180.52.80"
kolla_internal_fqdn: "controller"
kolla_external_fqdn: "controller"
network_interface: "eth0"
neutron_external_interface: "eth1"
neutron_plugin_agent: "openvswitch"
enable_ceilometer: "no"
enable_haproxy: "no"
enable_cinder: "yes"
enable_horizon: "yes"hostnamectl set-hostname ubuntu将本机的host名对应的ip修改成外部可访问的ip，不能用127.0.0.1，再添加kolla_external_fqdn和kolla_internal_fqdn对应的hosts记录
10.180.52.80 ubuntu
10.180.52.80 controllerkolla-ansible prechecks如果检查没有问题，就可以开始部署了。
kolla-ansible deploy
# 部署指定容器
kolla-ansible deploy -t mariadb如果部署失败，需要重新部署，kolla提供了几个脚本方便我们清理环境：
#清理容器
tools/cleanup-containers
#清理配置
tools/cleanup-host
#清理docker镜像
tools/cleanup-images1.使用kolla自动生成环境变量文件，保存在/etc/kolla/目录下
kolla-ansible post-deploy2.修改horizon的监听host地址：
修改/etc/kolla/horizon/horizon.conf文件中的Listen配置项。
build marathon error: KeyError: 'logs'
INFO:kolla.cmd.build:marathon:Removing intermediate container a0952c7ad4f7
ERROR:kolla.cmd.build:marathon:Error'd with the following message
ERROR:kolla.cmd.build:marathon:The command '/bin/sh -c useradd --user-group marathon && chmod 755 /usr/bin/marathon' returned a non-zero code: 9
Traceback (most recent call last):
File "/usr/local/bin/kolla-build", line 10, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 739, in main
kolla.summary()
File "/usr/local/lib/python2.7/dist-packages/kolla/cmd/build.py", line 559, in summary
for line in image['logs'].split('\n'):
KeyError: 'logs'找到对应代码，将log输出注释即可。
TASK: [ceph | Fetching Ceph keyrings] *****************************************
fatal: [localhost -> ubuntu] => SSH Error: ssh: Could not resolve hostname ubuntu: Name or service not known
It is sometimes useful to re-run the command using -vvvv, which prints SSH debug output to help diagnose the issue.
FATAL: all hosts have already failed -- aborting增加host名到/etc/hosts
failed: [localhost] => {"attempts": 10, "changed": false, "cmd": ["docker", "exec", "-t", "kolla_toolbox", "/usr/bin/ansible", "localhost", "-m", "mysql_user", "-a", "login_host='10.8.132.72' login_port='3306' login_user='root' login_password='hillstone' name='haproxy' password='' host='%' priv=*.*:USAGE"], "delta": "0:00:00.622472", "end": "2019-02-01 15:18:25.299043", "failed": true, "rc": 0, "start": "2019-02-01 15:18:24.676571", "stdout_lines": ["/usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.", " utils.DeprecatedIn23,", "localhost | SUCCESS => {", " \"changed\": false, ", " \"user\": \"haproxy\"", "}"], "warnings": []}
stdout: /usr/lib/python2.7/dist-packages/cryptography/hazmat/primitives/constant_time.py:26: CryptographyDeprecationWarning: Support for your Python version is deprecated. The next version of cryptography will remove support. Please upgrade to a 2.7.x release that supports hmac.compare_digest as soon as possible.
utils.DeprecatedIn23,
localhost | SUCCESS => {
"changed": false,
"user": "haproxy"
}
msg: Task failed as maximum retries was encountered镜像中的python版本太低，需要修改base的dockerfile，升级python，在我的github仓库中已经修改。
TASK: [rabbitmq | fail msg="Hostname has to resolve to IP address of api_interface"] ***
failed: [localhost] => (item={'cmd': ['getent', 'ahostsv4', 'ubuntu'], 'end': '2019-02-03 13:23:55.317541', 'stderr': '', 'stdout': '10.180.169.193 STREAM ubuntu\n10.180.169.193 DGRAM \n10.180.169.193 RAW ', 'changed': False, 'rc': 0, 'item': 'localhost', 'warnings': [], 'delta': '0:00:00.004081', 'invocation': {'module_name': u'command', 'module_complex_args': {}, 'module_args': u'getent ahostsv4 ubuntu'}, 'stdout_lines': ['10.180.169.193 STREAM ubuntu', '10.180.169.193 DGRAM ', '10.180.169.193 RAW '], 'start': '2019-02-03 13:23:55.313460'}) => {"failed": true, "item": {"changed": false, "cmd": ["getent", "ahostsv4", "ubuntu"], "delta": "0:00:00.004081", "end": "2019-02-03 13:23:55.317541", "invocation": {"module_args": "getent ahostsv4 ubuntu", "module_complex_args": {}, "module_name": "command"}, "item": "localhost", "rc": 0, "start": "2019-02-03 13:23:55.313460", "stderr": "", "stdout": "10.180.169.193 STREAM ubuntu\n10.180.169.193 DGRAM \n10.180.169.193 RAW ", "stdout_lines": ["10.180.169.193 STREAM ubuntu", "10.180.169.193 DGRAM ", "10.180.169.193 RAW "], "warnings": []}}
msg: Hostname has to resolve to IP address of api_interface
FATAL: all hosts have already failed -- abortinghost名必须解析到apt_interface所配置的接口的IP
提示没有_member_角色
执行命令手动创建：openstack role create _member_
修改horizon容器中的cinderclient：/usr/lib/python2.7/site-packages/cinderclient/v2/services.py
class Service(base.Resource):
def __repr__(self):
# return "<Service: %s>" % self.service
return "<Service: %s>" % self._info
原因是neutron openvswitch agent启动卡住了，neutron执行sudo命令被要求输入密码，是rootwrap配置的问题，还未发现原因，比较粗暴的解决办法是：
进入容器修改sudo配置文件，让neutorn可以免密码执行任意命令
visudo
# 添加以下配置
neutron ALL=(ALL) NOPASSWD: ALL所有的neutron相关的容器都有这个问题，都需要修改。
原因有两个：
1、cinder用户没办法免密码执行相关命令，进入容器修改sudo配置文件
visudo
# 添加以下配置
cinder ALL=(ALL) NOPASSWD: ALL2、host上没有cinder-volume的volume group
dd if=/dev/zero of=./cinder-volumes bs=1 count=0 seek=200G
losetup /dev/loop0 cinder-volumes
pvcreate /dev/loop0
vgcreate cinder-volumes /dev/loop0
vgdisplay这个问题非常难搞，耗费了整整大半天才找到原因，主要是因为udev的同步问题导致容器中执行lvcreate或lvdelete命令时，一直卡在那里了，只要关掉lvm的udev同步即可。
vi /etc/lvm/lvm.conf
udev_sync = 0
udev_rules = 0