ceph集群搭建

Published on with views and comments

Ceph 集群文档

集群架构:
image.png

环境:
10.200.51.4 admin、osd、mon 作为管理和监控节点
10.200.51.9 osd、mds
10.200.51.10 osd、mds
10.200.51.113~client 节点

ceph1 作为管理,osd.mon 节点。前三台新增硬盘

[root@ceph1 ~]# mkfs.xfs /dev/sdb meta-data=/dev/sdb isize=512 agcount=4, agsize=1310720 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=5242880, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@ceph1 ~]# mkdir /var/local/osd{0,1,2} [root@ceph1 ~]# mount /dev/sdb /var/local/osd0/
[root@ceph2 ~]# mkfs.xfs /dev/sdb meta-data=/dev/sdb isize=512 agcount=4, agsize=1310720 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=5242880, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@ceph2 ~]# mkdir /var/local/osd{0,1,2} [root@ceph2 ~]# mount /dev/sdb /var/local/osd1/
[root@ceph3 ~]# mkfs.xfs /dev/sdb meta-data=/dev/sdb isize=512 agcount=4, agsize=1310720 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=5242880, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=2560, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@ceph3 ~]# mkdir /var/local/osd{0,1,2} [root@ceph3 ~]# mount /dev/sdb /var/local/osd2/
  • 编辑 host 文件(各节点操作)
10.200.51.4 ceph1 10.200.51.9 ceph2 10.200.51.10 ceph3 10.200.51.113 ceph4
echo -e "192.168.51.206 ceph1\n192.168.51.207 ceph2\n192.168.51.208 ceph3\n192.168.51.209 ceph4\n192.168.51.212 controller\n192.168.51.211 cinder\n192.168.51.210 computer" >> /etc/hosts
  • SSH 免密码登陆(各节点操作)
ssh-keygen ssh-copy-id ceph1 ssh-copy-id ceph2 ssh-copy-id ceph3 ssh-copy-id ceph4
  • 同步时间
    在 ceph1 上安装 ntp 服务,是 2、3 同步 1 的时间
[root@ceph1 ~]# yum install ntp -y [root@ceph1 ~]# systemctl start ntpd [root@ceph2 ~]# ntpdate 10.200.51.4 [root@ceph3 ~]# ntpdate 10.200.51.4

或者所有节点配置统一互联网 ntp 服务

ntpdate ntp1.aliyun.com

管理节点安装

添加 yum 源,参考阿里镜像源
官方源

[ceph] name=ceph baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/x86_64/ gpgcheck=0 priority=1 [ceph-noarch] name=ceph noarch baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/noarch/ gpgcheck=0 priority=1 [ceph-source] name=Ceph source packages baseurl=http://mirrors.aliyun.com/ceph/rpm-jewel/el7/SRPMS gpgcheck=0 priority=1
yum clean all yum makecache
  • 更新软件源安装 ceph-deploy
yum -y install ceph-deploy
  • 创建 monitor 服务
mkdir /etc/ceph && cd /etc/ceph ceph-deploy new ceph1 [root@ceph1 ceph]# ls ceph.conf ceph-deploy-ceph.log ceph.mon.keyring

Ceph 配置文件、一个 monitor 密钥环和一个日志文件
修改副本数
#配置文件的默认副本数从 3 改成 2,这样只有两个 osd 也能达到 active+clean 状态,把下面这行加入到[global]段(可选配置),参考这里

[root@ceph1 ceph]# vim ceph.conf [global] fsid = 94b37984-4bf3-44f4-ac08-3bb1a638c771 mon_initial_members = ceph1 mon_host = 10.200.51.4 auth_cluster_required = cephx auth_service_required = cephx auth_client_required = cephx osd_pool_default_size = 2
  • 安装 ceph(所有节点安装)
[root@ceph1 ceph]# ceph-deploy install ceph1 ceph2 ceph3 ceph4

完成后:

[root@ceph1 ceph]# ceph --version ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)
[root@ceph1 ceph]# ceph-deploy mon create ceph1
[root@ceph1 ceph]# ceph --cluster=ceph --admin-daemon /var/run/ceph/ceph-mon.ceph1.asok mon_status { "name": "ceph1", "rank": 0, "state": "leader", "election_epoch": 3, "quorum": [ 0 ], "outside_quorum": [], "extra_probe_peers": [], "sync_provider": [], "monmap": { "epoch": 1, "fsid": "0fe8ad12-6f71-4a94-80cc-2d19a9217b4b", "modified": "2019-10-31 01:50:50.330750", "created": "2019-10-31 01:50:50.330750", "mons": [ { "rank": 0, "name": "ceph1", "addr": "10.200.51.4:6789\/0" } ] } }
[root@ceph1 ceph]# ceph-deploy gatherkeys ceph1

部署 osd 服务

添加 osd 节点
之前创建的目录/var/local/osd{id}

  • 准备 OSD
[root@ceph1 ceph]# ceph-deploy osd prepare ceph1:/var/local/osd0 ceph2:/var/local/osd1 ceph3:/var/local/osd2
  • 激活 OSD:
    首先给足权限(在各个节点上给/var/local/osd0/、/var/local/osd1/和/var/local/osd2/添加权限):
chmod 777 -R /var/local/osd0/ chmod 777 -R /var/local/osd1/ chmod 777 -R /var/local/osd2/
[root@ceph1 ceph]# ceph-deploy osd activate ceph1:/var/local/osd0 ceph2:/var/local/osd1 ceph3:/var/local/osd2 ceph4:/var/local/osd3
  • 查看状态:
[root@ceph1 ceph]# ceph-deploy osd list ceph1 ceph2 ceph3 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy osd list ceph1 ceph2 ceph3 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] subcommand : list [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f0d07370290> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] func : <function osd at 0x7f0d073be050> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.cli][INFO ] disk : [('ceph1', None, None), ('ceph2', None, None), ('ceph3', None, None)] [ceph1][DEBUG ] connected to host: ceph1 [ceph1][DEBUG ] detect platform information from remote host [ceph1][DEBUG ] detect machine type [ceph1][DEBUG ] find the location of an executable [ceph1][DEBUG ] find the location of an executable [ceph1][INFO ] Running command: /bin/ceph --cluster=ceph osd tree --format=json [ceph1][DEBUG ] connected to host: ceph1 [ceph1][DEBUG ] detect platform information from remote host [ceph1][DEBUG ] detect machine type [ceph1][DEBUG ] find the location of an executable [ceph1][INFO ] Running command: /usr/sbin/ceph-disk list [ceph1][INFO ] ---------------------------------------- [ceph1][INFO ] ceph-0 [ceph1][INFO ] ---------------------------------------- [ceph1][INFO ] Path /var/lib/ceph/osd/ceph-0 [ceph1][INFO ] ID 0 [ceph1][INFO ] Name osd.0 [ceph1][INFO ] Status up [ceph1][INFO ] Reweight 1.0 [ceph1][INFO ] Active ok [ceph1][INFO ] Magic ceph osd volume v026 [ceph1][INFO ] Whoami 0 [ceph1][INFO ] Journal path /var/local/osd0/journal [ceph1][INFO ] ---------------------------------------- [ceph2][DEBUG ] connected to host: ceph2 [ceph2][DEBUG ] detect platform information from remote host [ceph2][DEBUG ] detect machine type [ceph2][DEBUG ] find the location of an executable [ceph2][INFO ] Running command: /usr/sbin/ceph-disk list [ceph2][INFO ] ---------------------------------------- [ceph2][INFO ] ceph-1 [ceph2][INFO ] ---------------------------------------- [ceph2][INFO ] Path /var/lib/ceph/osd/ceph-1 [ceph2][INFO ] ID 1 [ceph2][INFO ] Name osd.1 [ceph2][INFO ] Status up [ceph2][INFO ] Reweight 1.0 [ceph2][INFO ] Active ok [ceph2][INFO ] Magic ceph osd volume v026 [ceph2][INFO ] Whoami 1 [ceph2][INFO ] Journal path /var/local/osd1/journal [ceph2][INFO ] ---------------------------------------- [ceph3][DEBUG ] connected to host: ceph3 [ceph3][DEBUG ] detect platform information from remote host [ceph3][DEBUG ] detect machine type [ceph3][DEBUG ] find the location of an executable [ceph3][INFO ] Running command: /usr/sbin/ceph-disk list [ceph3][INFO ] ---------------------------------------- [ceph3][INFO ] ceph-2 [ceph3][INFO ] ---------------------------------------- [ceph3][INFO ] Path /var/lib/ceph/osd/ceph-2 [ceph3][INFO ] ID 2 [ceph3][INFO ] Name osd.2 [ceph3][INFO ] Status up [ceph3][INFO ] Reweight 1.0 [ceph3][INFO ] Active ok [ceph3][INFO ] Magic ceph osd volume v026 [ceph3][INFO ] Whoami 2 [ceph3][INFO ] Journal path /var/local/osd2/journal [ceph3][INFO ] ----------------------------------------
  • 统一配置
    (用 ceph-deploy 把配置文件和 admin 密钥拷贝到所有节点,这样每次执行 Ceph 命令行时就无需 指定 monitor 地址和 ceph.client.admin.keyring 了)
[root@ceph1 ceph]# ceph-deploy admin ceph1 ceph2 ceph3 [ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf [ceph_deploy.cli][INFO ] Invoked (1.5.39): /usr/bin/ceph-deploy admin ceph1 ceph2 ceph3 [ceph_deploy.cli][INFO ] ceph-deploy options: [ceph_deploy.cli][INFO ] username : None [ceph_deploy.cli][INFO ] verbose : False [ceph_deploy.cli][INFO ] overwrite_conf : False [ceph_deploy.cli][INFO ] quiet : False [ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7feff2220cb0> [ceph_deploy.cli][INFO ] cluster : ceph [ceph_deploy.cli][INFO ] client : ['ceph1', 'ceph2', 'ceph3'] [ceph_deploy.cli][INFO ] func : <function admin at 0x7feff2f39a28> [ceph_deploy.cli][INFO ] ceph_conf : None [ceph_deploy.cli][INFO ] default_release : False [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph1 [ceph1][DEBUG ] connected to host: ceph1 [ceph1][DEBUG ] detect platform information from remote host [ceph1][DEBUG ] detect machine type [ceph1][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph2 [ceph2][DEBUG ] connected to host: ceph2 [ceph2][DEBUG ] detect platform information from remote host [ceph2][DEBUG ] detect machine type [ceph2][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf [ceph_deploy.admin][DEBUG ] Pushing admin keys and conf to ceph3 [ceph3][DEBUG ] connected to host: ceph3 [ceph3][DEBUG ] detect platform information from remote host [ceph3][DEBUG ] detect machine type [ceph3][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf

各节点修改 ceph.client.admin.keyring 权限:

chmod +r /etc/ceph/ceph.client.admin.keyring
  • 查看 OSD 状态:
[root@ceph1 ceph]# ceph health HEALTH_OK

部署 MDS 服务

[root@ceph1 ceph]# ceph-deploy mds create ceph2 ceph3

查看集群状态:

[root@ceph1 ceph]# ceph mds stat e4:, 3 up:standby [root@ceph1 ceph]# ceph -s cluster 0fe8ad12-6f71-4a94-80cc-2d19a9217b4b health HEALTH_OK monmap e1: 1 mons at {ceph1=10.200.51.4:6789/0} election epoch 3, quorum 0 ceph1 osdmap e15: 3 osds: 3 up, 3 in flags sortbitwise,require_jewel_osds pgmap v25: 64 pgs, 1 pools, 0 bytes data, 0 objects 15681 MB used, 45728 MB / 61410 MB avail 64 active+clean

创建存储系统

首先查看:

[root@ceph1 ceph]# ceph fs ls No filesystems enabled
  • 创建存储池
[root@ceph1 ceph]# ceph osd pool create cephfs_data 128 pool 'cephfs_data' created [root@ceph1 ceph]# ceph osd pool create cephfs_metadata 128 pool 'cephfs_metadata' created
  • 创建文件系统
    创建好存储池后,就可以用 fs new 命令创建文件系统了
[root@ceph1 ceph]# ceph fs new 128 cephfs_metadata cephfs_data new fs with metadata pool 2 and data pool 1 [root@ceph1 ceph]# ceph fs ls name: 128, metadata pool: cephfs_metadata, data pools: [cephfs_data ]

文件系统创建完毕后, MDS 服务器就能达到 active 状态了,比如在一个单 MDS 系统中:

[root@ceph1 ceph]# ceph mds stat e7: 1/1/1 up {0=ceph3=up:active}, 2 up:standby

active 是活跃的,另 1 个是处于热备份的状态

挂载 ceph 文件系统

不同的挂载方式

用内核驱动挂载 CEPH 文件系统

client 端配置:
#创建挂载点 存储密钥(如果没有在管理节点使用 ceph-deploy 拷贝 ceph 配置文件)

[root@ceph3 ~]# cat /etc/ceph/ceph.client.admin.keyring [client.admin] key = AQA6drpdei+IORAA7Ogqh9Wn0NbdGI/juTXnqw==

将如上 key 对应的值保存到客户端(/etc/ceph/admin.secret)

[root@ceph4 ceph]# cat /etc/ceph/admin.secret AQA6drpdei+IORAA7Ogqh9Wn0NbdGI/juTXnqw==

要挂载启用了 cephx 认证的 Ceph 文件系统,你必须指定用户名、密钥。

[root@ceph4 ~]# mount -t ceph 10.200.51.4:6789:/ /opt/ -o name=admin,secretfile=/etc/ceph/admin.secret

查看挂载情况:

[root@ceph4 ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 475M 0 475M 0% /dev tmpfs 487M 0 487M 0% /dev/shm tmpfs 487M 7.6M 479M 2% /run tmpfs 487M 0 487M 0% /sys/fs/cgroup /dev/mapper/centos-root 27G 1.6G 26G 6% / /dev/sda1 1014M 136M 879M 14% /boot tmpfs 98M 0 98M 0% /run/user/0 10.200.51.4:6789:/ 60G 16G 45G 26% /opt

取消挂载

[root@ceph4 ~]# umount /opt/ [root@ceph4 ~]# df -h Filesystem Size Used Avail Use% Mounted on devtmpfs 475M 0 475M 0% /dev tmpfs 487M 0 487M 0% /dev/shm tmpfs 487M 7.6M 479M 2% /run tmpfs 487M 0 487M 0% /sys/fs/cgroup /dev/mapper/centos-root 27G 1.6G 26G 6% / /dev/sda1 1014M 136M 879M 14% /boot tmpfs 98M 0 98M 0% /run/user/0

用户空间挂载 CEPH 文件系统


标题:ceph集群搭建
作者:cuijianzhe
地址:https://cjzshilong.cn/articles/2019/10/30/1572426357896.html