如何在 CentOS 7.0 上配置 Ceph 存储

Ceph 是一个将数据存储在单一分布式计算机集群上的开源软件平台。当你计划构建一个云时，你首先需要决定如何实现你的存储。开源的 Ceph 是红帽原生技术之一，它基于称为 RADOS 的对象存储系统，用一组网关 API 表示块、文件、和对象模式中的数据。由于它自身开源的特性，这种便携存储平台能在公有云和私有云上安装和使用。Ceph 集群的拓扑结构是按照备份和信息分布设计的，这种内在设计能提供数据完整性。它的设计目标就是容错、通过正确配置能运行于商业硬件和一些更高级的系统。

Ceph 能在任何 Linux 发行版上安装，但为了能正确运行，它需要最近的内核以及其它最新的库。在这篇指南中，我们会使用最小化安装的 CentOS-7.0。

系统资源

1. <span class="pun">**</span><span class="pln">CEPH</span><span class="pun">-</span><span class="pln">STORAGE</span><span class="pun">**</span>
2. <span class="pln">OS</span><span class="pun">:</span><span class="typ">CentOS</span><span class="typ">Linux</span><span class="lit">7</span><span class="pun">(</span><span class="typ">Core</span><span class="pun">)</span>
3. <span class="pln">RAM</span><span class="pun">:</span><span class="lit">1</span><span class="pln"> GB</span>
4. <span class="pln">CPU</span><span class="pun">:</span><span class="lit">1</span><span class="pln"> CPU</span>
5. <span class="pln">DISK</span><span class="pun">:</span><span class="lit">20</span>
6. <span class="typ">Network</span><span class="pun">:</span><span class="lit">45.79</span><span class="pun">.</span><span class="lit">136.163</span>
7. <span class="pln">FQDN</span><span class="pun">:</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">storage</span><span class="pun">.</span><span class="pln">linoxide</span><span class="pun">.</span><span class="pln">com</span>
8. <span class="pun">**</span><span class="pln">CEPH</span><span class="pun">-</span><span class="pln">NODE</span><span class="pun">**</span>
9. <span class="pln">OS</span><span class="pun">:</span><span class="typ">CentOS</span><span class="typ">Linux</span><span class="lit">7</span><span class="pun">(</span><span class="typ">Core</span><span class="pun">)</span>
10. <span class="pln">RAM</span><span class="pun">:</span><span class="lit">1</span><span class="pln"> GB</span>
11. <span class="pln">CPU</span><span class="pun">:</span><span class="lit">1</span><span class="pln"> CPU</span>
12. <span class="pln">DISK</span><span class="pun">:</span><span class="lit">20</span>
13. <span class="typ">Network</span><span class="pun">:</span><span class="lit">45.79</span><span class="pun">.</span><span class="lit">171.138</span>
14. <span class="pln">FQDN</span><span class="pun">:</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">node</span><span class="pun">.</span><span class="pln">linoxide</span><span class="pun">.</span><span class="pln">com</span>

安装前的配置

在安装 Ceph 存储之前，我们要在每个节点上完成一些步骤。第一件事情就是确保每个节点的网络已经配置好并且能相互访问。

配置 Hosts

要在每个节点上配置 hosts 条目，要像下面这样打开默认的 hosts 配置文件（LCTT 译注：或者做相应的 DNS 解析）。

1. <span class="com">#</span><span class="kwd">vi</span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">hosts</span>

1. <span class="lit">45.79</span><span class="pun">.</span><span class="lit">136.163</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">storage ceph</span><span class="pun">-</span><span class="pln">storage</span><span class="pun">.</span><span class="pln">linoxide</span><span class="pun">.</span><span class="pln">com</span>
2. <span class="lit">45.79</span><span class="pun">.</span><span class="lit">171.138</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">node ceph</span><span class="pun">-</span><span class="pln">node</span><span class="pun">.</span><span class="pln">linoxide</span><span class="pun">.</span><span class="pln">com</span>

安装 VMware 工具

工作环境是 VMWare 虚拟环境时，推荐你安装它的 open VM 工具。你可以使用下面的命令安装。

1. <span class="com">#</span><span class="kwd">yum</span><span class="pln"> install </span><span class="pun">-</span><span class="pln">y open</span><span class="pun">-</span><span class="pln">vm</span><span class="pun">-</span><span class="pln">tools</span>

配置防火墙

如果你正在使用启用了防火墙的限制性环境，确保在你的 Ceph 存储管理节点和客户端节点中开放了以下的端口。

你必须在你的 Admin Calamari 节点开放 80、2003、以及4505-4506 端口，并且允许通过 80 号端口访问到 Ceph 或 Calamari 管理节点，以便你网络中的客户端能访问 Calamari web 用户界面。

你可以使用下面的命令在 CentOS 7 中启动并启用防火墙。

1. <span class="com">#</span><span class="kwd">systemctl</span><span class="pln"> start firewalld</span>
2. <span class="com">#</span><span class="kwd">systemctl</span><span class="pln"> enable firewalld</span>

运行以下命令使 Admin Calamari 节点开放上面提到的端口。

1. <span class="com">#</span><span class="pln"> firewall</span><span class="pun">-</span><span class="pln">cmd </span><span class="pun">--</span><span class="pln">zone</span><span class="pun">=</span><span class="kwd">public</span><span class="pun">--</span><span class="pln">add</span><span class="pun">-</span><span class="pln">port</span><span class="pun">=</span><span class="lit">80</span><span class="pun">/</span><span class="pln">tcp </span><span class="pun">--</span><span class="pln">permanent</span>
2. <span class="com">#</span><span class="pln"> firewall</span><span class="pun">-</span><span class="pln">cmd </span><span class="pun">--</span><span class="pln">zone</span><span class="pun">=</span><span class="kwd">public</span><span class="pun">--</span><span class="pln">add</span><span class="pun">-</span><span class="pln">port</span><span class="pun">=</span><span class="lit">2003</span><span class="pun">/</span><span class="pln">tcp </span><span class="pun">--</span><span class="pln">permanent</span>
3. <span class="com">#</span><span class="pln"> firewall</span><span class="pun">-</span><span class="pln">cmd </span><span class="pun">--</span><span class="pln">zone</span><span class="pun">=</span><span class="kwd">public</span><span class="pun">--</span><span class="pln">add</span><span class="pun">-</span><span class="pln">port</span><span class="pun">=</span><span class="lit">4505</span><span class="pun">-</span><span class="lit">4506</span><span class="pun">/</span><span class="pln">tcp </span><span class="pun">--</span><span class="pln">permanent</span>
4. <span class="com">#</span><span class="pln"> firewall</span><span class="pun">-</span><span class="pln">cmd </span><span class="pun">--</span><span class="pln">reload</span>

在 Ceph Monitor 节点，你要在防火墙中允许通过以下端口。

1. <span class="com">#</span><span class="pln"> firewall</span><span class="pun">-</span><span class="pln">cmd </span><span class="pun">--</span><span class="pln">zone</span><span class="pun">=</span><span class="kwd">public</span><span class="pun">--</span><span class="pln">add</span><span class="pun">-</span><span class="pln">port</span><span class="pun">=</span><span class="lit">6789</span><span class="pun">/</span><span class="pln">tcp </span><span class="pun">--</span><span class="pln">permanent</span>

然后允许以下默认端口列表，以便能和客户端以及监控节点交互，并发送数据到其它 OSD。

1. <span class="com">#</span><span class="pln"> firewall</span><span class="pun">-</span><span class="pln">cmd </span><span class="pun">--</span><span class="pln">zone</span><span class="pun">=</span><span class="kwd">public</span><span class="pun">--</span><span class="pln">add</span><span class="pun">-</span><span class="pln">port</span><span class="pun">=</span><span class="lit">6800</span><span class="pun">-</span><span class="lit">7300</span><span class="pun">/</span><span class="pln">tcp </span><span class="pun">--</span><span class="pln">permanent</span>

如果你工作在非生产环境，建议你停用防火墙以及 SELinux 设置，在我们的测试环境中我们会停用防火墙以及 SELinux。

1. <span class="com">#</span><span class="kwd">systemctl</span><span class="pln"> stop firewalld</span>
2. <span class="com">#</span><span class="kwd">systemctl</span><span class="pln"> disable firewalld</span>

系统升级

现在升级你的系统并重启使所需更改生效。

1. <span class="com">#</span><span class="kwd">yum</span><span class="pln"> update</span>
2. <span class="com">#</span><span class="kwd">shutdown</span><span class="pun">-</span><span class="pln">r </span><span class="lit">0</span>

设置 Ceph 用户

现在我们会新建一个单独的 sudo 用户用于在每个节点安装 ceph-deploy工具，并允许该用户无密码访问每个节点，因为它需要在 Ceph 节点上安装软件和配置文件而不会有输入密码提示。

运行下面的命令在 ceph-storage 主机上新建有独立 home 目录的新用户。

1. <span class="pun">[</span><span class="pln">root@ceph</span><span class="pun">-</span><span class="pln">storage </span><span class="pun">~]#</span><span class="kwd">useradd</span><span class="pun">-</span><span class="pln">d </span><span class="pun">/</span><span class="pln">home</span><span class="pun">/</span><span class="pln">ceph </span><span class="pun">-</span><span class="pln">m ceph</span>
2. <span class="pun">[</span><span class="pln">root@ceph</span><span class="pun">-</span><span class="pln">storage </span><span class="pun">~]#</span><span class="kwd">passwd</span><span class="pln"> ceph</span>

节点中新建的每个用户都要有 sudo 权限，你可以使用下面展示的命令赋予 sudo 权限。

1. <span class="pun">[</span><span class="pln">root@ceph</span><span class="pun">-</span><span class="pln">storage </span><span class="pun">~]#</span><span class="kwd">echo</span><span class="str">"ceph ALL = (root) NOPASSWD:ALL"</span><span class="pun">|</span><span class="kwd">sudo</span><span class="kwd">tee</span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">sudoers</span><span class="pun">.</span><span class="pln">d</span><span class="pun">/</span><span class="pln">ceph</span>
2. <span class="pln">ceph ALL </span><span class="pun">=</span><span class="pun">(</span><span class="pln">root</span><span class="pun">)</span><span class="pln"> NOPASSWD</span><span class="pun">:</span><span class="pln">ALL</span>
3. <span class="pun">[</span><span class="pln">root@ceph</span><span class="pun">-</span><span class="pln">storage </span><span class="pun">~]#</span><span class="kwd">sudo</span><span class="kwd">chmod</span><span class="lit">0440</span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">sudoers</span><span class="pun">.</span><span class="pln">d</span><span class="pun">/</span><span class="pln">ceph</span>

设置 SSH 密钥

现在我们会在 Ceph 管理节点生成 ssh 密钥并把密钥复制到每个 Ceph 集群节点。

在 ceph-node 运行下面的命令复制它的 ssh 密钥到 ceph-storage。

1. <span class="pun">[</span><span class="pln">root@ceph</span><span class="pun">-</span><span class="pln">node </span><span class="pun">~]#</span><span class="kwd">ssh</span><span class="pun">-</span><span class="pln">keygen</span>
2. <span class="typ">Generating</span><span class="kwd">public</span><span class="pun">/</span><span class="kwd">private</span><span class="pln"> rsa key pair</span><span class="pun">.</span>
3. <span class="typ">Enter</span><span class="kwd">file</span><span class="kwd">in</span><span class="pln"> which to save the key </span><span class="pun">(</span><span class="str">/root/</span><span class="pun">.</span><span class="kwd">ssh</span><span class="pun">/</span><span class="pln">id_rsa</span><span class="pun">):</span>
4. <span class="typ">Created</span><span class="pln"> directory </span><span class="str">'/root/.ssh'</span><span class="pun">.</span>
5. <span class="typ">Enter</span><span class="pln"> passphrase </span><span class="pun">(</span><span class="kwd">empty</span><span class="kwd">for</span><span class="kwd">no</span><span class="pln"> passphrase</span><span class="pun">):</span>
6. <span class="typ">Enter</span><span class="pln"> same passphrase again</span><span class="pun">:</span>
7. <span class="typ">Your</span><span class="pln"> identification has been saved </span><span class="kwd">in</span><span class="pun">/</span><span class="pln">root</span><span class="pun">/.</span><span class="kwd">ssh</span><span class="pun">/</span><span class="pln">id_rsa</span><span class="pun">.</span>
8. <span class="typ">Your</span><span class="kwd">public</span><span class="pln"> key has been saved </span><span class="kwd">in</span><span class="pun">/</span><span class="pln">root</span><span class="pun">/.</span><span class="kwd">ssh</span><span class="pun">/</span><span class="pln">id_rsa</span><span class="pun">.</span><span class="pln">pub</span><span class="pun">.</span>
9. <span class="typ">The</span><span class="pln"> key fingerprint </span><span class="kwd">is</span><span class="pun">:</span>
10. <span class="lit">5b</span><span class="pun">:*:*:*:*:*:*:*:*:*:</span><span class="pln">c9 root@ceph</span><span class="pun">-</span><span class="pln">node</span>
11. <span class="typ">The</span><span class="pln"> key</span><span class="str">'s randomart image is:</span>
12. <span class="str">+--[ RSA 2048]----+</span>

1. <span class="pun">[</span><span class="pln">root@ceph</span><span class="pun">-</span><span class="pln">node </span><span class="pun">~]#</span><span class="kwd">ssh</span><span class="pun">-</span><span class="pln">copy</span><span class="pun">-</span><span class="kwd">id</span><span class="pln"> ceph@ceph</span><span class="pun">-</span><span class="pln">storage</span>

SSH key

配置 PID 数目

要配置 PID 数目的值，我们会使用下面的命令检查默认的内核值。默认情况下，是一个小的最大线程数 32768。

如下图所示通过编辑系统配置文件配置该值为一个更大的数。

更改 PID 值

配置管理节点服务器

配置并验证了所有网络后，我们现在使用 ceph 用户安装 ceph-deploy。通过打开文件检查 hosts 条目（LCTT 译注：你也可以用 DNS 解析来完成）。

1. <span class="com">#</span><span class="kwd">vim</span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="pln">hosts</span>
2. <span class="pln">ceph</span><span class="pun">-</span><span class="pln">storage </span><span class="lit">45.79</span><span class="pun">.</span><span class="lit">136.163</span>
3. <span class="pln">ceph</span><span class="pun">-</span><span class="pln">node </span><span class="lit">45.79</span><span class="pun">.</span><span class="lit">171.138</span>

运行下面的命令添加它的库。

1. <span class="com">#</span><span class="pln"> rpm </span><span class="pun">-</span><span class="typ">Uhv</span><span class="pln"> http</span><span class="pun">:</span><span class="com">//ceph.com/rpm-giant/el7/noarch/ceph-release-1-0.el7.noarch.rpm</span>

添加 Ceph 仓仓库

或者创建一个新文件并更新 Ceph 库参数，别忘了替换你当前的 Release 和版本号。

1. <span class="pun">[</span><span class="pln">root@ceph</span><span class="pun">-</span><span class="pln">storage </span><span class="pun">~]#</span><span class="kwd">vi</span><span class="pun">/</span><span class="pln">etc</span><span class="pun">/</span><span class="kwd">yum</span><span class="pun">.</span><span class="pln">repos</span><span class="pun">.</span><span class="pln">d</span><span class="pun">/</span><span class="pln">ceph</span><span class="pun">.</span><span class="pln">repo</span>

1. <span class="pun">[</span><span class="pln">ceph</span><span class="pun">-</span><span class="pln">noarch</span><span class="pun">]</span>
2. <span class="pln">name</span><span class="pun">=</span><span class="typ">Ceph</span><span class="pln"> noarch packages</span>
3. <span class="pln">baseurl</span><span class="pun">=</span><span class="pln">http</span><span class="pun">:</span><span class="com">//ceph.com/rpm-{ceph-release}/{distro}/noarch</span>
4. <span class="pln">enabled</span><span class="pun">=</span><span class="lit">1</span>
5. <span class="pln">gpgcheck</span><span class="pun">=</span><span class="lit">1</span>
6. <span class="pln">type</span><span class="pun">=</span><span class="pln">rpm</span><span class="pun">-</span><span class="pln">md</span>
7. <span class="pln">gpgkey</span><span class="pun">=</span><span class="pln">https</span><span class="pun">:</span><span class="com">//ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc</span>

之后更新你的系统并安装 ceph-deploy 软件包。

安装 ceph-deploy 软件包

我们运行下面的命令以及 ceph-deploy 安装命令来更新系统以及最新的 ceph 库和其它软件包。

1. <span class="com">#</span><span class="kwd">yum</span><span class="pln"> update </span><span class="pun">-</span><span class="pln">y </span><span class="pun">&&</span><span class="kwd">yum</span><span class="pln"> install ceph</span><span class="pun">-</span><span class="pln">deploy </span><span class="pun">-</span><span class="pln">y</span>

配置集群

使用下面的命令在 ceph 管理节点上新建一个目录并进入新目录，用于收集所有输出文件和日志。

1. <span class="com">#</span><span class="kwd">mkdir</span><span class="pun">~/</span><span class="pln">ceph</span><span class="pun">-</span><span class="pln">cluster</span>
2. <span class="com">#</span><span class="kwd">cd</span><span class="pun">~/</span><span class="pln">ceph</span><span class="pun">-</span><span class="pln">cluster</span>

1. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy </span><span class="kwd">new</span><span class="pln"> storage</span>

设置 ceph 集群

如果成功执行了上面的命令，你会看到它新建了配置文件。

现在配置 Ceph 默认的配置文件，用任意编辑器打开它并在会影响你公共网络的 global 参数下面添加以下两行。

1. <span class="com">#</span><span class="kwd">vim</span><span class="pln"> ceph</span><span class="pun">.</span><span class="pln">conf</span>
2. <span class="pln">osd pool </span><span class="kwd">default</span><span class="kwd">size</span><span class="pun">=</span><span class="lit">1</span>
3. <span class="kwd">public</span><span class="pln"> network </span><span class="pun">=</span><span class="lit">45.79</span><span class="pun">.</span><span class="lit">0.0</span><span class="pun">/</span><span class="lit">16</span>

安装 Ceph

现在我们准备在和 Ceph 集群关联的每个节点上安装 Ceph。我们使用下面的命令在 ceph-storage 和 ceph-node 上安装 Ceph。

1. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy install ceph</span><span class="pun">-</span><span class="pln">node ceph</span><span class="pun">-</span><span class="pln">storage</span>

安装 ceph

处理所有所需仓库和安装所需软件包会需要一些时间。

当两个节点上的 ceph 安装过程都完成后，我们下一步会通过在相同节点上运行以下命令创建监视器并收集密钥。

1. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy mon create</span><span class="pun">-</span><span class="pln">initial</span>

Ceph 初始化监视器

设置 OSD 和 OSD 守护进程

现在我们会设置磁盘存储，首先运行下面的命令列出你所有可用的磁盘。

1. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy disk </span><span class="kwd">list</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">storage</span>

结果中会列出你存储节点中使用的磁盘，你会用它们来创建 OSD。让我们运行以下命令，请使用你的磁盘名称。

1. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy disk zap storage</span><span class="pun">:</span><span class="pln">sda</span>
2. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy disk zap storage</span><span class="pun">:</span><span class="pln">sdb</span>

为了最后完成 OSD 配置，运行下面的命令配置日志磁盘以及数据磁盘。

1. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy osd prepare storage</span><span class="pun">:</span><span class="pln">sdb</span><span class="pun">:</span><span class="str">/dev/</span><span class="pln">sda</span>
2. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy osd activate storage</span><span class="pun">:</span><span class="str">/dev/</span><span class="pln">sdb1</span><span class="pun">:</span><span class="str">/dev/</span><span class="pln">sda1</span>

你需要在所有节点上运行相同的命令，它会清除你磁盘上的所有东西。之后为了集群能运转起来，我们需要使用以下命令从 ceph 管理节点复制不同的密钥和配置文件到所有相关节点。

1. <span class="com">#</span><span class="pln"> ceph</span><span class="pun">-</span><span class="pln">deploy admin ceph</span><span class="pun">-</span><span class="pln">node ceph</span><span class="pun">-</span><span class="pln">storage</span>

测试 Ceph

我们快完成了 Ceph 集群设置，让我们在 ceph 管理节点上运行下面的命令检查正在运行的 ceph 状态。

1. <span class="com">#</span><span class="pln"> ceph status</span>
2. <span class="com">#</span><span class="pln"> ceph health</span>
3. <span class="pln">HEALTH_OK</span>

如果你在 ceph status 中没有看到任何错误信息，就意味着你成功地在 CentOS 7 上安装了 ceph 存储集群。

总结

在这篇详细的文章中我们学习了如何使用两台安装了 CentOS 7 的虚拟机设置 Ceph 存储集群，这能用于备份或者作为用于其它虚拟机的本地存储。我们希望这篇文章能对你有所帮助。当你试着安装的时候记得分享你的经验。

Ceph 的详细介绍：请点这里
Ceph 的下载地址：请点这里

via: http://linoxide.com/storage/setup-red-hat-ceph-storage-centos-7-0/

作者：Kashif Siddique 译者：ictlyh 校对：wxy

本文由 LCTT 原创编译，Linux中国 荣誉推出