?
?
ceph分布式存储-集群容量评估
?
?
1.环境介绍
1.1软件环境
ceph集群：
mon：ceph-xxx-osd02.ys,ceph-xxx-osd03.ys,ceph-xxx-osd01.ys
osd:36
mds：ceph-xxx-osd04.ys=up:active
ceph版本：
cephversion12.2.2.3(277c813c8cdeb79eba8a11bfe08faf1bf7809f07)luminous(stable)
os系统：
CentOSLinuxrelease7.2.1511(Core)
3.10.0-514.16.1.el7.x86_64
1.2硬件环境
cpu型号：Intel(R)Xeon(R)CPUE5-2630v4@2.20GHz
cpu核数：40
硬盘：hdd
2.测试场景
2.1单个磁盘的能力
下表以测试随机写IOPS的命令为例，说明命令中各种参数的含义。
czt01h3dvw
iz6he9e5o8
2.1.1顺序读(blocksize4M)
//blocksize是4M,30个线程并发，持续时间200s
测试结果：带宽：377MB/s平均IOPS：89运行时长：2848ms耗时：1249ms
$fio-direct=1-iodepth=128-rw=read-ioengine=libaio-bs=4M-size=1G-numjobs=1-runtime=1000-group_reporting-filename=/var/lib/ceph/osd/ceph-3/test0012-name=Read_PPS_Testing
Read_PPS_Testing:(g=0):rw=read,bs=(R)4096KiB-4096KiB,(W)4096KiB-4096KiB,(T)4096KiB-4096KiB,ioengine=libaio,iodepth=128
fio-3.1
Starting1process
Jobs:1(f=1):[R(1)][30.0%][r=232MiB/s,w=0KiB/s][r=58,w=0IOPS][eta00m:07s]
Read_PPS_Testing:(groupid=0,jobs=1):err=0:pid=26652:ThuJun716:34:272018
read:IOPS=89,BW=360MiB/s(377MB/s)(1024MiB/2848msec)
slat(usec):min=623,max=54269,avg=10938.52,stdev=8604.56
clat(msec):min=46,max=2750,avg=1238.56,stdev=741.72
lat(msec):min=47,max=2751,avg=1249.50,stdev=743.03
clatpercentiles(msec):
|1.00th=[48],5.00th=[56],10.00th=[65],20.00th=[342],
|30.00th=[802],40.00th=[1200],50.00th=[1401],60.00th=[1586],
|70.00th=[1787],80.00th=[1972],90.00th=[2140],95.00th=[2165],
|99.00th=[2198],99.50th=[2198],99.90th=[2735],99.95th=[2735],
|99.99th=[2735]
bw(KiB/s):min=155648,max=253952,per=57.11%,avg=210261.33,stdev=50053.95,samples=3
iops:min=38,max=62,avg=51.33,stdev=12.22,samples=3
lat(msec):50=1.95%,100=13.67%,250=3.12%,500=5.08%,750=5.08%
lat(msec):1000=5.86%,2000=46.48%,=2000=18.75%
cpu:usr=0.00%,sys=9.41%,c