* 硬件规划

(1)Processor
· ceph-osd进程在运行过程中会消耗CPU资源,所以一般会将每一个ceph-osd进程绑定一个
CPU核上。当然如果你使用EC方式,可能需要更多的CPU资源。
· ceph-mon进程并不十分消耗CPU资源,所以不必为ceph-mon进程预留过多的CPU资源。
· ceph-msd也是非常消耗CPU资源的,所以需要提供更多的CPU资源。
(2)内存
 ceph-mon和ceph-mds需要2GB内存,每个ceph-osd进程需要1GB内存,当然2GB更好。
(3)网络规划
 万兆网络现在基本上是运行Ceph必备的,在网络规划上,也尽量考虑分离Cilent和Cluster网
络

* SSD的选择

SATA SSD作为Journal,
Intel SSD DC S3500 Series基本是目前的方案中的首选
http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device

打开超线程
DCU Streamer Prefetcher

关闭节能
for CPUFREQ in sysdevices/system/cpu/cpu*/cpufreq/scaling_govern

grub关闭numa
numa=off

* linux内核优化

如果是SSD,做如下优化
echo noop > /sysblock/sdX/queue/scheduler
如果是机械硬盘
echo deadline > /sys/kernal/sysblock/sdX/queue/scheduler
#
echo noop > /sys/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/queue/scheduler
echo noop > /sys/devices/pci0000:00/0000:00:17.0/ata3/host2/target2:0:0/2:0:0:0/block/sdb/queue/scheduler

磁盘预读调大
root@ops-dell:/home/ops/Downloads# find /sys -name read_ahead_kb
/sys/devices/pci0000:00/0000:00:17.0/ata1/host0/target0:0:0/0:0:0:0/block/sda/queue/read_ahead_kb
/sys/devices/pci0000:00/0000:00:17.0/ata3/host2/target2:0:0/2:0:0:0/block/sdb/queue/read_ahead_kb
/sys/devices/virtual/bdi/0:42/read_ahead_kb
/sys/devices/virtual/bdi/0:40/read_ahead_kb
/sys/devices/virtual/bdi/8:16/read_ahead_kb
/sys/devices/virtual/bdi/8:17-fuseblk/read_ahead_kb
/sys/devices/virtual/bdi/8:0/read_ahead_kb

调整最大进程号
echo 4194303 > /proc/sys/kernel/pid_max

调整cpu频率
echo performance | tee /sys/devices/system/cpu/cpufreq/policy2/scaling_available_governors
echo performance | tee /sys/devices/system/cpu/cpufreq/policy0/scaling_available_governors
echo performance | tee /sys/devices/system/cpu/cpufreq/policy3/scaling_available_governors
echo performance | tee /sys/devices/system/cpu/cpufreq/policy1/scaling_available_governors

* 内存

使用smp架构,关闭numa
尽量避免内存换页 vm.swappiness=0

全闪存环境下 TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES=268435456 但是再编译的时候 需要 --with-jemalloc --without-tcmalloc

* cpu

cpu cgroup绑定脚本
mkdir -p sysfs/cgroup/cpuset/ceph
#CPU Number: 0,1,2,3 = 0-3
echo 0,4 > sysfs/cgroup/cpuset/ceph/cpuset.cpus
#NUMA Node
echo 0 > sysfs/cgroup/cpuset/ceph/cpuset.mems
osd_pid_list=$(ps aux | grep osd | grep -v grep |awk '{print $2}
for osd_pid in ${osd_pid_list}
do
echo ${osd_pid} > sysfs/cgroup/cpuset/ceph/cgroup.procs
done
Copyright © opschina.org 2017 with zzlyzq@gmail.com all right reserved,powered by Gitbook该文件修订时间: 2017-07-11 11:32:48

results matching ""

    No results matching ""