## 问题背景 GPDB是中央控制节点式的架构,在一个 GreenPlum 集群中,有一个 Master 节点和多个 Segment 节点。Master 是中央控制节点,Segment 是数据存放节点。所有的Segment节点平等,均由Master管理。架构如下图: ![](https://oss.showapi.com/doc/2096/58/742cbea5-3b3d-4b92-82c1-8bf7dcfd875c.jpg) GreenPlum架构图 当GP Master出现问题的时候,可以通过外部的HA监控模块发现并激活备库,Standby Master 正常后删除原来的 Master 进行重建备库。 而 Segment 的修复与此不同!由上图可知,Segment 也分为主备,称为 Primary 和 Mirror,Mirror 是 Primary 的备。Primary与Mirror之间强同步保证数据一致性和可靠性,其间的监控与切换则由Master的FTS模块负责。当FTS发现Primary宕机、Mirror健康后会激活Mirror,并标记Primary为’d’,Mirror进入 ChangeTracking 状态。(详细的原理此处不作赘述,有兴趣可以参考本期月报的[GPDB · 特性分析· GreenPlum Segment事务一致性与异常处理](http://mysql.taobao.org/monthly/2016/04/02/)和上期的[GPDB · 特性分析· GreenPlum FTS 机制](http://mysql.taobao.org/monthly/2016/03/08/)) 当有Segment被标记为’d’后,Master将不会对其做处理,GP实例的启动(重启)也会将其忽略。这个时候,整个GP集群是处于有风险的状况中: 1. 切过去的Mirror压力增大(需要做change tracking); 2. 节点单点,可靠性风险加大。 这个时候需要及时地对Segment进行修复。 ## GP的Segment修复 GP提供了一系列的控制脚本用于对GP进行操作,其中用于修复Segment的是gprecoverseg。使用方式比较简单,有限的几个主要参数如下: * -i 主要参数,用于指定一个配置文件,该配置文件描述了需要修复的Segment和修复后的目的位置。 * -F 可选项,指定后,gprecoverseg会将”-i”中指定的或标记”d”的实例删除,并从活着的Mirror复制一个完整一份到目标位置。 * -r 当FTS发现有Primary宕机并进行主备切换,在gprecoverseg修复后,担当Primary的Mirror角色并不会立即切换回来,就会导致部分主机上活跃的Segment过多从而引起性能瓶颈。因此需要恢复Segment原先的角色,称为re-balance。 举个使用的例子: 下面是一个正常的实例, ~~~ $ gpstate -s /opt/python27/lib/python2.7/site-packages/Crypto/Util/number.py:57: PowmInsecureWarning: Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability. _warn("Not using mpz_powm_sec. You should rebuild using libgmp >= 5 to avoid timing attack vulnerability.", PowmInsecureWarning) 20160418:21:39:29:016547 gpstate:host1:gpuser-[INFO]:-Starting gpstate with args: -s 20160418:21:39:29:016547 gpstate:host1:gpuser-[INFO]:-local Greenplum Version: 'postgres (Greenplum Database) 4.3.99.00 build dev' 20160418:21:39:29:016547 gpstate:host1:gpuser-[INFO]:-master Greenplum Version: 'PostgreSQL 8.3 (Greenplum Database 4.3.99.00 build dev) compiled on Apr 11 2016 22:02:39' 20160418:21:39:29:016547 gpstate:host1:gpuser-[INFO]:-Obtaining Segment details from master... 20160418:21:39:29:016547 gpstate:host1:gpuser-[INFO]:-Gathering data from segments... . 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:----------------------------------------------------- 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:--Master Configuration & Status 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:----------------------------------------------------- 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Master host = host1 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Master postgres process ID = 72447 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Master data directory = /workspace/gpuser/3007 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Master port = 3007 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Master current role = dispatch 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Greenplum initsystem version = 4.3.99.00 build dev 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Greenplum current version = PostgreSQL 8.3 (Greenplum Database 4.3.99.00 build dev) compiled on Apr 11 2016 22:02:39 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Postgres version = 8.3 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Master standby = host2 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Standby master state = Standby host passive 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:----------------------------------------------------- 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:-Segment Instance Status Report 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:----------------------------------------------------- 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Segment Info 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Hostname = host1 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Address = host1 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Datadir = /workspace/gpuser/3008 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Port = 3008 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Mirroring Info 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Current role = Primary 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Preferred role = Primary 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Mirror status = Synchronized 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Status 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- PID = 72388 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Configuration reports status as = Up 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Database status = Up 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:----------------------------------------------------- ...... [INFO]:----------------------------------------------------- 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Segment Info 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Hostname = host1 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Address = host1 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Datadir = /workspace/gpuser/3012 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Port = 3012 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Mirroring Info 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Current role = Mirror 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Preferred role = Mirror 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Mirror status = Synchronized 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Status 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- PID = 75247 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Configuration reports status as = Up 20160418:21:39:30:016547 gpstate:host1:gpuser-[INFO]:- Segment status = Up ~~~ 选择一个kill之后(如3012这个端口的实例),执行gprecoverseg,如下: ~~~ 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host1 -p 3008 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host1 -p 3008 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3014 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3014 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host1 -p 3010 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host1 -p 3010 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3015 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3015 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3008 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3008 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host1 -p 3011 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host1 -p 3011 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3013 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3013 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host1 -p 3012 20160418:21:40:58:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host1 -p 3012 ...... 20160418:21:41:18:017989 gpstate:host1:gpuser-[DEBUG]:-[worker6] finished cmd: Get segment status cmdStr='sshpass -e ssh -o 'StrictHostKeyChecking no' host1 ". /workspace/gpdb/greenplum_path.sh; $GPHOME/bin/gp_primarymirror -h host1 -p 3012"' had result: cmd had rc=15 completed=True halted=False stdout='' stderr='failed to connect: Connection refused (errno: 111) Retrying no 1 failed to connect: Connection refused (errno: 111) Retrying no 2 ...... 20160418:21:41:18:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Encountered error Not ready to connect to database mode: PrimarySegment segmentState: Fault dataState: InSync faultType: FaultMirror mode: PrimarySegment segmentState: Fault dataState: InSync faultType: FaultMirror ~~~ 这个时候连接这个实例去获取信息是失败的,失败的原因后面再说。这个时候失败后会重试5次,当再一次尝试的时候发现了不同: ~~~ 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host1 -p 3008 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host1 -p 3008 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3014 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3014 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host1 -p 3010 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host1 -p 3010 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3015 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3015 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3008 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3008 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host1 -p 3011 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host1 -p 3011 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Sending msg getStatus and cmdStr $GPHOME/bin/gp_primarymirror -h host2 -p 3013 20160418:21:41:23:017989 gprecoverseg:host1:gpuser-[DEBUG]:-Adding cmd to work_queue: $GPHOME/bin/gp_primarymirror -h host2 -p 3013 ~~~ 会发现,少了一个Segment的命令,而这个Segment正是刚才kill的Segment。继续往下看执行结果,gprecoverseg执行了下面的內容: ~~~ 20160418:23:16:20:085203 gprecoverseg:host1:gpuser-[DEBUG]:-[worker7] finished cmd: Get segment status information cmdStr='sshpass -e ssh -o 'StrictHostKeyChecking no' host2 ". /workspace/gpdb/greenplum_path.sh; $GPHOME/bin/gp_primarymirror -h host2 -p 3013"' had result: cmd had rc=1 completed=True halted=False stdout='' stderr='mode: PrimarySegment segmentState: Ready dataState: InChangeTracking faultType: NotInitialized mode: PrimarySegment segmentState: Ready dataState: InChangeTracking faultType: NotInitialized ' ~~~ 这个实例为什么单独检查呢?而且这个时候如果失败,则会直接退出无法继续执行。 在一系列的检查之后,先更新catalog中的操作记录表: ~~~ UPDATE pg_catalog.gp_segment_configuration ~~~ 之后即调用命令进行数据的恢复: ~~~ /workspace/gpdb/bin/lib/gpconfigurenewsegment -c /workspace/gpuser/3012:3012:false:false:9 -v -B 16 --write-gpid-file-only ~~~ 最后再启动Segment,并更新catalog: ~~~ $GPHOME/sbin/gpsegstart.py -C en_US.utf8:C:C -M quiescent -V 'postgres (Greenplum Database) 4.3.99.00 build dev' -n 4 --era df86ca11ca2fc214_160418165251 -t 600 -v -p KGRwMApTJ2Ric0J5UG9ydCcKcDEKKGRwMgpJMzAxMgooZHAzClMndGFyZ2V0TW9kZScKcDQKUydtaXJyb3InCnA1CnNTJ2RiaWQnCnA2Ckk5CnNTJ2hvc3ROYW1lJwpwNwpTJzEwLjk3LjI0OC43MycKcDgKc1MncGVlclBvcnQnCnA5CkkzNTEzCnNTJ3BlZXJQTVBvcnQnCnAxMApJMzAxMwpzUydwZWVyTmFtZScKcDExClMncnQxYjA3MDI0LnRiYycKcDEyCnNTJ2Z1bGxSZXN5bmNGbGFnJwpwMTMKSTAwCnNTJ21vZGUnCnAxNApTJ3InCnAxNQpzUydob3N0UG9ydCcKcDE2CkkzNTEyCnNzcy4= -D '9|3|m|m|r|d|host1|host1|3012|3512|/workspace/gpuser/3012||' ...... 20160419:01:21:05:042692 gprecoverseg:host1:gpuser-[DEBUG]:-UPDATE pg_catalog.gp_segment_configuration SET mode = 'r', status = 'u' WHERE dbid = 5 20160419:01:21:05:042692 gprecoverseg:host1:gpuser-[DEBUG]:-INSERT INTO gp_configuration_history (time, dbid, "desc") VALUES( now(), 5, 'gprecoverseg: segment resync marking mirrors up and primaries resync: segment mode and status' ) 20160419:01:21:05:042692 gprecoverseg:host1:gpuser-[DEBUG]:-UPDATE pg_catalog.gp_segment_configuration SET mode = 'r', status = 'u' WHERE dbid = 9 20160419:01:21:05:042692 gprecoverseg:host1:gpuser-[DEBUG]:-INSERT INTO gp_configuration_history (time, dbid, "desc") VALUES( now(), 9, 'gprecoverseg: segment resync marking mirrors up and primaries resync: segment mode and status' ) 20160419:01:21:05:042692 gprecoverseg:host1:gpuser-[DEBUG]:-UPDATE gp_fault_strategy ~~~ 这样即是一个完整的gprecoverseg过程。执行过后,对应的Primary和Mirror会进入”r”状态,表示正在做数据同步。 下面来看其中的详细步骤和原理。 ## 实现原理 上面的例子中,遗留了几个问题: * 在gprecoverseg过程中,第一次获取Segment状态是不对的; * 第二次获取Segment信息,比第一次少了一条; * 单独检查了“-h host2 -p 3013”这个实例。 这几个问题在了解了原理后就很容易理解了。想要了解原理,可以先看下执行的步骤。从代码看来,其大致的步骤如下: ### 参数处理 GP的脚本用了较多的环境变量,且不同的脚本、不同的地方略有不同。如gprecoverseg用的就是MASTER_DATA_DIRECTORY,从MASTER_DATA_DIRECTORY指定的目录中得到Master相关的信息(如port)以进行相关操作。 gprecoverseg的参数,最重要的莫过于”-i”了,其指定了需要做修复的Segment,并且可以指定到不同的主机上,例如: ~~~ filespaceOrder= host1:3012:/workspace/gpuser/3012 host2:3012:3512:/workspace/gpuser/3012 ~~~ 具体执行不再赘述。 ### 判断Segment当前的状态 调用gp_primarymirror,向活着的segment发送消息,以判断Segment当前的状态。这是非常重要的一步,也是遇到问题最多的一步,经常会出现问题”Unable to connect to database”。事实上,造成这个失败的原因有很多,比较多的是: * 其对应的Primary(Mirror)也宕机; * 其对应的Primary的状态不对,如已经有gprecoverseg在进行(或执行失败,状态出问题等)。 在做这一步的时候,是依赖gp_segment_configuration中的数据的,即会首先从GP Master上获取相应的数据,与下一步中的描述基本相同。 如果这个Segment被标记为”d”,那么是不会向该Segment发起状态信息请求。 而如果对应的Primary/Mirror都宕机了,他们的状态不会同时为”d”(有可能都为”u”,比如同时异常的时候,FTS不会更新他们)。因此对标记为”u”实际已经宕机的Segment连接获取状态信息的时候,则会报错。这个时候就不是gprecoverseg所能处理的问题了,只能重启整个实例。 回到前面的问题。第一次执行失败即因为Segment的状态尚未更新;第二次执行少了一个Segment,即状态被更新为”d”后不进行连接。 在检查完所有状态为”u”的Segment连接后,则会针对宕机的Mirror进行检查,查看其对应的主库是否正常,可以用于修复数据,即是第三个问题的答案。如: ~~~ stdout='' stderr='mode: PrimarySegment segmentState: Ready dataState: InSync faultType: NotInitialized mode: PrimarySegment segmentState: Ready dataState: InSync faultType: NotInitialized ' ~~~ 或者这样 ~~~ stdout='' stderr='mode: PrimarySegment segmentState: Ready dataState: InChangeTracking faultType: NotInitialized mode: PrimarySegment segmentState: Ready dataState: InChangeTracking faultType: NotInitialized ' ~~~ 正常情况下,当Mirror出现问题,Primary发现后会进入ChangeTracking的状态。在这个状态里,Primary会记录下切换状态时间点之后的变更,用于当Mirror恢复时进行数据同步,而不用每次都做一次全量。 ### 从master获取segment的信息 包括IP、PORT、ROLE、Status、数据目录、临时空间等,如下: ~~~ dbid | content | role | preferred\_role | mode | status | hostname | address | port | replication\_port | oid | fselocation ------+---------+------+----------------+------+--------+---------------+---------------+------+------------------+------+----------------- 1 | -1 | p | p | s | u | host1 | host1 | 3007 | | 3052 | /workspace/gpuser/3007 10 | -1 | m | m | s | u | host2 | host2 | 3007 | | 3052 | /workspace/gpuser/3007 2 | 0 | p | p | s | u | host1 | host1 | 3008 | 3508 | 3052 | /workspace/gpuser/3008 6 | 0 | m | m | s | u | host2 | host2 | 3014 | 3514 | 3052 | /workspace/gpuser/3014 3 | 1 | p | p | s | u | host1 | host1 | 3010 | 3510 | 3052 | /workspace/gpuser/3010 7 | 1 | m | m | s | u | host2 | host2 | 3015 | 3515 | 3052 | /workspace/gpuser/3015 4 | 2 | p | p | s | u | host2 | host2 | 3008 | 3508 | 3052 | /workspace/gpuser/3008 8 | 2 | m | m | s | u | host1 | host1 | 3011 | 3511 | 3052 | /workspace/gpuser/3011 5 | 3 | p | p | s | u | host2 | host2 | 3013 | 3513 | 3052 | /workspace/gpuser/3013 9 | 3 | m | m | s | u | host1 | host1 | 3012 | 3512 | 3052 | /workspace/gpuser/3012 ~~~ IP/PORT/ROLE/STATUS/目录/FILESPACE等信息,后面的Mirror修复列表、临时空间、操作对象的信息都依赖于此。 ### 修复准备 在获取所有的Segment信息后,会针对配置文件、参数等相关信息确定,包括: * Segment修复对象 确定Segment修复对象和数据源,即Primary;需要修复的Segment有可能是多个。并获取需要修复的Segment的相关信息,包括端口、流复制端口、数据目录、临时空间、文件空间等信息,以及是否强制修复等。 * 主机环境 在获取所需要修复的Segment列表后,需要确保所在主机环境是可以的,包括端口占用、目录的占用等有可能冲突的地方。 如果没有指定主机,则会在已有的主机中选择一个。 ### 修复 修复的步骤是: * 关闭宕机的Mirror,并清理shared memory * 确定需要修复的Segment已经被标记为”d” * 如有需要,则进行删除,如”-F”的情况 * 打包压缩、复制数据到目标位置 * 关闭SIGINT、SIG_IGN,更新元数据库,打开SIGINT、SIG_IGN 以上步骤后,即可实现对Segment的本地(in-place)或跨机修复。 ### re-balance 当修复完Segment之后,原先因为Primary宕机而切到Mirror上的Segment并不会主动切回来,这个时候有可能出现性能倾斜而影响性能,因此需要做”re-balance”,执行: ~~~ gprecoverseg -r ~~~ 执行该命令会将role切换为preferred_role,保证整个集群的角色平衡而不致于部分主机Primay更多引起性能瓶颈。