【小工具】hdfs balance脚本（均衡操作）
hdfs balance脚本（均衡操作）前言shell源码执行语句执行结果效果
前言
作者：Jack_Roy
集群增加新节点后，hdfs 节点和disk之间的数据分配十分不均匀，需要进行均衡操作，这里写了一个简单的balance操作脚本
shell源码#!/bin/bash
# 904857600 动态分配balance操作的带宽
# 可执行ethtool bond0命令查看自己的带宽
# 其中bond0是网卡
sudo -u hdfs hadoop dfsadmin -setBalancerBandwidth 904857600
# -threshold 10 各个节点相差10%就开始执行均衡操作
sudo -u hdfs hdfs balancer -policy datanode -threshold 10
执行语句
将源代码复制、粘贴至hdfs_balance.sh ，然后执行：
sh hdfs_balance.sh
执行结果
运行执行命令以后，控制行会打印均衡过程相关的提示信息，如：
19/11/06 19:42:52 INFO balancer.Dispatcher: Successfully moved blk_1103357650_29654216 with size=20550985 from 000.00.00.005:9866:DISK to 000.00.00.009:9866:DISK through 000.00.00.005:9866
19/11/06 19:42:52 INFO balancer.Dispatcher: Successfully moved blk_1074354988_614164 with size=25516685 from 000.00.00.005:9866:DISK to 000.00.00.009:9866:DISK through 000.00.00.005:9866
19/11/06 19:42:52 INFO balancer.Dispatcher: Successfully moved blk_1073965043_224219 with size=113539842 from 000.00.00.002:9866:DISK to 000.00.00.010:9866:DISK through 000.00.00.007:9866
19/11/06 19:42:52 INFO balancer.Dispatcher: Successfully moved blk_1195850710_122155295 with size=128862687 from 000.00.00.005:9866:DISK to 000.00.00.009:9866:DISK through 000.00.00.004:9866
19/11/06 19:42:52 INFO balancer.Dispatcher: Successfully moved blk_1074000301_259477 with size=134217728 from 000.00.00.001:9866:DISK to 000.00.00.007:9866:DISK through 000.00.00.002:9866
19/11/06 19:42:53 INFO balancer.Dispatcher: Successfully moved blk_1108721075_35020878 with size=134217728 from 000.00.00.001:9866:DISK to 000.00.00.007:9866:DISK through 000.00.00.006:9866
19/11/06 19:42:53 INFO balancer.Dispatcher: Successfully moved blk_1073965321_224497 with size=134217728 from 000.00.00.002:9866:DISK to 000.00.00.010:9866:DISK through 000.00.00.009:9866
19/11/06 19:42:53 INFO balancer.Dispatcher: Successfully moved blk_1074145728_404904 with size=11910838 from 000.00.00.001:9866:DISK to 000.00.00.007:9866:DISK through 000.00.00.009:9866
19/11/06 19:42:53 INFO balancer.Dispatcher: Successfully moved blk_1091588579_17848971 with size=134217728 from 000.00.00.004:9866:DISK to 000.00.00.006:9866:DISK through 000.00.00.004:9866
19/11/06 19:42:53 INFO balancer.Dispatcher: Successfully moved blk_1075519943_1779125 with size=134217728 from 000.00.00.002:9866:DISK to 000.00.00.010:9866:DISK through 000.00.00.007:9866
19/11/06 19:42:53 INFO balancer.Dispatcher: Successfully moved blk_1075519802_1778984 with size=134217728 from 000.00.00.002:9866:DISK to 000.00.00.010:9866:DISK through 000.00.00.007:9866
19/11/06 19:42:54 INFO balancer.Dispatcher: Successfully moved blk_1074525331_784507 with size=54867322 from 000.00.00.004:9866:DISK to 000.00.00.006:9866:DISK through 000.00.00.010:9866
19/11/06 19:42:54 INFO balancer.Dispatcher: Successfully moved blk_1073965066_224242 with size=134217728 from 000.00.00.002:9866:DISK to 000.00.00.010:9866:DISK through 000.00.00.007:9866
19/11/06 19:42:54 INFO balancer.Dispatcher: Successfully moved blk_1091114087_17374323 with size=134217728 from 000.00.00.004:9866:DISK to 000.00.00.006:9866:DISK through 000.00.00.004:9866
19/11/06 19:42:54 INFO balancer.Dispatcher: Successfully moved blk_1074525394_784570 with size=61481483 from 000.00.00.004:9866:DISK to 000.00.00.006:9866:DISK through 000.00.00.010:9866
19/11/06 19:42:54 INFO balancer.Dispatcher: Successfully moved blk_1091112258_17372494 with size=134217728 from 000.00.00.004:9866:DISK to 000.00.00.006:9866:DISK through 000.00.00.004:9866
19/11/06 19:42:54 INFO balancer.Dispatcher: Successfully moved blk_1074145961_405137 with size=13746781 from 000.00.00.001:9866:DISK to 000.00.00.007:9866:DISK through 000.00.00.001:9866
19/11/06 19:42:55 INFO balancer.Dispatcher: Successfully moved blk_1073962387_221563 with size=10956569 from 000.00.00.005:9866:DISK to 000.00.00.009:9866:DISK through 000.00.00.002:9866
19/11/06 19:42:55 INFO balancer.Dispatcher: Successfully moved blk_1074663041_922217 with size=134217728 from 000.00.00.005:9866:DISK to 000.00.00.009:9866:DISK through 000.00.00.005:9866
19/11/06 19:42:55 INFO balancer.Dispatcher: Successfully moved blk_1073984129_243305 with size=97805226 from 000.00.00.004:9866:DISK to 000.00.00.006:9866:DISK through 000.00.00.001:9866
可以见到连续的如同：
"Successfully moved blk****** with size=block.size from 数据倾斜节点IP :PORT:DISK to 新节点IP :PORT:DISK through 中间节点IP :PORT "
的提示信息。
变为：
大家在实际操作中如果有更高效的调优办法，欢迎在评论区留言交流！
作者：Jack_Roy
