公有云(华为)下的高可用负载均衡器
高可用负载均衡器选用方案 VIP+NGINX
| node | ip |
|---|---|
| node1 | 10.0.0.11 |
| node2 | 10.0.0.12 |
| node3 | 10.0.0.13 |
VIP 10.0.0.10
在虚拟机上很容易操作，详情参看在京奋斗者的 nginx和keepalived实现nginx高可用
但是在公有云上就有些问题了，买了三台服务器部署在同一个VPC下，但是配置好Keepalived和Nginx，但是访问不了后端服务。
如下是在三个节点上的初步配置和测试。
root@hw1:~# service keepalived status
● keepalived.service - Keepalive Daemon (LVS and VRRP)
Loaded: loaded (/lib/systemd/system/keepalived.service; enabled; vendor preset: enabl
Active: active (running) since Tue 2020-02-18 17:48:57 CST; 2min 40s ago
Process: 9058 ExecStart=/usr/sbin/keepalived $DAEMON_ARGS (code=exited, status=0/SUCCE
Main PID: 9067 (keepalived)
Tasks: 3 (limit: 4662)
CGroup: /system.slice/keepalived.service
├─9067 /usr/sbin/keepalived
├─9072 /usr/sbin/keepalived
└─9073 /usr/sbin/keepalived
Feb 18 17:48:57 hw1 Keepalived_vrrp[9073]: WARNING - default user 'keepalived_script' fo
Feb 18 17:48:57 hw1 Keepalived_vrrp[9073]: Unsafe permissions found for script '/etc/kee
Feb 18 17:48:57 hw1 Keepalived_vrrp[9073]: SECURITY VIOLATION - scripts are being execut
Feb 18 17:48:57 hw1 Keepalived_vrrp[9073]: Using LinkWatch kernel netlink reflector...
Feb 18 17:48:57 hw1 Keepalived_vrrp[9073]: VRRP_Instance(VI_1) Entering BACKUP STATE
Feb 18 17:48:57 hw1 Keepalived_vrrp[9073]: VRRP_Script(chk_nginx) succeeded
Feb 18 17:50:12 hw1 Keepalived_vrrp[9073]: VRRP_Instance(VI_1) Transition to MASTER STAT
Feb 18 17:50:13 hw1 Keepalived_vrrp[9073]: VRRP_Instance(VI_1) Entering MASTER STATE
Feb 18 17:50:18 hw1 Keepalived_vrrp[9073]: VRRP_Instance(VI_1) Received advert with high
Feb 18 17:50:18 hw1 Keepalived_vrrp[9073]: VRRP_Instance(VI_1) Entering BACKUP STATE
root@hw1:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw1:~#
同样的keepalived都是启动的
root@hw2:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw3:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw3:~#
并且查看eth0网卡，并没有绑定上虚拟IP。
查阅资料发现，公有云为了安全，禁止我们直接配置一个IP进行漂移。
好在，腾讯云提供了高可用的虚拟IP可以完成keepalived+vip的任务。
我所使用的华为云，只有虚拟IP，并且每个虚拟IP只能绑定一个服务器。只能绑定一个服务器那还怎么漂移？
先在vpc上申请一个虚拟IP.
给node3绑定了VIP，在三个节点上测试。发现node3确实可以通过VIP访问到对应的服务了，但是其他节点仍是不能访问，且查看node3的eth0网卡，发现并没有绑定VIP。
root@hw1:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw1:~#
root@hw2:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw2:~#
root@hw3:~# curl 10.0.0.10
Welcome to nginx03!
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
Welcome to nginx03!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
root@hw3:~# ifconfig |grep eth0 -n7
2- inet 172.17.0.1 netmask 255.255.0.0 broadcast 172.17.255.255
3- ether 02:42:6a:c3:06:a5 txqueuelen 0 (Ethernet)
4- RX packets 0 bytes 0 (0.0 B)
5- RX errors 0 dropped 0 overruns 0 frame 0
6- TX packets 0 bytes 0 (0.0 B)
7- TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
8-
9:eth0: flags=4163 mtu 1500
10- inet 10.0.0.13 netmask 255.255.255.0 broadcast 10.0.0.255
11- inet6 fe80::f816:3eff:fe60:c6ec prefixlen 64 scopeid 0x20
12- ether fa:16:3e:60:c6:ec txqueuelen 1000 (Ethernet)
13- RX packets 220545 bytes 312462163 (312.4 MB)
14- RX errors 0 dropped 0 overruns 0 frame 0
15- TX packets 66533 bytes 5748953 (5.7 MB)
16- TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
root@hw3:~#
接着测试关闭node3的keepalived服务。发现，vip确实漂移了，在node2上可以通过VIP访问到服务，但是其他节点都无法访问了。
root@hw3:~# service keepalived stop
root@hw3:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw3:~#
root@hw2:~# curl 10.0.0.10
Welcome to nginx02!
...
Welcome to nginx02!
...
root@hw2:~#
root@hw1:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw1:~#
只能在VIP漂移到的节点上通过VIP访问本节点的服务，这叫什么高可用？
继续查阅资料，发现一篇博客Centos7.2下基于Nginx+Keepalived搭建高可用负载均衡(一.基于Keepalived搭建HA体系)
在华为云上也是通过虚拟IP实现了所以的IP漂移，但是比较恶心的是，他只是在管理node1的keeaplived服务后，再去node2上访问vip获取到服务的。也就是本文上述的测试过程。这根本说不通好吧。
继续查资料，羡慕腾讯云的高可用虚拟IP，再查华为云的虚拟IP文档，在VPC虚拟IP接口操作指导处发现了高可用字样。但是那一页写的是啥啥啥呀。
关闭源和目的检查（适用于高可用负载均衡集群场景）
尝试把所有主机上的源和目的检查关闭(在ecs主机网卡标签处关闭)。再次测试。所有节点都可以通过VIP访问服务了。注意刚在我们关闭了node3的keepalived的服务。
root@hw1:~# curl 10.0.0.10
Welcome to nginx02!
...
...
root@hw1:~#
root@hw2:~# curl 10.0.0.10
Welcome to nginx02!
...
Welcome to nginx02!
...
root@hw2:~#
root@hw3:~# service keepalived stop
root@hw3:~# curl 10.0.0.10
curl: (7) Failed to connect to 10.0.0.10 port 80: No route to host
root@hw3:~# curl 10.0.0.10
Welcome to nginx02!
...
...
root@hw3:~#
测试一下VIP漂移。
上述结果显示现在VIP在node2上，关闭node2上的keepalived，注意这时关闭了node3和node2的keepalived服务，VIP应该会漂移到node1上。如下的测试结果也证实了这一点，node2关闭keepalived服务后，访问VIP，访问到了node1上的服务。其他节点也均能通过VIP访问node1上的服务。这才是真正的高可用及IP漂移验证成功嘛！
root@hw2:~# service keepalived stop
root@hw2:~# curl 10.0.0.10
Welcome to nginx01!
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
Welcome to nginx01!
If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.
For online documentation and support please refer to
nginx.org.
Commercial support is available at
nginx.com.
Thank you for using nginx.
root@hw2:~#
root@hw1:~# curl 10.0.0.10
Welcome to nginx01!
...
...
root@hw1:~#
root@hw3:~# curl 10.0.0.10
Welcome to nginx01!
...
...
root@hw3:~#
总结，在华为云上要使用keepalived+vip实现IP漂移，需要做的是在保证所以节点在同一个vpc下，并且每个节点配置好keepalived,还需要在这个vpc下申请一个虚拟IP，最重要的是每个节点要关闭源和目的检查，否则也只能体验到VIP漂移，并没有什么作用。
回头看，华为云给虚拟IP的限制，每个虚拟IP只能绑定一个服务器。事实上测试发现，keepalived+vip根本不需要给服务器绑定虚拟IP，只需要我们申请一个就好。这个限制坑呀，明明没啥作用。但真正要注意的是关闭目的和源检查。
参考
华为云 【华为云网络技术分享】【第六弹】VIP特性典型应用案例指导
华为云 关闭源和目的检查（适用于高可用负载均衡集群场景）
腾讯云 VPC 内通过 keepalived 搭建高可用主备集群
腾讯云 高可用虚拟 IP
阿里云支持使用Keepalived搭建负载均衡软件吗？
Centos7.2下基于Nginx+Keepalived搭建高可用负载均衡(一.基于Keepalived搭建HA体系)
作者：longtails
