ubuntu+haproxy+keepalived主主負載
一.測試環境：
系統：ubuntu server 12.04
haproxy版本：1.4.24
keepalived版本：keepalived-1.2.7
haporxy01：eth0：172.16.1.36 eth1：192.168.100.36
haporxy02：eth0：172.16.1.37 eth1：192.168.100.37
vip1：172.16.1.30
vip2：172.16.1.31
nignx1 website顯示：nginx1
nginx2 website顯示：ningx2
nginx3 website顯示：nginx3
nginx4 website顯示：nginx4
二.網絡結構
user
|
|
(vip1) | (vip2)
haproxy01-------keepalived-------haproxy02
/ \ / \
/ \ / \
/ \ / \
/ \ / \
--------------------------------------------------------
| nginx1 nginx2 nginx3 nginx4 |
--------------------------------------------------------
三.安裝
1.安裝keepalvied
主機haproxy01：
wget http://www.keepalived.org/software/keepalived-1.2.7.tar.gz
tar xf keepalived-1.2.7.tar.gz
cd keepalived-1.2.7
./configure --prefix=/usr/local/
提示：
checking for openssl/ssl.h... no
configure: error:
!!! OpenSSL is not properly installed on your system. !!!
!!! Can not include OpenSSL headers files. !!!
安裝libssl.dev
apt-get install libssl.dev
繼續：
./configure --prefix=/usr/local/
提示：
checking for poptGetContext in -lpopt... no
configure: error: Popt libraries is required
安裝libpopt-dev
apt-get install libpopt-dev
繼續：
./configure --prefix=/usr/local/
make
make install
2.編輯keepalived.conf
mkdir /etc/keepalived/
vi /etc/keepalived/keepalived.conf
######## keepalived.conf ########
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state MASTER # haproxy02:BACKUP
interface eth0
virtual_router_id 51
priority 91 # 比haproxy02大
advert_int 1
authentication {
auth_type PASS
auth_pass 123456789
}
virtual_ipaddress {
172.16.1.30
}
vrrp_instance VI_2 {
state BACKUP # haproxy02:MASTER
interface eth0
virtual_router_id 52
priority 90 # 比haproxy02小
advert_int 1
authentication {
auth_type PASS
auth_pass 123456789
}
virtual_ipaddress {
172.16.1.31
}
######## keepalived.conf ########
3.啟動keepalived
/usr/local/sbin/keepalvied -f /etc/keepalived/keepalived.conf
4.查看vip
ip addr
顯示：
eth0：
inet 172.16.1.30/32 scope global eth0
inet 172.16.1.31/32 scope global eth0 #（haproxy02的keepalived未啟動，vip2在haproxy01中，haproxy02中keepalived啟動後，vip2自動跳轉到haproxy02中eth0上）
主機haproxy02：
keepalived安裝同上，keepalived.conf如下：
######## keepalived.conf ########
global_defs {
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP # haproxy01:MASTER
interface eth0
virtual_router_id 51
priority 90 # 比haproxy01小
advert_int 1
authentication {
auth_type PASS
auth_pass 123456789
}
virtual_ipaddress {
172.16.1.30
}
vrrp_instance VI_2 {
state MASTER # haproxy01:BACKUP
interface eth0
virtual_router_id 52
priority 91 # 比haproxy01大
advert_int 1
authentication {
auth_type PASS
auth_pass 123456789
}
virtual_ipaddress {
172.16.1.31
}
######## keepalived.conf ########
啟動keepalived
/usr/local/sbin/keepalvied -f /etc/keepalived/keepalived.conf
查看vip：
ip addr
顯示：
eth0：
inet 172.16.1.31/32 scope global eth0 # (haproxy中eth0上vip2消失，只有vip1)
停止任意一台keepalived服務，另外一台均可自動生成vip1和vip2，確保2台haproxy高可用性
2.安裝haproxy
wget http://haproxy.1wt.eu/download/1.4/src/haproxy-1.4.24.tar.gz
tar xf haproxy-1.4.24.tar.gz
cd haproxy-1.4.24
查看ubuntu版本信息：
uname -a
顯示：
Linux ubuntu37 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
make TARGET=37-ubuntu
make PREFIX=/usr/local/haproxy install
建立haproxy用戶
useradd haproxy
2.編輯haproxy.cfg
mkdir /etc/haproxy
vi /etc/haproxy/haproxy.cfg
######## haproxy.cfg ########
global
log 127.0.0.1 local0
maxconn 51200
user haproxy
group haproxy
daemon
defaults
log 127.0.0.1 local3
mode http
option dontlognull
balance roundrobin
retries 2
option redispatch
option abortonclose
maxconn 51200
contimeout 5000
clitimeout 50000
srvtimeout 50000
listen haproxy01
bind 172.16.1.30:80
mode http
option httpclose
option forceclose
option forwardfor
option originalto
option redispatch
balance leastconn
timeout check 5s
stats uri /stats # haproxy負載監控頁面 例：http://172.16.1.30/stats
stats refresh 15s # 監控頁面刷新時間
stats realm baison-test-Haproxy
stats auth admin:admin # 監控頁面賬號密碼
stats hide-version # 隱藏haproxy版本信息
cookie SESSION_COOKIE insert indirect nocache
maxconn 40960
server nginx1 192.168.100.11:80 weight 1 cookie nginx1 check inter 2000 rise 2 fall 3
server nginx2 192.168.100.12:80 weight 1 cookie nginx2 check inter 2000 rise 2 fall 3
listen haproxy02
bind 172.16.1.31:80
mode http
option httpclose
option forceclose
option forwardfor
option originalto
option redispatch
balance leastconn
cookie SESSION_COOKIE insert indirect nocache
maxconn 40960
server nginx3 192.168.100.13:80 weight 1 cookie nginx3 check inter 2000 rise 2 fall 3
server nginx4 192.168.100.14:80 weight 1 cookie nginx4 check inter 2000 rise 2 fall 3
######## haproxy.cfg ########
啟動haproxy：
/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg
主機haproxy02：
安裝同上，haproxy.cfg同上
啟動haproxy：
/usr/local/haproxy/sbin/haproxy -f /etc/haproxy/haproxy.cfg
四：測試
通過浏覽器訪問：
http://172.16.1.30
頁面顯示：nginx1、nginx2輪詢切換
http://172.16.1.31
頁面顯示：nginx3、nginx4輪詢切換
可通過監控頁面查看負載情況：
http://172.16.1.30/stats
停止任意一台haproxy服務，站點均不受影響
五：haproxy日志
1.在/etc/rsyslog.conf中：
添加haproxy日志路徑:
local0.* /var/log/haproxy.log
local3.* /var/log/haproxy.log
取消注釋:
#$ModLoad imudp ==》$ModLoad imudp
#$UDPServerRun 514 ==》$UDPServerRun 514
重啟rsyslog服務
service rsyslog restart
2.核實services文件
grep 514 /etc/services
顯示：
syslog 514/udp 為ok!!!
如不存在，手動添加：
vi /etc/default/rsyslog
RSYSLOGD_OPTIONS="-r -c 5"
重啟syslog服務，
service rsyslog restart
六：優化haproxy服務器
1.內核結果如下：
sysctl -p
net.ipv4.ip_forward = 1
net.ipv4.ip_nonlocal_bind = 1
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65535
net.nf_conntrack_max = 1310720
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_fin_timeout = 15
net.core.netdev_max_backlog = 4096
net.core.rmem_max = 16777216
net.core.somaxconn = 1310720
net.core.wmem_max = 16777216
net.ipv4.tcp_max_syn_backlog = 20480
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_no_metrics_save = 1
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_syn_retries = 2
net.ipv4.tcp_synack_retries = 2
net.ipv4.tcp_wmem = 4096 65536 16777216
vm.min_free_kbytes = 65536
net.ipv4.tcp_sack = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_tw_recycle = 1
拷貝至sysctl.conf中
vi /etc/sysctl.conf
2.ulimit數值永久化
執行：
ulimit -SHn
顯示：
1024
永久化：
a.在/etc/pam.d/login中，添加：
pam_limits.so (有時候系統默認添加)
查找文件位置：
find / -name pam_limits.so
顯示：
/lib/x86_64-linux-gnu/security/pam_limits.so
b.在/etc/security/limits.conf中，添加：
root soft nofile 10240 #實際值
root hard nofile 10240 #實際值
c.修改 /etc/rc.local 添加：
echo 8061540 > /proc/sys/fs/file-max
執行：
echo 8061540 > /proc/sys/fs/file-max
3.nginx服務器內核參數優化
sysctl -p
net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_intvl = 2
net.ipv4.tcp_keepalive_probes = 2
net.ipv4.tcp_keepalive_time = 120
net.ipv4.tcp_syn_retries = 10
net.ipv4.tcp_sack = 1
net.ipv4.tcp_timestamps = 1
