db2 purecale是ibm推出的基於共享磁盤的，可擴展的數據庫集群軟件，類似於oracle的rac。具體功能這裡就不過多介紹了，直接介紹在rehdat上安裝purescale的實戰經歷
1 安裝手冊
軟硬件版本
IB交換機：Mellanox
PC服務器：X3850 X5
存儲版本：DS5100
OS版本：RHEL6.1
DB2版本：db2 purescale10.1.0.2
建議使用purescale認證的軟硬件版本，否則可能會碰到各種莫名其妙的問題
主機名
IP地址
管理
IB地址
CF
member
Wtydb21
192.168.2.101
192.168.1.101
192.168.0.101
primary
YES
Wtydb22
192.168.2.102
192.168.1.102
192.168.0.102
second
YES
Wtydb23
192.168.2.103
192.168.1.103
192.168.0.103
YES
IB地址規劃
管理IP
浮動IP
IB
Switch-1
192.168.1.104
192.168.1.106
192.168.0.201
Switch-2
192.168.1.105
192.168.0.202
Wtydb21為NFS server，其它主機為client，為了在各服務器安裝軟件包方便，建議配置yum
1，手工安裝nfs軟件包
rpm –qa nfs
修改為nfs默認打開,所有服務器設置
[root@stydb1 ~]# chkconfig --list nfs
nfs 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@stydb1 ~]# chkconfig nfs on
[root@stydb1 ~]# chkconfig --list nfs
nfs 0:off 1:off 2:on 3:on 4:on 5:on 6:off
service portmap start
service nfs start
service rpcbind start
2, 確保nfs server端防火牆處於關閉狀態
service iptables stop
3，nfs server端配置
A，Mount NFS磁盤mount nfs disks，本例共享/data目錄
mount /dev/mapper/vg_1-lv_data /data
B，vi /etc/fstab，添加如下一行，修改為自動掛載
/dev/mapper/vg_1-lv_data /data ext4 defaults 1 2
C，nfs共享/data目錄
vi /etc/exports
/data *(rw)
/data wtydb22(rw) wtydb23(rw)
/usr/sbin/exportfs -r
4，客戶端設置
A，查看nfs服務器端的設置
[root@wtydb22 ~]# showmount -e wtydb21
Export list for wtydb21:
/data wtydb23,wtydb22
B，vi /etc/fstab
wtydb21:/data /data nfs rw
rpm -qa|grep nfs
C，手工mount
mkdir /data
mount wtydb21:/data/ /data/
1，新建目錄mkidr/data/rhel6.1
cp 光盤內容到/data/rhel6.1目錄，也可以直接使用光盤或者iso文件的介質文件
2，配置yum（三台服務器上配置）
vi/etc/yum.repos.d/rhel6.repo
[Server]
name=rhel6server
baseurl=file:///data/rhel6.1/Server/
enable=1
gpcheck=1
gpgkey=file:///data/rhel6.1/RPM-GPG-KEY-redhat-release
3，測試
yum list
也可通過iso文件實現
mount -o loop /software/rhel-server-6.1-x86_64-dvd.iso /mnt/cdrom
vi /etc/yum.repos.d/rhel6.repo
[Server]
name=rhel6server
baseurl=file:///mnt/cdrom/Server/
enable=1
gpcheck=1
gpgkey=file:///mnt/cdrom/RPM-GPG-KEY-redhat-release
1， 生成ssh key，在~/.ssh/目錄下會生成兩個文件
ssh-keygen -t dsa，一路默認回車
在.ssh目錄下
id_dsa (the private key) and id_dsa.pub (the public key) for DSA encryption
2，把三台主機的id_dsa.pub文件的內容復制到authorized_keys文件中，把authorized_keys放於三台主機的.ssh目錄下
正常情況下，GPFS不需要單獨安裝，隨著安裝db2 purescale軟件會自動完成gpfs安裝。.
[root@wtydb22 data]# cd /data/esefp2/server/db2/linuxamd64/gpfs
[root@wtydb22 gpfs]# ls
base db2ckgpfs db2gutil errMsg fp installGPFS uninstallGPFS
[root@wtydb22 gpfs]# ./installGPFS -i -g
"GPFS" is not installed.
DBI1070I Program installGPFS completed successfully.
# vi /tmp/gpfsprofile
wtydb21:quorum
wtydb22:quorum
wtydb23:quorum
[root@pure1, /]
mmcrcluster -n /tmp/gpfsprofile -p wtydb21-s wtydb22 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp
# mmcrcluster -n /home/gpfs/gpfs.allnodes-p node1 -s node2 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp
# cd /opt/ibm/db2/V10.1/bin
# ./db2cluster -cfs -add -license
The license for the shared file system
./db2cluster -cfs -start -all
./db2cluster -cfs -create -filesystem db2fs-disk /dev/dm-3 -mount /db2fs
1， 安裝gpfs軟件（三台主機上運行）
本次從db2 purescale安裝介質中直接安裝，其它安裝方法見參考文獻
rpm –qa gpfs驗證軟件是否已經正確安裝
2， cd /usr/lpp/mmfs/src && make Autoconfig && make World && make InstallImages
3， 修改PATH路徑（三台主機上運行）
在3 台主機上修改HOME目錄下的。bash_profile文件，在文件末尾增加
export PATH=$PATH:/usr/lpp/mmfs/bin
3， 新建目錄,用做GPFS文件系統（三台主機上運行）
mkdir /db2fs
4， 創建gpfs 群集配置文件
[root@wtydb21 tmp]# vi /tmp/gpfsprofile
wtydb21:quorum-manager
wtydb22:quorum-manager
wtydb23:quorum-manager
5，創建集群，注意指定ssh方式
[root@wtydb21 pam.d]#
[root@wtydb21 gpfs]# mmcrcluster -N /tmp/gpfsprofile -p wtydb21 -s wtydb22 -C gpfs_cluster -r /usr/bin/ssh -R /usr/bin/scp
Sat Apr 6 12:17:35 CST 2013: mmcrcluster: Processing node wtydb21
Sat Apr 6 12:17:35 CST 2013: mmcrcluster: Processing node wtydb22
Sat Apr 6 12:17:38 CST 2013: mmcrcluster: Processing node wtydb23
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
Use the mmchlicense command to designate licenses as needed.
mmcrcluster: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
mmcrcluster 命令其中參數含義
-C bgbcrun 設定集群名稱
-U bgbc 定義域名
-N /tmp/gpfs/nodefile 指定節點文件名
-p NSD1 指定主NSD 服務器為 NSD1
-s NSD1 指定備NSD 服務器為 NSD1
6，接受許可協議
[root@wtydb21 pam.d]# mmchlicense server --accept -N wtydb21,wtydb22,wtydb23
7，確認創建集群狀況
[root@wtydb21 ~]# mmlscluster
[root@wtydb21 gpfs]# mmlscluster
GPFS cluster information
========================
GPFS cluster name: gpfs_cluster.wtydb21
GPFS cluster id: 12146727015547904479
GPFS UID domain: gpfs_cluster.wtydb21
Remote shell command: /usr/bin/ssh
Remote file copy command: /usr/bin/scp
GPFS cluster configuration servers:
-----------------------------------
Primary server: wtydb21
Secondary server: wtydb22
Node Daemon node name IP address Admin node name Designation
-------------------------------------------------------------------
1 wtydb21 192.168.2.101 wtydb21 quorum-manager
2 wtydb22 192.168.2.102 wtydb22 quorum-manager
3 wtydb23 192.168.2.103 wtydb23 quorum-manager
8，生成nsd 盤，使用/dev/dm-3
[root@wtydb21 etc]# vi /tmp/nsdprofile
dm-3:::dataAndMetadata::
[root@wtydb21 gpfs]# mmcrnsd -F /tmp/nsdprofile
mmcrnsd: Processing disk dm-3
mmcrnsd: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
此時系統自動修改/tmp/gpfsnsdprofile 文件內容如下：
[root@wtydb21 ~]# cat /tmp/nsdprofile
# dm-4:::dataAndMetadata::
gpfs1nsd:::dataAndMetadata:-1::system
9,啟動集群
[root@wtydb21 /]# mmstartup -a
.
[root@wtydb21 src]# mmgetstate -a -L
Node number Node name Quorum Nodes up Total nodes GPFS state Remarks
------------------------------------------------------------------------------------
1 wtydb21 2 2 3 active quorum node
2 wtydb22 2 2 3 active quorum node
3 wtydb23 2 2 3 active
10,創建gpfs 文件系統
[root@wtydb21 src]# mmcrfs /db2fs gpfs_lv -F /tmp/nsdprofile -A yes -n 30 -v no
The following disks of gpfs_lv will be formatted on node wtydb21:
gpfs1nsd: size 100829184 KB
Formatting file system ...
Disks up to size 848 GB can be added to storage pool system.
Creating Inode File
Creating Allocation Maps
Creating Log Files
Clearing Inode Allocation Map
Clearing Block Allocation Map
Formatting Allocation Map for storage pool system
Completed creation of file system /dev/gpfs_lv.
mmcrfs: Propagating the cluster configuration data to all
affected nodes. This is an asynchronous process.
參數含義如下：
/datafs 文件系統 mount 點名
gpfs_lv 指定文件系統 lv 名
-F 指定 NSD 的文件名
-A 自動 mount 選項為 yes
-B 塊大小為64K
-n 掛載文件系統的節點估計數30 個
-v 校驗建立磁盤是否已有文件系統 為否
11，掛載文件系統
[root@wtydb21 /]# mount /db2fs
[root@wtydb21 /]# df
12，設置gpfs 開機自啟
[root@wtydb21 /]# mmchconfig autoload=yes
13，查詢GPFS 配置信息
[root@wtydb21 share]# mmlsconfig
[root@wtydb21 share]# mmgetstate -a
db2 purescale要求必須符合以下參數配置
[root@wtydb21 src]# /opt/ibm/db2/V10.1/bin/db2cluster -cfs -verify -configuration
The shared file system cluster option 'adminMode' has not been set to the optimal value of 'allToAll'. See the DB2 Information Center for more details.
The shared file system cluster option 'maxFilesToCache' has been set too low. The value should be at least '10000'. See the DB2 Information Center for more details.
The shared file system cluster option 'usePersistentReserve' has not been set to the optimal value of 'yes'. See the DB2 Information Center for more details.
The shared file system cluster option 'verifyGpfsReady' has not been set to the optimal value of 'yes'. See the DB2 Information Center for more details.
The shared file system cluster option 'failureDetectionTime' could not be verified. Re-issue this command after the cluster manager peer domain has been created and is online.
A diagnostic log has been saved to '/tmp/ibm.db2.cluster.toBL0B'.修改過程如下
mmchconfig adminMode allToAll
mmchconfig maxFilesToCache=20000
mmchconfig pagepool=1024M
mmchconfig usePersistentReserve=yes
mmchconfig verifyGpfsReady=yes
mmshutdown -a
db2cluster -cfs -stop -all
mmchconfig failureDetectionTime=35
db2cluster -cfs -start -all
mmstartup -a
db2cluster -cfs -verify -configuration
db2cluster -cfs -verify -filesystem mygpfs1
實例的所有者db2sdin1對文件系統具有訪問權限
1.fuser –kcu /ods/gpfs
2. unmount /ods/gpfs #在所有節點上執行
3. mmdelfs gpfslv,
4. mmlsfs gpfslv #檢查結果
5. mmdelnsd –F /ods/gpfsnsdprofile
6. mmshutdown –a
7. mmdelnode -a
8. mmdelnode –f #最後清除集群
Optional. For DB2 managed GPFSinstallations, verify the remote shell and
remote file copy settings default todb2locssh and db2scp. For example:
/usr/lpp/mmfs/bin/mmlscluster
Remote shell command:/var/db2/db2ssh/db2locssh
Remote file copy command:/var/db2/db2ssh/db2scp
1，卸載所有gpfs文件系統，mmumount all -a
2，刪除文件系統，mmdelfs gpfs_lv
3，刪除nsd節點，mmdelnsd -F /tmp/nsdprofile
4，停止gpfs系統，mmshutdown-a
5，卸載gpfs軟件，先查看已經安裝的軟件rpm -qa|grep gpfs
rpm -e gpfs.gpl
rpm -e gpfs.msg.en_us
rpm -e gpfs.base
rpm -e gpfs.docs
6，刪除目錄 /var/mmfs和/usr/lpp/mmfs
7，刪除/var/adm/ras目錄下以mm開頭的文件
8，刪除/tmp/mmfs（如果存在）目錄和其內容
gpfs log file path /var/adm/ras
The verifyGpfsReady=yesconfiguration attribute is set, but the /var/mmfs/etc/gpfsreadyscript could not be executed.
Make sure /var/mmfs/etc/gpfsreadyexists and is executable, or disable the verifyGpfsReadyoption via mmchconfig verifyGpfsReady=no.
ERROR: DBI20093E The DB2 installer failed to set up thedb2sshid because the
GPFS file system is a user-managed filesystem.
參考文獻
[root@wtydb21 ~]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.101 wtydb21-ib0
192.168.3.101 wtydb21-ib3
192.168.0.102 wtydb22-ib0
192.168.3.102 wtydb22-ib3
192.168.0.103 wtydb23-ib0
192.168.3.103 wtydb23-ib3
192.168.2.101 wtydb21
192.168.2.102 wtydb22
192.168.2.103 wtydb23
備注：IP和主機名之間用空格，一定不要用tab鍵，否則報錯。lookback不要匹配主機名
第一步：
#hostnamewtydb21
第二步：
修改/etc/sysconfig/network中的hostname
第三步：
修改/etc/hosts文件
eth0和eth4做網卡綁定，主備模式
使用ethtool確認網絡連接狀態
1，cd/etc/sysconfig/network-scripts/ 修改相關配置文件
新建 ifcfg-bond0
DEVICE=bond0
BOOTPROTO=static
IPADDR=192.168.2.101
NETMASK=255.255.255.0
GATEWAY=192.168.2.254
ONBOOT=YES
TYPE=Ethernet
USERCTL=no
BONDING_OPTS="miimon=100 mode=1"
修改ifcfg-eth0內容：
DEVICE=eth0
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
修改ifcfg-eth4內容
DEVICE=eth4
BOOTPROTO=none
ONBOOT=yes
USERCTL=no
MASTER=bond0
SLAVE=yes
2，vi/etc/modprobe.conf /etc/modprobe.d/bonding.conf
aliasbond0 bonding
3，重啟網絡配置
servicenetwork
4， 關閉networkmanger(rhel6.3版本需要，之前rhel5配置時無此步驟)
chkconfig NetworkManager off;
/etc/init.d/NetworkManager stop;
chkconfig network on;
/etc/init.d/network restart
ifconfig bond1 up
1，chkconfig iptablesoff
service iptables stop
2，selinux
vi /etc/selinux/config
#SELINUX=enforcing
SELINUX=disabled
將文件中的SELINUX="" 為 disabled ，然後重啟。
如果不想重啟系統，使用命令setenforce 0
With the Mellanox SM HA software, thesystem manager can enter and modify
all IB subnet configurations for the subnetmanagers from a single location.
Assign a virtual IP address (VIP) to themanagement port of the switch to
manage the high availability domain. Thesystem manager must configure all
the switches in a Mellanox SM HAenvironment to join the same IB subnet, and
assign the subnet a name. After joining thesubnet, the subnet managers are
synchronized and you must select one as themaster subnet manager, and the
others become standby subnet managers.
infiniband軟件
Install the packages required and reboot:
rhel 6.3版本
[root@serv]# yum groupinstall "Infiniband Support"
[root@serv]# yum install infiniband-diags perftest qperf opensm
[root@serv]# chkconfig rdma on
[root@serv]# chkconfig opensm on
[root@serv]# shutdown -r now
同普通以太網卡配置，例外db2數據庫使用RDMA協議，而不是TCP|IP協議
[root@wtydb22 network-scripts]# cat ifcfg-ib0
DEVICE="ib0"
BOOTPROTO="static"
IPADDR=192.168.0.102
#NETMASK=255.255.255.0
PREFIX=24
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="InfiniBand"
1，log_mtts_per_seg參數
如果 CF 或成員所在的主機有超過 64 GB 的內存，那麼 Mellanox HCA 驅動程序（mlx4_core）模塊的參數 log_mtts_per_seg 必須從 3（默認）增加到 7，以便有更大的內存注冊。
要增加大小，以根用戶身份發出以下命令：
在 SUSE 上：
echo "optionsmlx4_core log_mtts_per_seg=7" >>
/etc/modprobe.conf.local
在 RHEL 上：
echo "optionsmlx4_core log_mtts_per_seg=7" >> /etc/modprobe.conf
RHEL 6 上需要運行如下命令
echo "options mlx4_corelog_mtts_per_seg=7" >> /etc/modprobe.d/modprobe.conf
要讓該更改生效，您必須重啟服務器。要檢查您的更改在模塊上是否有效，輸入：
<host-name>/sys/module/mlx4_core/parameters# cat /sys/module/mlx4_core/parameters/log_mtts_per_seg
2，/etc/rdma/dat.conf配置文件
On RHEL 6.1, the DAT configuration file islocated in /etc/rdma/dat.conf
and it is updated by the group installationof the "InfiniBand Support"
package
系統默認配置好了
[root@wtydb21 Packages]# cat/etc/rdma/dat.conf
ofa-v2-mlx4_0-1 u2.0 nonthreadsafe defaultlibdaploscm.so.2 dapl.2.0 "mlx4_0 1" ""
ofa-v2-mlx4_0-2 u2.0 nonthreadsafe defaultlibdaploscm.so.2 dapl.2.0 "mlx4_0 2" ""
ofa-v2-ib0 u2.0 nonthreadsafe defaultlibdaplofa.so.2 dapl.2.0 "ib0 0" ""
……
網絡配置
2）檢查infiniband交換機的配置
a,管理軟件需要最低版本
image-PPC_M405EXEFM_1.1.2500.img
b,啟動子網管理SM，管理口配置VIP，互聯口配置IP地址
c，配置故障轉移
d,兩個交換機之間需要2根以上的互連線，建議CF和member連接至交換機的總port數除以2
使用yum install命令在三台主機同時安裝即可
yum install libstdc++
yum install pam-1.1.1
yum install pam_krb5-2.3.11
yum install pam-devel
yum install pam_pkcs11
所需軟件包，詳見官方安裝文檔說明。
libstdc++-4.4.5-6.el6.i686,
pam-1.1.1-8.el6.i686,
pam_krb5-2.3.11-6.el6.i686,
pam-devel-1.1.1-8.el6.i686,
pam_pkcs11-0.6.2-11.1.el6.i686,
pam_ldap-185-8.el6.i686
下列包rhel 6.1 均已經默認安裝
For InfiniBand network
type (both 32- bit and 64-bit libraries unless specified) :
libibcm
dapl (64-bit libraries only)
ibsim (64-bit libraries only)
ibutils (64-bit libraries only)
libibverbs
librdmacm
libcxgb3
libibmad
libibumad
libipathverbs (64-bit libraries only)
libmlx4
libmthca
libnes (64-bit libraries only)
libmlx4
rdma (no architecture)
ntp-4.2.4p8-2.el6.x86_64/ntpdate-4.2.4p8-2.el6.x86_64
libstdc++-4.4.5-6.el6.x86_64
libstdc++-4.4.5-6.el6.i686
glibc-2.12-1.25.el6.x86_64
glibc-2.12-1.25.el6.i686
gcc-c++-4.4.5-6.el6.x86_64
gcc-4.4.5-6.el6.x86_64
kernel-2.6.32-131.0.15.el6.x86_64
kernel-devel-2.6.32-131.0.15.el6.x86_64
kernel-headers-2.6.32-131.0.15.el6.x86_64
kernel-firmware-2.6.32-131.0.15.el6.noarch
ntp-4.2.4p8-2.el6.x86_64
ntpdate-4.2.4p8-2.el6.x86_64
sg3_utils-1.28-3.el6.x86_64
sg3_utils-libs-1.28-3.el6.x86_64
binutils-2.20.51.0.2-5.20.el6.x86_64
binutils-devel-2.20.51.0.2-5.20.el6.x86_64
openssh-5.3p1-52.el6.x86_64
cpp-4.4.5-6.el6.x86_64
ksh-20100621-16.el6.x86_64
安裝示例說明
[root@wtydb23 ~]# yum install compat-libstdc++-33-3.2.3-69.el6.x86_64.rpm
另外，GPFS軟件還需要I686等軟件包需要安裝
v For multiple communication adapter portson InfiniBand network and single or multiple communication adapter port at CFson 10GE network,
the minimum support level is RHEL 6.1.
i686 which is 32-bit packages might not getinstalled by default when installing x86_64 server. Make sure that all the32-bit dependencies are
explicitly installed. For example:
libstdc++-4.4.5-6.el6.i686,
pam-1.1.1-8.el6.i686,
pam_krb5-2.3.11-6.el6.i686,
pam-devel-1.1.1-8.el6.i686,
pam_pkcs11-0.6.2-11.1.el6.i686,
pam_ldap-185-8.el6.i686
Alternatively, run the yum command aftercreating a source from local DVD or after registering to RHN:
yum install *.i686
Ensure the following 32-bit RSCT packagesare installed:
– libibcm.i686
– libibverbs-rocee.i686
– librdmacm.i686
– libcxgb3.i686
– libibmad.i686
– libibumad.i686
– libmlx4-rocee.i686
– libmthca.i686·
DB2 pureScale Feature requireslibstdc++.so.6. Verify that the files exist with
the following commands:
ls /usr/lib/libstdc++.so.6*
ls /usr/lib64/libstdc++.so.6*
[root@wtydb21 tsamp]# cat /tmp/prereqSAM.1.log
prereqSAM: >>> Prerequisite on wtydb21 check - log started : Fri Apr 12 14:22:16 CST 2013
prereqSAM: OPTIONS = ''
prereqSAM: OPT_SILENT = 0
prereqSAM: OPT_NOLICCHECK = 0
prereqSAM: Detected operating system Linux
prereqSAM: Detected architecture i386x
prereqSAM: Detected distribution RH
prereqSAM: Supported operating system versions RH Linux i386x - 5.0 6.0
prereqSAM: Detected operating system version
Red Hat Enterprise Linux Server release 6.1 (Santiago)
prereqSAM: rpm package and version installed 'ksh'
prereqSAM: Using default prerequisite checking on the following rpm package 'perl'
prereqSAM: rpm package and version installed 'perl'
prereqSAM: Using default prerequisite checking on the following rpm package 'libstdc++' 'i686'
prereqSAM: rpm package and version installed 'libstdc++' 'i686'
prereqSAM: Using default prerequisite checking on the following rpm package 'libstdc++' 'x86_64'
prereqSAM: rpm package and version installed 'libstdc++' 'x86_64'
prereqSAM: Using default prerequisite checking on the following rpm package 'compat-libstdc++-33' 'x86_64'
prereqSAM: Error: The following rpm package is not installed 'compat-libstdc++-33' 'x86_64'
prereqSAM: Using default prerequisite checking on the following rpm package 'pam' 'i686'
prereqSAM: rpm package and version installed 'pam' 'i686'
1 missing package: compat-libstdc++-33 (x86_64)
prereqSAM: Error: Prerequisite checking for the ITSAMP installation failed Linux i386x RH
Red Hat Enterprise Linux Server release 6.1 (Santiago)
prereqSAM: Most severe error code returned 21
prereqSAM: One or more prerequisite packages were not found installed
prereqSAM: <<< Prerequisite on wtydb21 check - log ended : Fri Apr 12 14:22:19 CST 2013
db2 10.1 可以自動調整內核參數為最優值，但是個人設置一下還是沒啥壞處吧
able 1. Enforced minimum settings for Linuxinterprocess communication kernel parametersIPC kernel parameter Enforced minimum setting
kernel.shmmni (SHMMNI) 256 * <size of RAM in GB>
kernel.shmmax (SHMMAX) <size of RAM in bytes>
kernel.shmall (SHMALL) 2 * <size of RAM in the defaultsystem page size>
共享內存段的設置建議
Beginning with the first section on SharedMemory Limits, the SHMMAX limit is the maximum size of a shared memory segmenton a Linux system. The SHMALL limit is the maximum allocation of shared memorypages on a system.
It is recommended to set the SHMMAX value to be equal to the amount ofphysical memory on your system. However, the minimum required on x86 systems is268435456 (256 MB) and for 64-bit systems, it is 1073741824 (1 GB).
kernel.sem (SEMMNI) 256 * <size of RAM in GB>
kernel.sem (SEMMSL) 250
kernel.sem (SEMMNS) 256 000
kernel.sem (SEMOPM) 32
信號量相關的內核設置
kernel.msgmni (MSGMNI) 1 024 * <size of RAM in GB>
kernel.msgmax (MSGMAX) 65 536
kernel.msgmnb (MSGMNB) 65 536
消息相關的內核設置
設置的小例子
#Example for a computer with 128GB of RAM，100GB分配給數據庫使用
kernel.shmmni=4096
kernel.shmmax=107374182400
//單個共享內存段的最大大小，建議設置為最大物理內存大小。本例設置為100GB；大小為100*1024*1024*1024=107374182400
kernel.shmall=8388608
//系統中能夠分配的共享內存頁的大小，頁的大小是4KB，總大小設置為100GB，應該是100*1024*1024*1024/4/1024=26214400
#kernel.sem=<SEMMSL> <SEMMNS><SEMOPM> <SEMMNI>
kernel.sem=250 1024000 32 4096
kernel.msgmni=16384
kernel.msgmax=65536
kernel.msgmnb=65536
參數設置說明
具體設計的例子
vi /etc/sysctl.conf
2，網絡相關內核參數
注釋掉rp_filter=1的行，
# net.ipv4.conf.all.rp_filter = 1
# changes for DB2 pureScale
然後添加
net.ipv4.conf.all.rp_filter = 2
net.ipv4.conf.all.arp_ignore = 1
我的服務器設置內存128GB 100GB分配給數據庫使用
在所有的機器(pcf、scf、mb1、mb2)上，修改文件：
/etc/modprobe.d/blacklist.conf
添加如下內容：
# RSCT hatsd, add for pureScale
blacklist iTCO_wdt
blacklist iTCO_vendor_support
一般至少需要四塊物理盤，
tirebreker disk 25M
實例共享文件系統 10G
數據文件系統
日志文件系統
1，安裝device-mapper-multipath rpm。
當在 DM-多路徑中添加新設備時，這些新設備會位於 /dev 目錄的三個不同位置：dev/mapper/mpathn、/dev/mpath/mpathn 和 /dev/dm-n。 /dev/mapper 中的設備是在引導過程中生成的。可使用這些設備訪問多路徑設備，例如在生成邏輯卷時。提供 /dev/mpath 中的設備是為了方便，這樣可在一個目錄中看到所有多路徑設備。這些設備是由 udev 設備管理器生成的，且在系統需要訪問它們時不一定能啟動。請不要使用這些設備生成邏輯卷或者文件系統。
所有 /dev/dm-n 格式的設備都只能是作為內部使用，且應該永遠不要使用。
（備注 db2 purescale不能使用/dev/mapper/mpath設備，只能使用dm-設備名稱）
yuminstall device-mapper-multipath*
2，打開multipathd
[root@wtydb21 ~]# chkconfig --list multipathd
multipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
[root@wtydb21 ~]# chkconfig multipathd on
[root@wtydb21 ~]# chkconfig --list multipathd
multipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
3，修改配置文件，保持多路徑設備名稱保持一致（安裝db2 purescale 本步驟可省略）
一般可以修改/etc/multipath.conf文件，但為了保持集群（oraclerac），也可以修改/var/lib/multipath/bindings，並把文件復制到集群中的其它節點
4，查看配置情況
modprobe dm-multipath
service multipathd start
multipath -v2
multipath -v2 命令輸出多路徑設備的路徑來顯示哪些設備是多路徑設備。如果沒有輸出任何結果，請確定正確調試了所有 SAN 連接並使系統多路徑。
執行以下命令確定在引導時啟動了多路徑守護進程
mpathconf --enable --with_multipathd y
chkconfig multipathd on
chkconfig --list multipathd
$ service multipathd stop
$ service multipathd start
$ multipathd –F
$ multipathd –v2
參考文獻
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/DM_Multipath/mpio_setup.html
可手工創建，也可在安裝db purescale軟件時配置。
兩個用戶分別屬於不同的組，集群中不同的主機上需要相同的UID和GID，相同的home目錄
You should use two different users with twodifferent groups. Each of the two users should have the same UID, GID, groupname, and the home directory on all the hosts.
Required user User name Groupname
Instance owner db2sdin1 db2iadm1
Fenced user db2sdfe1 db2fadm1
groupadd -g 999 db2iadm1
groupadd -g 998 db2fadm1
useradd -g db2iadm1 -u 1004-m db2sdin1
useradd -g db2fadm1 -u 1003 -m db2sdfe1
passwd db2sdin1
passwd db2fenc1
vi /etc/ntp.conf
server 192.168.0.81 # local clock
fudge 192.168.0.81 stratum 10
restrict 192.168.0.81 mask 255.255.255.255 nomodify notrap noquery
service ntp start
chkconfig ntp on
chkconfig --list ntpd
DB2使用自己的watch dog，因此需要關閉機器帶的intelwatch dog
In some installations, if Intel TCOWatchDog Timer Driver modules are loaded by default, they should beblacklisted, so that they do not start automatically or conflict with RSCT. Toblacklist the modules, edit the following files:
To verify if the modules are loaded
lsmod | grep -i iTCO_wdt; lsmod | grep -iiTCO_vendor_support
Edit the configuration files:
On RHEL 5.x and RHEL 6.1, edit file /etc/modprobe.d/blacklist.conf:
# RSCT hatsd
blacklist iTCO_wdt
blacklist iTCO_vendor_support
使用db2prereqcheck命令進行安裝前的檢查，一定確保檢查完成通過。其實即使了在安裝tivoli，gpfs等還需要一些依賴包，也可能出現安裝報錯
$ ./db2prereqcheck -i
檢查是否滿足具體安裝版本需求，使用-v命令
$ ./db2prereqcheck -v 10.1.0.0
To generate a report file containingvalidation information and output from the db2prereqcheck command (including asummary section listing all failing tests), use the -o <filename>parameter. Specify the full report filename path to generate the report inanother directory. The prerequisites for installing the DB2 product will onlybe displayed on the command screen without the -o <filename> parameter.
$ ./db2prereqcheck -i -o report.rpt
檢查IB網絡是否滿足要求
$ ./db2prereqcheck -i -p -t <networkconfiguration type>
db2prereqcheck –I –p –t MULT_IB_PORT_CLUSTER
Purescale有個bug，不在認證中的陣列會出現不能使用/dev/dm-硬盤安裝的情況，可以先手工安裝gpfs，使用dm-盤，在使用db2cluster接管gpfs
解決不能直接使用/dev/dm- 進行安裝的問題
[root@wtydb21 db2sdin1]#/opt/ibm/db2/V10.1/bin/db2cluster -cfs -verify -configuration
/opt/ibm/db2/V10.1/instance/db2cluster_prepare-cfs_takeover
take over a user-managed GPFS cluster:
1. Log on as root on any machine in your cluster.
2. Run the db2cluster_prepare command with the following parameters:
db2cluster_prepare -cfs_takeover
3. Check the error return code using the echo $? command. If there are errors, resolve the errors and rerun the command as specified in Step 2.
4. To verify that you've properly added the record, run the following command:
[root@wtydb21 tmp]# db2greg -dump
S,GPFS,3.5.0.4,/usr/lpp/mmfs,-,-,0,0,-,1365173996,0
S,DB2,10.1.0.2,/opt/ibm/db2/V10.1,,,2,0,,1365174117,0
V,DB2GPRF,DB2SYSTEM,wtydb21,/opt/ibm/db2/V10.1,
V,GPFS_CLUSTER,NAME,gpfs_cluster.wtydb21,-,DB2_MANAGED
[root@wtydb21 src]#/usr/lpp/mmfs/bin/tsprinquiry dm-3
IBM :2810XIV :0000
vi /var/mmfs/etc/prcapdevices
IBM:2810XIV:0000
/usr/lpp/mmfs/bin/mmchconfigusePersistentReserve=yes
echo "options mlx4_corelog_mtts_per_seg=7" >>/etc/modprobe.conf
僅安裝purescale Feature，不安裝db2 實例
本文通過db2_install手工實現安裝
需要在兩台主機分別進行安裝
[root@wtydb21 server]# ./db2_install -t /tmp/db2_install.trc -l db2_install.log
DBI1324W Support of the db2_install command is deprecated. For
more information, see the DB2 Information Center.
Default directory for installation of products - /opt/ibm/db2/V10.1
***********************************************************
Install into default directory (/opt/ibm/db2/V10.1) ? [yes/no]
yes
Specify one of the following keywords to install DB2 products.
AESE
ESE
CONSV
WSE
EXP
CLIENT
RTCL
Enter "help" to redisplay product names.
Enter "quit" to exit.
***********************************************************
ESE
***********************************************************
Do you want to install the DB2 pureScale Feature? [yes/no]
yes
DB2 installation is being initialized.
Total number of tasks to be performed: 50
Total estimated time for all tasks to be performed: 1910 second(s)
Task #1 start
Description: Checking license agreement acceptance
Estimated time 1 second(s)
Task #1 end
Task #2 start
Description: Base Client Support for installation with root privileges
Estimated time 3 second(s)
Task #2 end
Task #54 start
Description: Updating global profile registry
Estimated time 3 second(s)
Task #54 end
The execution completed successfully.
For more information see the DB2 installation log at
"/data/esefp2/server/db2_install.log"
.…….
安裝前檢查，沒有輸出即正常
[root@wtydb21 ~]# preprpnode wtydb21 wtydb22wtydb23
開始安裝
/opt/ibm/db2/V10.1/instance/db2icrt -d -cf wtydb21 -cfnet wtydb21-ib0 -m wtydb22 -mnet wtydb22-ib0 -instance_shared_dev /dev/dm-15 -tbdev /dev/dm-2 -u db2sdfe1 db2sdin1
--添加第二個CF
/opt/ibm/db2/V10.1/instance/db2iupdt -d -add -cf wtydb22 -cfnet wtydb22-ib0 db2sdin1
--添加member
/opt/ibm/db2/V10.1/instance/db2iupdt -d -add -m wtydb22 -mnet wtydb22-ib0 db2sdin1
/opt/ibm/db2/V10.1/instance/db2iupdt -d -add -m wtydb23 -mnet wtydb23-ib0 db2sdin1
本次直接使用fp2最新補丁集安裝。
更新前檢查db2nodes.cfg文件
[root@wtydb23 msdata]# cat /home/db2sdin1/sqllib/db2nodes.cfg
0 wtydb22 0 wtydb22-ib0 - MEMBER
1 wtydb23 0 wtydb23-ib0 - MEMBER
128 wtydb23 0 wtydb23-ib0 - CF
129 wtydb22 0 wtydb22-ib0 - CF
更新IB網絡配置
db2iupdt -update -cf wtydb23 -cfnet wtydb23-ib0,wtydb23-ib3 db2sdin1
db2iupdt -update -m wtydb23 -mnet wtydb23-ib0, wtydb23-ib3 db2sdin1
安裝過程中如果出現問題，可以使用如下方法刪除相關軟件
1, 備份數據庫
2，使用db2stop 命令停止所有實例
3，使用db2idrop -g db2sdin1刪除所有主機上的數據庫實例.另外，db2idrop -g命令會保留最後主機上的gpfs數據庫，需要手工清除gpfs。
4，使用db2_deinstall -a命令刪除db2 purescale軟件
1. /dev/mapper/mapth*設備不能使用
/dev/mapper/mapth*設備本質上是鏈接文件，GPFS不能使用，應該使用多路徑軟件映射後的/dev/mapth設備
2. 直接/dev/dm-*設備報錯
該問題是軟件的bug，主要是由於XIV存儲不在軟件支持列表導致，可以通過手工創建GPFS，然後db2cluster接管的方式或者改由
初始狀態，系統正常
[db2sdin1@wtydb22 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME
-- ---- ----- --------- ------------ ----- ---------------- ------------ -------
0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0
1 MEMBER STARTED wtydb21 wtydb21 NO 0 0 wtydb21-ib0
2 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0
128 CF PRIMARY wtydb21 wtydb21 NO - 0 wtydb21-ib0
129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT
-------- ----- ---------------- -----
wtydb23 ACTIVE NO NO
wtydb21 ACTIVE NO NO
wtydb22 ACTIVE NO NO
數據庫能正常連接
[db2sdin1@wtydb22 ~]$ db2 connect to msdb
Database Connection Information
Database server = DB2/LINUXX8664 10.1.2
SQL authorization ID = DB2SDIN1
Local database alias = MSDB
1，斷掉wtydb21主機的網卡
[root@wtydb21 ~]# ifdown bond0
2，此時，從其它主機觀察狀態，正常
[db2sdin1@wtydb22 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME
-- ---- ----- --------- ------------ ----- ---------------- ------------ -------
0 MEMBER STARTED wtydb22 wtydb22 YES 0 0 wtydb22-ib0
1 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb23 YES 0 1 wtydb23-ib0
2 MEMBER STARTED wtydb23 wtydb23 YES 0 0 wtydb23-ib0
128 CF ERROR wtydb21 wtydb21 YES - 0 wtydb21-ib0
129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT
-------- ----- ---------------- -----
wtydb23 ACTIVE NO NO
wtydb21 ACTIVE YES YES
wtydb22 ACTIVE NO NO
There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'.
[db2sdin1@wtydb22 ~]$ db2 connect to msdb
Database Connection Information
Database server = DB2/LINUXX8664 10.1.2
SQL authorization ID = DB2SDIN1
Local database alias = MSDB
3，網卡恢復後，系統能自動回復
一台服務器IB網絡斷
A,主機1 停止bond0網卡
[root@wtydb21 ~]# ifdown bond0
B, 主機2 仍能成功連接數據庫
[root@wtydb22 ~]# db2 connect to msdb
Database Connection Information
Database server = DB2/LINUXX8664 10.1.2
SQL authorization ID = ROOT
Local database alias = MSDB
C, 此時集群狀態，wtydb21處於waiting_for_fallback狀態
[db2sdin1@wtydb23 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME
-- ---- ----- --------- ------------ ----- ---------------- ------------ -------
0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0
1 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0
2 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb22 NO 0 1 wtydb22-ib0
128 CF PRIMARY wtydb23 wtydb23 NO - 0 wtydb23-ib0
129 CF PEER wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT
-------- ----- ---------------- -----
wtydb21 ACTIVE NO YES
wtydb23 ACTIVE NO NO
wtydb22 ACTIVE NO NO
There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'.
wtydb22主機上顯示，CF正常切換，數據庫仍能提供服務，測試通過。重新起來後，恢復正常
[db2sdin1@wtydb22 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME
-- ---- ----- --------- ------------ ----- ---------------- ------------ -------
0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0
1 MEMBER WAITING_FOR_FAILBACK wtydb23 wtydb21 NO 0 1 wtydb21-ib0
2 MEMBER STARTED wtydb21 wtydb21 NO 0 0 wtydb21-ib0
128 CF ERROR wtydb23 wtydb23 YES - 0 wtydb23-ib0
129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT
-------- ----- ---------------- -----
wtydb21 ACTIVE NO NO
wtydb23 INACTIVE NO YES
wtydb22 ACTIVE NO NO
There is currently an alert for a member, CF, or host in the data-sharing instance. For more information on the alert, its impact, and how to clear it, run the following command: 'db2cluster -cm -list -alert'
wtydb22成為提供服務的主機
[db2sdin1@wtydb22 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME
-- ---- ----- --------- ------------ ----- ---------------- ------------ -------
0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0
1 MEMBER WAITING_FOR_FAILBACK wtydb23 wtydb22 YES 0 2 wtydb22-ib0
2 MEMBER WAITING_FOR_FAILBACK wtydb21 wtydb22 YES 0 1 wtydb22-ib0
128 CF ERROR wtydb23 wtydb23 YES - 0 wtydb23-ib0
129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT
-------- ----- ---------------- -----
wtydb21 ACTIVE NO YES
wtydb23 ACTIVE NO YES
wtydb22 ACTIVE NO NO
switch-1 起來以後，數據庫自動恢復
[db2sdin1@wtydb23 ~]$ db2instance -list
ID TYPE STATE HOME_HOST CURRENT_HOST ALERT PARTITION_NUMBER LOGICAL_PORT NETNAME
-- ---- ----- --------- ------------ ----- ---------------- ------------ -------
0 MEMBER STARTED wtydb22 wtydb22 NO 0 0 wtydb22-ib0
1 MEMBER STARTED wtydb23 wtydb23 NO 0 0 wtydb23-ib0
128 CF PEER wtydb23 wtydb23 NO - 0 wtydb23-ib0
129 CF PRIMARY wtydb22 wtydb22 NO - 0 wtydb22-ib0
HOSTNAME STATE INSTANCE_STOPPED ALERT
-------- ----- ---------------- -----
wtydb23 ACTIVE NO NO
wtydb22 ACTIVE NO NO
前期環境准備，同前面
目標通過檢驗：/software/ese10.1.2/server/db2prereqcheck -p -tMULTI_IB_PORT_CLUSTER -v 10.1.0.2
/opt/ibm/db2/V10.1.01/instance/db2iupdt -d -add -m wtydb23 -mnet wtydb23-ib0 db2sdin1