Abbas | Bassu

Just came across this snap in my cell’s pic gallery which I took at Zia Intl. Airport. Funny!

Increasing the rebuild speed

Sometimes when you’re quite lazy or bored and don’t like the noticeable amount of free resources on your server, you may like to increase the raid building and resyncing process speed.

# echo 250000 > /proc/sys/dev/raid/speed_limit_max
# echo 250000 > /proc/sys/dev/raid/speed_limit_min

And this would spin up the hards faster to their maximum extent saving almost half of the time!
Defaults were:

 # cat /proc/sys/dev/raid/speed_limit_max
200000
# cat /proc/sys/dev/raid/speed_limit_min
1000

And looks like my hards have 100M/s speed.

#hddtemp /dev/sda
/dev/sda: ST3750330AS: 31°C
# hdparm -t /dev/sda
/dev/sda:
 Timing buffered disk reads:  280 MB in  3.01 seconds =  93.09 MB/sec
 

Checking and repairing

The larger the hard drive is the greater the probability of having its blocks corrupted. Its better to check for consistency against any bad blocks or md superblocks.

# cat /sys/block/md0/md/sync_action
idle
# echo check > /sys/block/md0/md/sync_action

And if there are any, repair them.

# echo repair > /sys/block/md0/md/sync_action

Setting up RAID alerts to send emails about any errors

Red Hat family distros come with built in service daemon mdmonitor which runs ‘mdadm –monitor’ in a daemon mode.

# mdadm --detail --scan >> /etc/mdadm.conf
# echo "MAILADDR some-email-address" >> /etc/mdadm.conf
# chkconfig mdmonitor on
# service mdmonitor restart

Remember to add “DEVICE partitions” or partitions being components of an md array to top of file /etc/mdadm.conf.

Monitoring all RAID events and changing default email template

Last edited on: Tue Mar  9 11:41:38 PST 2010
Added the two new tweaks.

I’m sometimes pretty sick of getting emails from ‘root’ reporting crons, updates, errors and rest of things I would usually use daily in any Linux machine. My eyes just get sore when I see email from root in all of my email clients specially Gmail. Perhaps that’s because I’m fed up of seeing root everywhere and that’s why I’ve an alias for root to forward all these rootish emails to one of my email addresses. But that’s just not enough as I actually want to see a humane human name as a sender just instead of root :>

To add to it, this just becomes more anguish when common, rather I would say major, Linux applications suck in changing the default sender address off from ‘root’ (although some apps like ‘exim -f’ and mutt having SET FROM defined in ~/.muttrc allow you to change sending address while emailing from command line but this isn’t what I needed). And so called mighty default MTA, sendmail just brings more panic to it when trying to change it during an in-mail transport (btw, I already dislike sendmail mainly cause of its sluggishness during startup and restart)! As I really had to do something about it and I already knew that Exim’s address rewriting can be helpful here so after installing it and changing default MTA with ‘alternative –config mta’ I added a quick rewrite at transport.

$ grep -A 5 remote_smtp /etc/exim/exim.conf | tail -5

driver = smtp
headers_rewrite = root@*  some-email-address@example.com fsr

This worked but didn’t change the FROM field’s name as expected. I tried combinations of different settings including the rewrite tag ‘F’ but the FROM envelop address kept showing the original sender root. I was searching till I stumbled upon http://www.exim.org/exim-html-2.00/doc/html/spec_32.html#SEC671 where it states the FULL ‘from’ envelop address is changed with ‘w’ tag and allows to have a new sender name (RFC 822) to be configured. So I added.

$ grep -A 5 remote_smtp /etc/exim/exim.conf | tail -5

driver = smtp
# Adding to rewrite the stupid root@ FROM field which I hate
headers_rewrite = root@* "Abbas <some-email-address@example.com>" fsrw
return_path = some-email-address@example.com

And boo you naughty (!) root 

I just finished setting my home XenServer making it fully redundant, mirrored and backup’ed. Thought, I would share what’s on it

- Raid1 setup on boot, grub as well as VM storage repository. If one of the hard goes down, it’ll boot from other one.
- Samba installed to have a share setup in a raid array of 700GB
- Raild alerts are configured to email me whenever there’s any issue
- Smartmontools are running to make sure that hard disks are healthy and have emails alerts configured as well.
- Auto on-the-fly backup of all virtual machines without shutting them off, every night at 1am keeping the two most recent backups of VMs all the time.

One of the easiest way to do so is by changing ’security=user’ to ’security=share’ in global configurations of Samba in /etc/samba/smb.conf. But this raises security concern in case if many shares could have restricted access. Samba maps Windows ‘guest’ account to Linux’s ‘nobody’ account so this is how I enabled it.

Added nobody’s account but seemed it was already there!

[root@ToughGuy ~]# useradd -s /sbin/nologin nobody
useradd: user nobody exists
[root@ToughGuy ~]# grep nobody /etc/passwd
nobody:*:99:99:Nobody:/:/sbin/nologin
nfsnobody:!!:65534:65534:Anonymous NFS User:/var/lib/nfs:/sbin/nologin

Assured that sweet Windows’ “guests” have access to my share

[root@ToughGuy ~]# chown -R root:nobody /Raid/
[root@ToughGuy ~]# chmod -R 775 /Raid/

Mapped guest user to login without a need of having a password prompt in smb.conf’s global settings!

[root@ToughGuy ~]# grep 'map to guest' /etc/samba/smb.conf
	map to guest = Bad Password

Btw, here’s my share:

[root@ToughGuy ~]# grep -A 5 Raid /etc/samba/smb.conf
	path=/Raid
	browseable=yes
	writeable=yes
	guest ok=yes
	public=yes

Restarted the service and look

Is this error freaking you out upon adding a new or second host to your existing set of clusters in Windows Network Load Balance Manager? Well, it always does somehow depending upon the dns and the way of addition being followed

0x800706d5 NLB Error

To fix, make sure:

1. The host you’re adding, your DNS server can resolve its FQDN/Computer name or you’ve a proper entry of it setup in Windows hosts file. This also means that a proper DNS suffix is setup if its a FQDN under Computer name properties.

2. And you’re logging in to this dialog box using full computername i.e. “computername/administrator” instead of merely using computername.

OR

Just in case if you prefer to do things manually like I usually do then this would have already set you free from catchy Windows wizards

Clustering is by no means having any similarity usually. I read somewhere that if you’ll ask ten persons the definition of clustering, you’re likely to get at least nine unique answers – which is what I’ve found true! There are many different kinds of clustering and a lot of ways to do it specially when it comes to Linux. As for as what I’ve learnt in Windows, there are two main categories of purposes where clustering is usually used for. i.e. Static content and non-static content. Network load balancing (NLB) is what used to fulfill the first one and clustering within any shared media for the later. Red Hat does it using the same LVS kernel modules, the one used in Debian but with different set of tools bundled in Piranha rpm (if you’ll Google it out you’ll only find how its done in Debian and that’s the reason why I am writing this tutorial) where its done with heartbeat and ldirectord both of which are available on Red Hat distributions as well but I’ve not been able to fully deploy the same solution on CentOS with LVS, heartbeat and ldirector no more further to a stable point with a mechanism of direct routing of LVS and without no errors. So I went ahead and tried configuring Piranha with an LVS-DR approach where load balanced requests are sent back to end users directly from real (web) servers.

The beauty of Piranha is that it comes with a web based administration panel to configure load balancer configuration which I prefer to configure my first load balancer and then simply copy over the configuration file under /etc/sysconfig/ha/lvs.cf to any further backup load balancer, though there does exist a few documentation files under /usr/share/doc/piranha-*/docs/ if you would like to check over built in sample configs. In this example we would be having these IPs for the machines.

Load Balancer:   12.12.12.40
Real or Web Servers:
VM1:   12.12.12.41
VM2:   12.12.12.41
Virtual IP (VIP):   12.12.12.50  (this is the IP where a FQDN should point to)

Lets move a head and follow the steps below.

1. Downloading and installing the packages

yum install piranha ipvsadm -y

2. Configuration

touch /etc/sysconfig/ipvsadm
sed -i 's/net.ipv4.ip_forward = 0/net.ipv4.ip_forward = 1/' /etc/sysctl.conf
sysctl -p
piranha-passwd
service piranha-gui start

You’ll then browse to http://12.12.12.40:3636/ or http://localhost:3636/ and will be presented a login screen.

Log on in and adjust the settings as below.
A. Configure the main load balancer settings
B. Create a Virtual Server
C. Create two Real Servers within it
D. Leave monitoring segment to its default

About VIP’s settings, of course you can customize; like that of scheduling alogrithims etc.

3. Configuration on Real Servers
Since we configured the webservers to cluster with direct routing, there’s need to configure real servers to prevent reverse ARP havoc. On both of real servers, run:

#Make sure that apache is running on both real servers
service httpd restart && chkconfig httpd on

#Install arptables
yum install arptables_jf -y
chkconfig arptables_jf on
arptables -I IN -d 12.1.2.12.50 -j REJECT
arptables -A OUT -d 12.12.12.50 -j mangle --mangle-ip-s <real-server-ip>
service arptables_jf save && service arptables_jf restart

#Add an alias of VIP on both real servers
ip addr add 12.12.12.50 eth0
echo "ip addr add 12.12.12.50 eth0" >> /etc/rc.local

This can be done with iptables as well but Red Hat recommends using arptables as iptables could be aggressive in ARP filtering. About network alias  some people will tell you to create a secondary loopback alias, either way works so it totally depends upon you!

4. Starting the service on load balancer

After configuring from web panel this is how a fully configured LVS config file looks like:

# cat /etc/sysconfig/ha/lvs.cf
serial_no = 34
primary = 12.12.12.40
service = lvs
backup_active = 0
backup = 0.0.0.0
heartbeat = 1
heartbeat_port = 539
keepalive = 3
deadtime = 25
network = direct
debug_level = NONE
virtual HTTP {
 active = 1
 address = 12.12.12.50 eth0:1
 vip_nmask = 255.255.255.0
 port = 80
 send = "GET / HTTP/1.1\r\n\r\n"
 expect = "HTTP"
 use_regex = 0
 load_monitor = /etc/sysconfig/ha/ruptime-piranha
 scheduler = rr
 protocol = tcp
 timeout = 4
 reentry = 4
 quiesce_server = 1
 server VM1 {
 address = 12.12.12.41
 active = 1
 weight = 1
 }
 server VM2 {
 address = 12.12.12.42
 active = 1
 weight = 1
 }
}

Make sure to click on ‘ACCEPT’ and ‘ACTIVATE/DEACTIVATE’ to make the VIP and Real Servers’ status “up” in the configuration panel – this would save the configuration and now its time to restart pulse service which would use the lvs.cf config file, configure the VIP and start nanny, lvsadm daemons.

chkconfig pulse on
service pulse start

After that it’ll take a few seconds to a minute, start up pulse and logs activity which can be seen from /var/log/messages.

5. Time to test out

Add some different content in both web servers’ virtual host document root. Go to http://12.12.12.50 on any other machine in your network and try reloading the page a couple of times. And you’ll see it’ll be loaded from both of web servers depending upon scheduling algorithm. Watch the output of ipvsadm on load balancer, stop apache on one of web servers and all of requests would get start serving from the other web server with a  noticable change on weight in ipvsadm output falling back to zero for a dead apache server.

 # ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  12.12.12.50:80 rr
 -> 12.12.12.42:80               Route   1      0          0
 -> 12.12.12.41:80               Route   1      0          0

Troubleshooting / Issues:

1. When using weighted scheduling algorithms with nanny’s load monitor, behaviour of load balancer can change than to what is expected.

2. Even load_monitor w/ nanny seems buggy itself as I got a lot of errors like these, after trying a fix found on Piranha’s mailing list which was to use to a customized ruptime in a bash script with rhowd turned on, on all of cluster nodes. The only workaround as for now is to disable load_monitor.

"Nov 29 14:57:03 Methane nanny[7719]: The following exited abnormally:
Nov 29 14:57:03 Methane nanny[7719]: failed to read remote load" 

3. And if you get an error like this one; that means you’re having SELinux on. Use ’sestatus’ to check and /etc/selinux/config to disable it or you better should how to change security context if you don’t want to!

# service piranha-gui start

Starting piranha-gui: (13)Permission denied: make_sock: could not bind to address [::]:3636

no listening sockets available, shutting down

Unable to open logs

[FAILED]