www/lists/mx/ftp4 inactivity

[Adam Gołębiowski]

To those who don't waste their time on irc, here's a short info on what
has happened in the past few days.


As you have noticed, lists (and www, mx and ftp4) were down since
Thursday. At Wednesday, about 11 PM, I/O error occured while trying to
operate on filesystem laying on lvm on a 3*250GB raid5. Windows-style
fix (let's try to reboot it) only made things worse - the machine didn't
boot up.

Long story short, with help from rmf, we managed to recover most of the
data (maybe just a few mails from mx's spool got lost) and are currently
running www,mx and lists from a machine located in Prague, Czech. 


For those of you who are interesed, what I *suspect* has happened is:

There is a faulty mainboard *or* a faulty sata controller (yet to be
checked). One of these resulted (can't provide you with exact message
right now, can do this in the evening) in a system going to become
unstable.

When both sata controller and mainboard were replaced, we tried to
rebuild the raid. Another problems occured:
	
	http://adamg.agmk.net/mdadm.txt

After performing some magic, RMF managed to force resynchronization of
this raid. We thought we were home, but we weren't. After 99% of the
synchronization was completed, it turned out one of the active disks (sdd
to be exact) had badblocks on it.

What we decided to do was to force synchronization once more, slow down
the process (through /proc/sys/dev/raid/speed_limit_m{ax,in}), run lvm
on it, mount the logical volumes and copy everything we wanted to copy
before the sync. process meets the badblocks.

I will try do some tests on akcyza's hardware in the next few days to
check if the mainboard is really faulty. The harddisk, of course, needs
replacement (it is on warranty, iirc). 


adamg.

-- 
 http://www.mysza.eu.org/ | Everybody needs someone sure, someone true,
   PLD Linux developer    | Everybody needs some solid rock, I know I do.