RAID 5 mdadm. Проблемы после замены жесткого диска.
Здравствуйте. Я новичок, прошу помощи.
Ubuntu 10.04.4 LTS x86_64, RAID 5 mdadm, размером в 11TB и почти полностью забитый.
# cat /etc/mdadm/mdadm.conf
DEVICE partitions
ARRAY /dev/md0 level=raid5 num-devices=5 metadata=01.00 name=0 UUID=9e051d43:7a446627:0d3aa958:a6c30ba9
Сбойнул один из дисков:
faulty spare /dev/sde1
State : clean, degraded
В таком состоянии он проработал несколько недель (может и больше). Я размонтировал рейд, удалил сбойный диск из рейда, подготовил новый жесткий для замены и добавил его в рейд.
Утром посмотрел mdstat.
# cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sde1[7](S) sdc1[5] sdd1[4] sdb1[1](F) sdf1[6]
11721058304 blocks super 1.0 level 5, 512k chunk, algorithm 2 [5/3] [__UUU]
unused devices: <none>
# mdadm --detail /dev/md0
mdadm: metadata format 01.00 unknown, ignored.
/dev/md0:
Version : 01.00
Creation Time : Mon Nov 4 09:51:43 2013
Raid Level : raid5
Array Size : 11721058304 (11178.07 GiB 12002.36 GB)
Used Dev Size : 5860529152 (5589.04 GiB 6001.18 GB)
Raid Devices : 5
Total Devices : 5
Preferred Minor : 0
Persistence : Superblock is persistent
Update Time : Sat Jan 6 07:14:55 2018
State : clean, degraded
Active Devices : 3
Working Devices : 4
Failed Devices : 1
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Name : 0
UUID : 9e051d43:7a446627:0d3aa958:a6c30ba9
Events : 750198
Number Major Minor RaidDevice State
0 0 0 0 removed
1 0 0 1 removed
5 8 33 2 active sync /dev/sdc1
4 8 49 3 active sync /dev/sdd1
6 8 81 4 active sync /dev/sdf1
1 8 17 - faulty spare /dev/sdb1
7 8 65 - spare /dev/sde1
Новый диск, который я добавил:
7 8 65 - spare /dev/sde1
И теперь появился еще один сбойный:
1 8 17 - faulty spare /dev/sdb1
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 5
3 Spin_Up_Time 0x0027 142 142 021 Pre-fail Always - 11858
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 31
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 050 050 000 Old_age Always - 36531
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 31
183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 25
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 5
194 Temperature_Celsius 0x0022 107 094 000 Old_age Always - 45
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 6
# cat /var/log/messages
Jan 6 04:10:18 access kernel: [33259.993923] ata2.00: configured for UDMA/133
Jan 6 04:10:18 access kernel: [33259.993950] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.285997] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.286026] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.390773] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.390797] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.482241] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.482265] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.573688] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.573712] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.665190] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.665228] sd 1:0:0:0: [sdb] Unhandled sense code
Jan 6 04:10:21 access kernel: [33260.665230] sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 6 04:10:21 access kernel: [33260.665233] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Jan 6 04:10:21 access kernel: [33260.665237] Descriptor sense data with sense descriptors (in hex):
Jan 6 04:10:21 access kernel: [33260.665239] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 01
Jan 6 04:10:21 access kernel: [33260.665245] 28 f9 78 f4
Jan 6 04:10:21 access kernel: [33260.665247] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Jan 6 04:10:21 access kernel: [33260.665252] sd 1:0:0:0: [sdb] CDB: Read(16): 88 00 00 00 00 01 28 f9 78 30 00 00 00 d0 00 00
Jan 6 04:10:21 access kernel: [33260.665263] raid5:md0: read error not correctable (sector 4982403312 on sdb1).
Jan 6 04:10:21 access kernel: [33260.665270] raid5:md0: read error not correctable (sector 4982403320 on sdb1).
Jan 6 04:10:21 access kernel: [33260.665279] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.764633] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.764655] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.856082] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.856105] ata2: EH complete
Jan 6 04:10:21 access kernel: [33260.955856] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33260.955878] ata2: EH complete
Jan 6 04:10:21 access kernel: [33261.055601] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33261.055623] sd 1:0:0:0: [sdb] Unhandled sense code
Jan 6 04:10:21 access kernel: [33261.055625] sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 6 04:10:21 access kernel: [33261.055628] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Jan 6 04:10:21 access kernel: [33261.055631] Descriptor sense data with sense descriptors (in hex):
Jan 6 04:10:21 access kernel: [33261.055633] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 01
Jan 6 04:10:21 access kernel: [33261.055638] 28 f9 79 00
Jan 6 04:10:21 access kernel: [33261.055641] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Jan 6 04:10:21 access kernel: [33261.055645] sd 1:0:0:0: [sdb] CDB: Read(16): 88 00 00 00 00 01 28 f9 79 00 00 00 00 10 00 00
Jan 6 04:10:21 access kernel: [33261.055655] raid5:md0: read error not correctable (sector 4982403328 on sdb1).
Jan 6 04:10:21 access kernel: [33261.055661] raid5:md0: read error not correctable (sector 4982403336 on sdb1).
Jan 6 04:10:21 access kernel: [33261.055672] ata2: EH complete
Jan 6 04:10:21 access kernel: [33261.155367] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33261.155389] ata2: EH complete
Jan 6 04:10:21 access kernel: [33261.246832] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33261.246854] ata2: EH complete
Jan 6 04:10:21 access kernel: [33261.346604] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33261.346626] ata2: EH complete
Jan 6 04:10:21 access kernel: [33261.446357] ata2.00: configured for UDMA/133
Jan 6 04:10:21 access kernel: [33261.446380] sd 1:0:0:0: [sdb] Unhandled sense code
Jan 6 04:10:21 access kernel: [33261.446382] sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 6 04:10:21 access kernel: [33261.446385] sd 1:0:0:0: [sdb] Sense Key : Medium Error [current] [descriptor]
Jan 6 04:10:21 access kernel: [33261.446388] Descriptor sense data with sense descriptors (in hex):
Jan 6 04:10:21 access kernel: [33261.446390] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 01
Jan 6 04:10:21 access kernel: [33261.446396] 28 f9 79 10
Jan 6 04:10:21 access kernel: [33261.446398] sd 1:0:0:0: [sdb] Add. Sense: Unrecovered read error - auto reallocate failed
Jan 6 04:10:21 access kernel: [33261.446402] sd 1:0:0:0: [sdb] CDB: Read(16): 88 00 00 00 00 01 28 f9 79 10 00 00 00 f0 00 00
Jan 6 04:10:21 access kernel: [33261.446413] raid5:md0: read error not correctable (sector 4982403344 on sdb1).
Jan 6 04:10:21 access kernel: [33261.446418] raid5:md0: read error not correctable (sector 4982403352 on sdb1).
Jan 6 04:10:21 access kernel: [33261.446421] raid5:md0: read error not correctable (sector 4982403360 on sdb1).
Jan 6 04:10:21 access kernel: [33261.446424] raid5:md0: read error not correctable (sector 4982403368 on sdb1).
Jan 6 04:10:21 access kernel: [33261.446426] raid5:md0: read error not correctable (sector 4982403376 on sdb1).
Jan 6 04:10:21 access kernel: [33261.446429] raid5:md0: read error not correctable (sector 4982403384 on sdb1).
Jan 6 04:10:21 access kernel: [33261.446454] ata2: EH complete
Jan 6 04:10:21 access kernel: [33261.453315] md: md0: recovery done.
Jan 6 04:10:21 access kernel: [33261.577963] RAID5 conf printout:
Jan 6 04:10:21 access kernel: [33261.577966] --- rd:5 wd:3
Jan 6 04:10:21 access kernel: [33261.577969] disk 0, o:1, dev:sde1
Jan 6 04:10:21 access kernel: [33261.577971] disk 1, o:0, dev:sdb1
Jan 6 04:10:21 access kernel: [33261.577973] disk 2, o:1, dev:sdc1
Jan 6 04:10:21 access kernel: [33261.577974] disk 3, o:1, dev:sdd1
Jan 6 04:10:21 access kernel: [33261.577976] disk 4, o:1, dev:sdf1
Jan 6 04:10:21 access kernel: [33262.252744] RAID5 conf printout:
Jan 6 04:10:21 access kernel: [33262.252748] --- rd:5 wd:3
Jan 6 04:10:21 access kernel: [33262.252751] disk 1, o:0, dev:sdb1
Jan 6 04:10:21 access kernel: [33262.252753] disk 2, o:1, dev:sdc1
Jan 6 04:10:21 access kernel: [33262.252755] disk 3, o:1, dev:sdd1
Jan 6 04:10:21 access kernel: [33262.252757] disk 4, o:1, dev:sdf1
Jan 6 04:10:21 access kernel: [33262.252765] RAID5 conf printout:
Jan 6 04:10:21 access kernel: [33262.252766] --- rd:5 wd:3
Jan 6 04:10:21 access kernel: [33262.252768] disk 1, o:0, dev:sdb1
Jan 6 04:10:21 access kernel: [33262.252770] disk 2, o:1, dev:sdc1
Jan 6 04:10:21 access kernel: [33262.252772] disk 3, o:1, dev:sdd1
Jan 6 04:10:21 access kernel: [33262.252774] disk 4, o:1, dev:sdf1
Jan 6 04:10:21 access kernel: [33262.278896] RAID5 conf printout:
Jan 6 04:10:21 access kernel: [33262.278900] --- rd:5 wd:3
Jan 6 04:10:21 access kernel: [33262.278903] disk 2, o:1, dev:sdc1
Jan 6 04:10:21 access kernel: [33262.278906] disk 3, o:1, dev:sdd1
Jan 6 04:10:21 access kernel: [33262.278908] disk 4, o:1, dev:sdf1
Попытался смонтировать рейд.
# mount -t ext4 /dev/md0 /media/test/
mount: wrong fs type, bad option, bad superblock on /dev/md0