My mdadm is insane.

[swestrup]

This is mainly long, boring linux geekery.

So, it turns out the reason I couldn't boot right yesterday is that I was letting mdadm create its own config file. Here's my partitions:

# fdisk -l

Disk /dev/hda: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hda1   *           1          13      104391   fd  Linux raid autodetect
/dev/hda2              14       30515   245007315    5  Extended
/dev/hda5              14        1229     9767488+  fd  Linux raid autodetect
/dev/hda6            1230        2445     9767488+  fd  Linux raid autodetect
/dev/hda7            2446        2567      979933+  82  Linux swap
/dev/hda8            2568        4520    15687441   fd  Linux raid autodetect
/dev/hda9            4521       10599    48829536   fd  Linux raid autodetect
/dev/hda10          10600       30515   159975238+  fd  Linux raid autodetect

Disk /dev/hdc: 251.0 GB, 251000193024 bytes
255 heads, 63 sectors/track, 30515 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hdc1   *           1          13      104391   fd  Linux raid autodetect
/dev/hdc2              14       30515   245007315    5  Extended
/dev/hdc5              14        1229     9767488+  fd  Linux raid autodetect
/dev/hdc6            1230        2445     9767488+  fd  Linux raid autodetect
/dev/hdc7            2446        2567      979933+  82  Linux swap
/dev/hdc8            2568        4520    15687441   fd  Linux raid autodetect
/dev/hdc9            4521       10599    48829536   fd  Linux raid autodetect
/dev/hdc10          10600       30515   159975238+  fd  Linux raid autodetect

Disk /dev/hde: 163.9 GB, 163927522816 bytes
255 heads, 63 sectors/track, 19929 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/hde1   *           1          13      104391   fd  Linux raid autodetect
/dev/hde2              14       19929   159975270    5  Extended
/dev/hde5              14       19929   159975238+  fd  Linux raid autodetect

and here's my old standard /etc/mdadm.conf file:

# cat /etc/mdadm.conf

DEVICE partitions
ARRAY /dev/md5 level=raid1 num-devices=2 UUID=616b1695:02cb7ebe:82e85d00:dedcc480
ARRAY /dev/md6 level=raid1 num-devices=2 UUID=982148de:7833605e:c8573b80:74e5333a
ARRAY /dev/md8 level=raid0 num-devices=2 UUID=c8f6b386:3e325f6a:db005aed:e3f5157f
ARRAY /dev/md9 level=raid1 num-devices=2 UUID=e06a9c30:76442164:8e8545fd:85f22bed
ARRAY /dev/md10 level=raid5 num-devices=3 UUID=bb49f597:9f4f4fd9:dd7766ef:e222238c

As you can see from /proc/mdstat, this is getting built correctly:

# cat /proc/mdstat

Personalities : [raid0] [raid1] [raid5]
md8 : active raid0 hda8[0] hdc8[1]
      31374720 blocks 64k chunks

md5 : active raid1 hda5[1] hdc5[0]
      9767424 blocks [2/2] [UU]

md6 : active raid1 hda6[0] hdc6[1]
      9767424 blocks [2/2] [UU]

md9 : active raid1 hda9[0] hdc9[1]
      48829440 blocks [2/2] [UU]

md1 : active raid1 hda1[2] hdc1[0] hde1[1]
      104320 blocks [3/3] [UUU]

md10 : active raid5 hda10[0] hde5[2] hdc10[1]
      319950336 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]

unused devices: <none>

But when I try to regenerate the config file in the standard way, this is what I get:

# mdadm -Esc partitions

ARRAY /dev/md5 level=raid1 num-devices=2 UUID=616b1695:02cb7ebe:82e85d00:dedcc480
ARRAY /dev/md6 level=raid1 num-devices=2 UUID=982148de:7833605e:c8573b80:74e5333a
ARRAY /dev/md7 level=raid0 num-devices=2 UUID=c8f6b386:3e325f6a:db005aed:e3f5157f
ARRAY /dev/md9 level=raid1 num-devices=2 UUID=e06a9c30:76442164:8e8545fd:85f22bed
ARRAY /dev/md10 level=raid5 num-devices=3 UUID=bb49f597:9f4f4fd9:dd7766ef:e222238c

Notice how they're the same except that /dev/md8 is replaced by /dev/md7?
What's more, if I try to generate it some other way like, for instance, iterating through the existing md devices, I get this:

# mdadm -E /dev/md5
mdadm: No md superblock detected on /dev/md5.

I get that for all the md devices, despite the fact I can query them:

# mdadm -Q /dev/md5
/dev/md5: 9.31GiB raid1 2 devices, 0 spares. Use mdadm --detail for more detail.

So, for the time being everything will be fine if I just don't autogenerate myself a mdadm.conf file, but I wonder how/why it broke?