md-cluster.txt 12.4 KB
Newer Older
1 2 3 4 5
The cluster MD is a shared-device RAID for a cluster.

1. On-disk format

Separate write-intent-bitmaps are used for each cluster node.
7 8 9 10 11 12 13 14 15 16
The bitmaps record all writes that may have been started on that node,
and may not yet have finished. The on-disk layout is:

0                    4k                     8k                    12k
| idle                | md super            | bm super [0] + bits |
| bm bits[0, contd]   | bm super[1] + bits  | bm bits[1, contd]   |
| bm super[2] + bits  | bm bits [2, contd]  | bm super[3] + bits  |
| bm bits [3, contd]  |                     |                     |

17 18 19
During "normal" functioning we assume the filesystem ensures that only
one node writes to any given block at a time, so a write request will

20 21 22 23
 - set the appropriate bit (if not already set)
 - commit the write to all mirrors
 - schedule the bit to be cleared after a timeout.

24 25
Reads are just handled normally. It is up to the filesystem to ensure
one node doesn't read from a location where another node (or the same
26 27 28 29 30
node) is writing.

2. DLM Locks for management

There are three groups of locks for managing the device:
32 33 34

2.1 Bitmap lock resource (bm_lockres)

35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60
 The bm_lockres protects individual node bitmaps. They are named in
 the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
 node joins the cluster, it acquires the lock in PW mode and it stays
 so during the lifetime the node is part of the cluster. The lock
 resource number is based on the slot number returned by the DLM
 subsystem. Since DLM starts node count from one and bitmap slots
 start from zero, one is subtracted from the DLM slot number to arrive
 at the bitmap slot number.

 The LVB of the bitmap lock for a particular node records the range
 of sectors that are being re-synced by that node.  No other
 node may write to those sectors.  This is used when a new nodes
 joins the cluster.

2.2 Message passing locks

 Each node has to communicate with other nodes when starting or ending
 resync, and for metadata superblock updates.  This communication is
 managed through three locks: "token", "message", and "ack", together
 with the Lock Value Block (LVB) of one of the "message" lock.

2.3 new-device management

 A single lock: "no-new-dev" is used to co-ordinate the addition of
 new devices - this must be synchronized across the array.
 Normally all nodes hold a concurrent-read lock on this device.
61 62 63

3. Communication

64 65 66
 Messages can be broadcast to all nodes, and the sender waits for all
 other nodes to acknowledge the message before proceeding.  Only one
 message can be processed at a time.
67 68 69

3.1 Message Types

 There are six types of messages which are passed:

72 73 74 75
 3.1.1 METADATA_UPDATED: informs other nodes that the metadata has
   been updated, and the node must re-read the md superblock. This is
   performed synchronously. It is primarily used to signal device

77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96
 3.1.2 RESYNCING: informs other nodes that a resync is initiated or
   ended so that each node may suspend or resume the region.  Each
   RESYNCING message identifies a range of the devices that the
   sending node is about to resync. This over-rides any pervious
   notification from that node: only one ranged can be resynced at a
   time per-node.

 3.1.3 NEWDISK: informs other nodes that a device is being added to
   the array. Message contains an identifier for that device.  See
   below for further details.

 3.1.4 REMOVE: A failed or spare device is being removed from the
   array. The slot-number of the device is included in the message.

 3.1.5 RE_ADD: A failed device is being re-activated - the assumption
   is that it has been determined to be working again.

 3.1.6 BITMAP_NEEDS_SYNC: if a node is stopped locally but the bitmap
   isn't clean, then another node is informed to take the ownership of
97 98 99 100 101 102

3.2 Communication mechanism

 The DLM LVB is used to communicate within nodes of the cluster. There
 are three resources used for the purpose:

  3.2.1 token: The resource which protects the entire communication
104 105 106
   system. The node having the token resource is allowed to

  3.2.2 message: The lock resource which carries the data to
108 109

  3.2.3 ack: The resource, acquiring which means the message has been
   acknowledged by all nodes in the cluster. The BAST of the resource
112 113
   is used to inform the receiving node that a node wants to
114 115 116

The algorithm is:

 1. receive status - all nodes have concurrent-reader lock on "ack".

119 120
   sender                         receiver                 receiver
   "ack":CR                       "ack":CR                 "ack":CR

122 123
 2. sender get EX on "token"
    sender get EX on "message"
    sender                        receiver                 receiver
125 126 127
    "token":EX                    "ack":CR                 "ack":CR

129 130 131
    Sender checks that it still needs to send a message. Messages
    received or other events that happened while waiting for the
    "token" may have made this message inappropriate or redundant.

133 134 135 136
 3. sender writes LVB.
    sender down-convert "message" from EX to CW
    sender try to get EX of "ack"
    [ wait until all receivers have *processed* the "message" ]

138 139
                                     [ triggered by bast of "ack" ]
                                     receiver get CR on "message"
140 141 142
                                     receiver read LVB
                                     receiver processes the message
                                     [ wait finish ]
143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158
                                     receiver releases "ack"
                                     receiver tries to get PR on "message"

   sender                         receiver                  receiver
   "token":EX                     "message":CR              "message":CR

 4. triggered by grant of EX on "ack" (indicating all receivers
    have processed message)
    sender down-converts "ack" from EX to CR
    sender releases "message"
    sender releases "token"
                               receiver upconvert to PR on "message"
                               receiver get CR of "ack"
                               receiver release "message"
159 160

   sender                      receiver                   receiver
   "ack":CR                    "ack":CR                   "ack":CR
162 163 164 165 166

4. Handling Failures

4.1 Node Failure
167 168 169 170 171

 When a node fails, the DLM informs the cluster with the slot
 number. The node starts a cluster recovery thread. The cluster
 recovery thread:

172 173 174 175 176 177 178
	- acquires the bitmap<number> lock of the failed node
	- opens the bitmap
	- reads the bitmap of the failed node
	- copies the set bitmap to local node
	- cleans the bitmap of the failed node
	- releases bitmap<number> lock of the failed node
	- initiates resync of the bitmap on the current node
179 180 181 182 183
		md_check_recovery is invoked within recover_bitmaps,
		then md_check_recovery -> metadata_update_start/finish,
		it will lock the communication by lock_comm.
		Which means when one node is resyncing it blocks all
		other nodes from writing anywhere on the array.

 The resync process is the regular md resync. However, in a clustered
186 187
 environment when a resync is performed, it needs to tell other nodes
 of the areas which are suspended. Before a resync starts, the node
188 189 190 191 192 193 194
 send out RESYNCING with the (lo,hi) range of the area which needs to
 be suspended. Each node maintains a suspend_list, which contains the
 list of ranges which are currently suspended. On receiving RESYNCING,
 the node adds the range to the suspend_list. Similarly, when the node
 performing resync finishes, it sends RESYNCING with an empty range to
 other nodes and other nodes remove the corresponding entry from the

196 197
 A helper function, ->area_resyncing() can be used to check if a
 particular I/O range should be suspended or not.
198 199

4.2 Device Failure

 Device failures are handled and communicated with the metadata update
202 203 204
 routine.  When a node detects a device failure it does not allow
 any further writes to that device until the failure has been
 acknowledged by all other nodes.
205 206

5. Adding a new Device
207 208 209

 For adding a new device, it is necessary that all nodes "see" the new
 device to be added. For this, the following algorithm is used:
210 211

    1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues
212 213
       ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)
    2. Node 1 sends a NEWDISK message with uuid and slot number
214 215 216 217
    3. Other nodes issue kobject_uevent_env with uuid and slot number
       (Steps 4,5 could be a udev rule)
    4. In userspace, the node searches for the disk, perhaps
       using blkid -t SUB_UUID=""
218 219
    5. Other nodes issue either of the following depending on whether
       the disk was found:
       ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and
             disc.number set to slot number)
223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318
    6. Other nodes drop lock on "no-new-devs" (CR) if device is found
    7. Node 1 attempts EX lock on "no-new-dev"
    8. If node 1 gets the lock, it sends METADATA_UPDATED after
       unmarking the disk as SpareLocal
    9. If not (get "no-new-dev" lock), it fails the operation and sends
   10. Other nodes get the information whether a disk is added or not
       by the following METADATA_UPDATED.

6. Module interface.

 There are 17 call-backs which the md core can make to the cluster
 module.  Understanding these can give a good overview of the whole

6.1 join(nodes) and leave()

 These are called when an array is started with a clustered bitmap,
 and when the array is stopped.  join() ensures the cluster is
 available and initializes the various resources.
 Only the first 'nodes' nodes in the cluster can use the array.

6.2 slot_number()

 Reports the slot number advised by the cluster infrastructure.
 Range is from 0 to nodes-1.

6.3 resync_info_update()

 This updates the resync range that is stored in the bitmap lock.
 The starting point is updated as the resync progresses.  The
 end point is always the end of the array.
 It does *not* send a RESYNCING message.

6.4 resync_start(), resync_finish()

 These are called when resync/recovery/reshape starts or stops.
 They update the resyncing range in the bitmap lock and also
 send a RESYNCING message.  resync_start reports the whole
 array as resyncing, resync_finish reports none of it.

 resync_finish() also sends a BITMAP_NEEDS_SYNC message which
 allows some other node to take over.

6.5 metadata_update_start(), metadata_update_finish(),

 metadata_update_start is used to get exclusive access to
 the metadata.  If a change is still needed once that access is
 gained, metadata_update_finish() will send a METADATA_UPDATE
 message to all other nodes, otherwise metadata_update_cancel()
 can be used to release the lock.

6.6 area_resyncing()

 This combines two elements of functionality.

 Firstly, it will check if any node is currently resyncing
 anything in a given range of sectors.  If any resync is found,
 then the caller will avoid writing or read-balancing in that

 Secondly, while node recovery is happening it reports that
 all areas are resyncing for READ requests.  This avoids races
 between the cluster-filesystem and the cluster-RAID handling
 a node failure.

6.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()

 These are used to manage the new-disk protocol described above.
 When a new device is added, add_new_disk_start() is called before
 it is bound to the array and, if that succeeds, add_new_disk_finish()
 is called the device is fully added.

 When a device is added in acknowledgement to a previous
 request, or when the device is declared "unavailable",
 new_disk_ack() is called.

6.8 remove_disk()

 This is called when a spare or failed device is removed from
 the array.  It causes a REMOVE message to be send to other nodes.

6.9 gather_bitmaps()

 This sends a RE_ADD message to all other nodes and then
 gathers bitmap information from all bitmaps.  This combined
 bitmap is then used to recovery the re-added device.

6.10 lock_all_bitmaps() and unlock_all_bitmaps()

 These are called when change bitmap to none. If a node plans
 to clear the cluster raid's bitmap, it need to make sure no other
 nodes are using the raid which is achieved by lock all bitmap
 locks within the cluster, and also those locks are unlocked
319 320 321 322 323 324

7. Unsupported features

There are somethings which are not supported by cluster MD yet.

- update size and change array_sectors.