MegaCLI, Raid Levels

February 7, 2023 Uncategorized

Common invocations of MegaCLI

Get an Overview of controllers and attached disk arrays

./MegaCli64 -AdpAllInfo -aALL (extracted info below)
./MegaCli64 -CFGDsply -aALL shows same info, with same verbosity

Adapter #0
========================================
Product Name : PERC H800 Adapter Serial No : 25B0088

Link Speed           : 0
 Number of Frontend Port: 0
 Device Interface  : PCIE
Number of Backend Port: 8
 Port  :  Address
 0        500c04f2e958eebf 
 1        500c04f2e958ee3f 
 2        0000000000000000 
 3        0000000000000000 
 4        0000000000000000 
 5        0000000000000000 
 6        0000000000000000 
 7        0000000000000000
Device Present
 ================
 Virtual Drives    : 1
 Degraded          : 0   
 Offline               : 0 
 Physical Devices  : 13
 Disks                : 12   
 Critical Disks     : 0   
 Failed Disks      : 0

(snip)

Adapter #1
==============================================================================
Product Name : PERC 6/i Integrated
Serial No : 1122334455667788

Device Present
 ================
 Virtual Drives      : 1   
 Degraded            : 0   
 Offline                : 0 
 Physical Devices  : 3   
 Disks                  : 2   
 Critical Disks       : 0   
 Failed Disks         : 0
Settings
 ================
 Current Time                     : 12:16:19 6/18, 2013
 Predictive Fail Poll Interval    : 300sec
 Rebuild Rate                     : 30%
 Reconstruction Rate              : 30%
 Cache Flush Interval             : 4s
 Max Drives to Spinup at One Time : 2
 Delay Among Spinup Groups        : 12s
 Restore HotSpare on Insertion    : Disabled
 Expose Enclosure Devices         : Disabled
 Maintain PD Fail History         : Disabled
 Maximum number of direct attached drives to spin up in 1 min : 0
 Auto Enhanced Import             : No

Capabilities
 ================
 The last paragraph above mentions many unusual RAID levels. Here is a brief explanation of the more exotic.
 RAID Level Supported             : 
     RAID0, RAID1, RAID5, RAID6, RAID00, RAID10, RAID50, RAID60, PRL 11, PRL 11 with spanning, 
     SRL 3 supported, PRL11-RLQ0 DDF layout with no span, PRL11-RLQ0 DDF layout with span
 Supported Drives                 : SAS, SATA

— Explanations of RAID Levels

This PDF explains secondary level raid etc: http://www.snia.org/sites/default/files/SNIA_DDF_Technical_Position_v2.0.pdf

RAID6 – Dual Parity RAID:
Striped stripes: Break up a file into stripes and stripe it across all drives. Need multiple raid controllers and large files for this to provide benefit.
PRL11-RLQ0 layout : aka RAID1E Integrated Adjacent Stripe Mirroring
Each block is written twice. The location of the mirror/parity block is defined is being in the same extent, but on the next adjacent disk. Spanning is needed when you have different sized Basic Virtual Drives (subdisks).
SRL3: Spanned Secondary Raid Level 3:
(less protection) : data stripes are allocated pretty much where ever they can be – striped or concatenated from any available subdisks (aka bvd). The subdisk may or may not be raid-protected.
RAID Level Qualifier:
This little guy says nothing about the level of protection, only that there are several ways to “skin a disk”. RLQ assigns numbers to them.

Examine drives on one adapter

root@auspcrpbak03:MegaCli> ./MegaCli64 -LdPdInfo -a0

Notes: There are 12 drives of about 1.8TB. We’re running RAID6, so we expect to lose 2 drives to parity and get 18TB.

Adapter #0
Number of Virtual Disks: 1 
 Virtual Drive: 0 (Target Id: 0) 
 Name                     :Virtual Disk 0 
 RAID Level              : Primary-6, Secondary-0, RAID Level Qualifier-3 
 Size                        : 18.188 TB                  ( This is size presented to OS ) 
 Sector Size              : 512 
 Parity Size               : 3.637 TB          (In addition to the 18T. For RAID6 we have two)
 State                       : Optimal 
 Number Of Drives     : 12 
 Current Cache Policy : WriteThrough, ReadAdaptive, Direct, No Write Cache if Bad BBU Number of PDs: 12

—
Explanation of Raid Levels:
[‘Primary-0, Secondary-0, RAID Level Qualifier-0’] = RAID-0
[‘Primary-1, Secondary-0, RAID Level Qualifier-0’] = RAID-1
[‘Primary-5, Secondary-0, RAID Level Qualifier-3’] = RAID-5
[‘Primary-6, Secondary-0, RAID Level Qualifier-3’] = RAID-6
[‘Primary-1, Secondary-3, RAID Level Qualifier-0’] = RAID-10

Source: http://tools.rapidsoft.de/perc/rs_check_raid_perc5i.class.php.txt

 --- PD: 0 
Information Enclosure Device ID: 16  
Slot Number        : 0  
Drive's position    : DiskGroup: 0, Span: 0, Arm: 0 
 Enclosure position: N/A  
Device Id             : 13  
WWN: 5000C500558ADE08  
Sequence Number: 2 
Media Error Count : 0  
PD Type               : SAS
Raw Size: 1.819 TB [0xe8e088b0 Sectors]       (This is real 1024M size, not 1000M. It's sold as a 2TB drive)
 Firmware state:    Online, Spun Up Device 
 SAS Address(0):   0x5000c500558ade09 
 SAS Address(1):   0x5000c500558ade0a 
 Connected Port Number:   1(path0) 0(path1) 
 Inquiry Data:         SEAGATE ST32000645SS    RS0FZ1K0AV9E 
 Drive has flagged a S.M.A.R.T alert : No
PD: 1
 Information Enclosure Device ID: 16 
 Slot Number:                              1 
 Drive's position:   DiskGroup: 0, Span: 0, Arm: 1 Device Id: 12 
 WWN:                                        5000C500558AEBA4 
 Sequence Number:                     2 
 Media Error Count: 0 PD Type:    SAS

< and much more >

Optional toppings for most LD configurations:

WT : WriteThrough safer. Only returns once data is written to disk.
WB: WriteBack faster. Returns as soon as data is is written to cache
NORA : No Read Ahead vs
RA: ReadAhead vs
ADRA : Adaptive ReadAhead where if the previous two requests were sequential, it pre-loads the next in sequence.
Cached: Cache reads.
Direct: Only the previous read is cached.
-strpszM : Stripe size so -strpsz64 means 64kb stripe size.
Hsp[E0:S0] : Choose this drive to be a hot-spare

Dump the Event Log for the Adapter

./MegaCli64 -AdpEventLog -GetLatest 20 -f /tmp/events.txt -a0

Build a Virtual Disk

rX : Raid Level X. You can see your options by looking at the raid adapter settings (top of this page).
In this example we can choose 0,1,5,6,00,10,50,60 and more

[E0:S1,E0:S3] – Enclosure0:Slot1, Enclosure0:Slot3 : 2 Drives to use in the VD. Include the square brackets.

Configure disks as JBOD

MegaCli64 -AdpSetProp -EnableJBOD 1 -a0

Configure RAID 10

MegaCli64 -CfgSpanAdd -R10 -Array0 [E0:S1,E0:S2,E0,S3] -Array1 [E1:S0,E1:S1,E1:S2] + Toppings above

Delete the first logical drive on the first controller

MegaCli64 -CfgLdDel -L0 -a0

Replace a Failed Drive

MegaCli64 -PDOffline -PhysDrv [32:4] -a0
MegaCli64 -PDPrpRmv -PhysDrv [32:4] -a0
Pull the drive, insert a new one
MegaCli64 -Pdgetmissing -a0Adapter 0 – Missing Physical drives
No. Array Row Size Expected
0 1 4 1907200 MB

MegaCli64 -PdReplaceMissing -PhysDrv [32:4] -Array1 -row4 -a0

Rebuild the Array

MegaCli64 -PDRbld -Start -PhysDrv [32:4] -a0

Show Battery Backup Info

MegaCli64 -AdpBbuCmd -aALL

RAID Rebuild Rate – Increase to 90 Percent

MegaCli64 -AdpSetProp BgiRate 90 -aALL

Save configuration to filesystem

MegaCli64 -CfgSave -fFilename -a0

When you replace controllers, you can duplicate the previous config. In BIOS look for ‘Add Saved Configuration’. Say ‘Y’ to the warning to import it from the disks. (Be sure that your disk array is healthy first.)

RAID 10 and variants

Due to Dawid’s comment below, I did some more research, refreshed my recollections about RAID10 and similar configurations and have the following comments.
RAID10 is shorthand for RAID1+0 which is a stripe across mirrors. (Take RAID1/mirrored disks and RAID0/stripe them).
RAID01 is shorthand for RAID0+1 which mirrors disks that are striped. (Take RAID0 disks and RAID1/mirror them).
RAID01 is a less common config because RAID10 offers more redundancy (a big reason for why we RAID) and few disadvantages compared to RAID01.
In the case of four disks (which is a common config), I don’t (personally) see a big difference, but things get interesting when you have six or more disks. So let’s start there.

At this point, an analogy may make things more interesting, even if most readers already fully understand what I am saying. When I was younger, I played a game called Battleships. Assuming you have to carry eight squares of cargo, which would you prefer: two four-unit battleships or four two-unit destroyers? When the bombs start falling, you would much prefer to have your cargo in four different locations. While this is true when a malevolent opponent is trying to locate and destroy your ships, it’s also true when chaos sets in on your disk arrays. Forcing Murphy to precisely take out both disks in the same mirror (for example both yellows from the left picture), rather than one disk from each stripe is a better strategy.

The image also clarifies somewhat the “Primary” and “Secondary” listings in the output from megacli. Primary is the “RAID1” part of “RAID1+0” (say the two yellow squares). It is the fundamental building block of the array. The “Secondary” field is how you organize the primal units you first created. For a RAID1+0, we stripe them (RAID 0).

Dawid’s situation was that he has 4 disks, but they show as (Pri-1, Sec-0, Qual-0) which is how megacli’s represents a plain mirror. I suspect that he manually created the striped disks then mirrored them and that megacli did not keep track of all the steps. I expect that it will work out just fine, but it would be more obvious and faster if next time he ran ‘megacli -CfgLdAdd -r10 …’