A couple years ago, I bought a 12G/s SAS disk to see if I could get it to work with the RAID controller with external SFF-8088 ports which came with the system I got while I was working at The Linux Foundation. I got an enclosure to go with it because I was enthusiastic and optimistic about my ability to get things all set up. My plan was to take the 1T disks I had in the storage server I had been using before, but which had since failed and to put them into this new enclosure. I got a couple of SFF-8088 cables, and the enclosure had some SFF-8088 to 7-pin SATA break-outs come with it. I bought an additional dual SFF-8088 to SFF-8087 adapter and a couple of SFF-8087 to 4x SFF-8482 cables. I got the 8482s because I imagined that these would be required to take full advantage of the 6TB 12G/s SAS disk mentioned above.
Unfortunately, when I attached the disks to the controller, none of them worked and I gave up frustrated and disillusioned for a good long time. Last week was spring break, and I took a few days off from work to hang out with my girls. While we weren’t all goofing off together, I was puttering around with a new HBA I got to replace the apparently non-functional HP RAID controller which came with the server.
However, when I removed the HP RAID controller and put in the LSI 9201-16E HBA I got to replace it with, there was again no SCSI love. The lsscsi command showed nothing. But the lspci command showed the controller, and I started digging through documentation to figure out what might be wrong. I eventually stumbled upon a firmware flashing ISO disk image on the Broadcom site. Broadcom apparently purchased LSI not too long ago and is now responsible for managing documentation and downloads for the legacy devices, of which I am now a proud owner.
I was able to build a USB disk from this ISO using unetbootin and emacs. I had to modify the syslinux.cfg file to correct the path and case of the filenames. And for some reason, unetbootin replaced the filename of the disk image from which freedos was supposed to boot with something useless. In any case, after corrections, the entry in syslinux.cfg looks like this:
label ubnentry20
menu label 9201_16e
kernel /BOOT/ISOLINUX/MEMDISK
append initrd=/LSI/IMG/9201_16E.IMG
Unfortunately, the firmware on the 9201_16E.IMG fat filesystem was old, so I loop mounted it and wrote a more recent version of the firmware, mptsas2.rom and sas2flsh.exe to it. I’ve uploaded it to 9201_16E.IMG in case anyone might find it useful. While I’m at it, I suppose I should put the whole usb boot image up for those who might need it. See lsi-flash-usb.img.bz2. To use this disk image, decompress the .bz2 file and dd it to your USB block device, for instance /dev/sdb in my case:
$ bunzip2 /tmp/lsi-flash-usb.img.bz2 | sudo dd of=/dev/sdb
My Proliant DL370 G6 server doesn’t have a fancy new UEFI system, so I used this boot disk to get access to the SAS controller in real mode. Once I booted it and selected 9201_16e (the 21st option from the top), I had to rapidly press the up or down arrow and select the EMM386 option to load the driver. For those playing along at home, I placed the more recent LSI controller firmware and BIOS in the \CJ directory, which has the same structure as the \LSI directory from the original disk image. You should be able to
cd \CJ
and run
9201.bat
Answer the prompts, and afterwards you should have a controller with P20 firmware and BIOS written to it.
Daunting as this was, it only got me most of the way through the process. All of my SATA disks were working at this point, but the SAS disk was not. After some sleuthing, I discovered that the disk was SCSI formatted to 520 byte sectors. Linux does not accept sector sizes that are not a power of 2, and so i was unable to do anything with the disk using fdisk, kpartx, etc. Research pointed me to the sg_format utility, but I was unable to make that work. Here’s the output of the command:
$ sudo sg_format --format --size 512 -vv /dev/sg4
open /dev/sg4 with flags=0x802
inquiry cdb: 12 00 00 00 24 00
SEAGATE DKS2F-H6R0SS 7FA6 peripheral_type: disk [0x0]
PROTECT=0
inquiry cdb: 12 01 00 00 24 00
inquiry cdb: 12 01 80 01 00 00
Unit serial number: Z4D0M2BF0000W515S4WH
inquiry cdb: 12 01 83 01 00 00
LU name: 5000c50062ba7973
mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00
mode sense (10): pass-through requested 252 bytes (data-in) but got 28 bytes
Mode Sense (block descriptor) data, prior to changes:
Mode sense number of blocks maxed out, set longlba
mode sense (10) cdb: 5a 10 01 00 00 00 00 00 fc 00
mode sense (10): pass-through requested 252 bytes (data-in) but got 36 bytes
Mode Sense (block descriptor) data, prior to changes:
<<< longlba flag set (64 bit lba) >>>
Number of blocks=11473076960 [0x2abd942e0]
Block size=520 [0x208]
mode select (10) cdb: 55 11 00 00 00 00 00 00 22 00
mode select (10) parameter list
00 00 00 00 01 00 00 10 00 00 00 00 00 00 00 00
00 00 00 00 00 00 04 00 01 0a 0c 14 ff 00 00 00
05 00
mode select (10):
Descriptor format, current; Sense key: Illegal Request
Additional sense: Parameter list length error
Descriptor type: Sense key specific: Field pointer:
Error in Command: byte 7 bit 7
Descriptor type: Field replaceable unit code: 0x5
Descriptor type: Vendor specific [0x80]
00 00 00 00 00 00 00 00 00 00 00 00 00 00
Raw sense data (in hex):
72 05 1a 00 00 00 00 1c 02 06 00 00 cf 00 07 00
03 02 00 05 80 0e 00 00 00 00 00 00 00 00 00 00
00 00 00 00
MODE SELECT command: Illegal request sense key, apart from Invalid opcode
I was about to give up, but for some reason did not. I tried to find firmware and read any documentation about my disk, a Seagate ST6000NM0024. Eventually I found a github repo called ToolBin. In this repo there is a program named SeaChest_Format_121_1183_64 which seems to use a proprietary SCSI format command to tell the drive to format itself using 512-byte sectors. I can’t say for certain yet that it works, but it did not immediately fail as sg_format did. The command I used (and this will vary for you, depending on what sg_scan -i tells you is your scsi generic device) is:
./SeaChest_Format_121_1183_64 -d /dev/sg5 --fastFormat 1 --formatUnit 512 --confirm I-understand-this-command-will-erase-all-data-on-the-drive
Assuming this command results in success after the 12 hour run, I will purchase another 7 of these drives and fill out my disk array!
Sun Apr 14 13:39:48 PDT 2019
Format Unit Progress = 3.61%
Thanks to the #linux-raid channel for helping me through this!
Update:
Mon Apr 15 08:16:00 PDT 2019
Format Unit Progress = 99.97%
Mon Apr 15 08:17:00 PDT 2019
Format Unit Progress = 100.00%
[70874.677389] sd 3:0:2:0: device_block, handle(0x0012)
[70876.176199] sd 3:0:2:0: device_unblock and setting to running, handle(0x0012)
[70876.249541] mpt2sas_cm0: removing handle(0x0012), sas_addr(0x5000c50062ba7971)
[70876.253344] mpt2sas_cm0: removing : enclosure logical id(0x500062b20059ecc0), slot(1)
[70886.680412] scsi 3:0:3:0: Direct-Access SEAGATE DKS2F-H6R0SS 7FA6 PQ: 0 ANSI: 5
[70886.684166] scsi 3:0:3:0: SSP: handle(0x0012), sas_addr(0x5000c50062ba7971), phy(2), device_name(0x5000c50062ba7970)
[70886.688833] scsi 3:0:3:0: SSP: enclosure_logical_id(0x500062b20059ecc0), slot(1)
[70886.750049] sd 3:0:3:0: Attached scsi generic sg5 type 0
[70886.750143] sd 3:0:3:0: [sdd] Spinning up disk...
[70887.765498] .
[70888.789522] .
[70889.813591] .
[70890.837718] .
[70891.861715] .
[70892.885876] .
[70893.909889] .
[70894.933912] .
[70895.957989] .
[70896.982028] .
[70898.006133] .
[70899.030215] .
[70900.054269] .
[70901.078340] .
[70902.102359] .
[70903.478491] .
[70904.502557] .
[70906.823608] ready
[70906.826165] sd 3:0:3:0: [sdd] 11721045168 512-byte logical blocks: (6.00 TB/5.46 TiB)
[70906.829779] sd 3:0:3:0: [sdd] 4096-byte physical blocks
[70906.834708] sd 3:0:3:0: [sdd] Write Protect is off
[70906.837364] sd 3:0:3:0: [sdd] Mode Sense: 13 00 10 08
[70906.838142] sd 3:0:3:0: [sdd] Write cache: disabled, read cache: enabled, supports DPO and FUA
[70906.867106] sd 3:0:3:0: [sdd] Attached SCSI disk
$ sudo ./SeaChest_Info_141_1183_64 -d /dev/sg5 -i
==========================================================================================
SeaChest_Info - Seagate drive utilities - NVMe Enabled
Copyright (c) 2014-2018 Seagate Technology LLC and/or its Affiliates, All Rights Reserved
SeaChest_Info Version: 1.4.1-1_18_3 X86_64
Build Date: Oct 18 2018
Today: Mon Apr 15 14:52:33 2019
==========================================================================================
/dev/sg5 - DKS2F-H6R0SS - Z4D0M2BF0000W515S4WH - SCSI
Vendor ID: SEAGATE
Model Number: DKS2F-H6R0SS
Serial Number: Z4D0M2BF0000W515S4WH
Firmware Revision: 7FA6
World Wide Name: 5000C50062BA7973
Copyright: Copyright (c) 2014 Seagate All rights reserved
Drive Capacity (TB/TiB): 6.00/5.46
Temperature Data:
Current Temperature (C): 45
Highest Temperature (C): Not Reported
Lowest Temperature (C): Not Reported
Power On Time: 21 days 11 hours 8 minutes
Power On Hours: 515.13
MaxLBA: 11721045167
Native MaxLBA: Not Reported
Logical Sector Size (B): 512
Physical Sector Size (B): 4096
Sector Alignment: 0
Rotation Rate (RPM): 7200
Form Factor (inch): 3.5
Last DST information:
DST has never been run
Long Drive Self Test Time: 10 hours 2 minutes
Interface speed:
Port 0 (Current Port)
Max Speed (GB/s): 12.0
Negotiated Speed (Gb/s): 6.0
Port 1
Max Speed (GB/s): 12.0
Negotiated Speed (Gb/s): Not Reported
Annualized Workload Rate (TB/yr): 0.00
Total Bytes Read (MB): 6.39
Total Bytes Written (MB): 14.99
Encryption Support: Not Supported
Cache Size (MiB): Not Reported
Read Look-Ahead: Not Supported
Write Cache: Disabled
SMART Status: Good
ATA Security Information: Not Supported
Firmware Download Support: Full, Segmented, Deferred
Specifications Supported:
SPC-3
SAM-5
SAS-3
SPL-3
SPC-4
SBC-3
Features Supported:
Application Client Logging
Self Test
Automatic Write Reassignment
Automatic Read Reassignment
EPC
Informational Exceptions [Mode 0]
Translate Address
Format Unit
Sanitize
9 responses to “The woes of 520-byte sectors”
https://forum.level1techs.com/t/how-to-reformat-520-byte-drives-to-512-bytes-usually/133021 says that sg_format fails because the drive is trying to prevent you from losing data, and that if you do an sg_format to 520-byte sectors followed immediately by a second sg_format to 512-byte sectors without writing any data in between, things will work.
Hey there Marius,
Thank you for commenting. Unfortunately, this didn’t work for me. I tried re-formatting at the 520-byte size and hoped after it was done (19 hours later) that I’d be able to format at 512. No such luck; I almost gave up after that failure.
I was so happy to stumble upon the SeaChest tools. It was a life saver!
Cheers,
C.J.
Thank you! sg_disk gave me the same error even after formatting to 520 then 512. It seems the SeaChest tools are working..
Thank you so much for this!
I too ran into the same problem. Used sg_format to try to format to 512, failed with the same message. Took 19 hours to format it to 520, then immediately tried to format to 512 and again same message. I’m trying the SeaChest tool now, and looks like its working. I’ll report back in 12 hours.
Glad to hear it helped Ashley! Looking forward to the report in a few hours.
The problem that most commonly produces this error is that most of these drives seem to have firmware that only uses the six-byte format of the MODE_SENSE SCSI command. The sg_format command uses the 10-byte format by default. All you would have to do to get the sg_format command to work is add “–six”. I’ve literally done this hundreds of times.
I have 6x 6TB Seagate SAS drives (`ST6000NM0034` / `DKS2F-H6R0SS`, `7FA6` firmware) that came to me with 520 byte sectors. I have reformatted them to 512 first with `sg_format` then with the SeaChest tools. I still get `/dev/sdh: close device failed: Input/output error` and `lost async page write` when `fdisk` tries to write a new partition table. `mkfs` straight on the disk fails as well. They are on a Super Micro backplane attached to a pair of LSI SAS9210-8i (BIOS `07.39.00.00` and firmware `20.00.02.00` per `systools`. Any ideas where I should go next?
https://gist.github.com/altendky/96fe6c1f799fdbf79516f9528d6fd644
Hi Kyle,
The first place I’d recommend is the linux-raid mailing list and IRC channel on libera. I can’t think of anything off the top of my head, but there’s likely to be guidance with patience and a post to the ml
Good luck, and maybe I’ll monitor the channel for updates!
C.J.
I did check on IRC and got a few responses but no solutions. I searched around again and ran into https://www.reddit.com/r/homelab/comments/9bu8tf/is_this_drive_actually_bad_or_did_i_screw/ where they noticed they could `sg_write_verify` successfully to the disk and read it back normally. I was able to confirm the same on my drives. Haven’t found any firmware that will load. Now I’m wondering if I can just ask Linux to use the write and verify command for all writes… but I honestly don’t know enough yet to know if that even really makes any sense. `:]`