Table of Contents
- 1. Preamble
- 2. Setup and Procedure
- 3. User's Guide
- 3.1. Overview
- 3.2. Detailed Instructions
- 3.2.1. Step 1 Boot the USB Drive
- 3.2.2. Step 2 Gather information about the system
- 3.2.3. Step 3 Connect to the internet
- 3.2.4. Step 4 Test the CPU
- 3.2.5. Step 5 Gather disk information
- 3.2.6. Step 6 Test disks
- 3.2.7. Step 7 Sanitize disks
- 3.2.8. 8. Test LCD and keyboard
- 3.2.9. 9. Submit report to database
- 4. Appendices
1 Preamble
The triage software testing suite is a collection of utilities and scripts collected at FAMCe HQ over the past three years which aids and largely automates the process of testing and documenting x64 computer hardware. The software tests all major components of a computer system including the CPU, RAM, Storage Media, Display, Input Devices such as keyboards and mice, as well as network interfaces. This testing process is documented via an offsite SQL server administrated by FAMCe staff, as the SQL server is critical to the documentation step of the testing process internet access will be required by all units which undergo testing via the triage system. Familiarity with Linux based operating systems, in particular Arch Linux upon which the system is constructed will assist greatly in troubleshooting any issues which may arise in operation of the testing software. The system has undergone extensive testing and debugging, but edge cases do still arise from time to time. In the event that a system which causes the testing software to malfunction is discovered please submit a detailed bug report and I'll try to fix it as soon as possible.
2 Setup and Procedure
The triage software suite ships burned onto USB flash drives. The creation and use of these drives is detailed in later sections. Technicians who use the software must at a minimum be familiar with configuration of UEFI firmware interfaces, as well as Legacy BIOS configuration more broadly. Exhaustive documentation of the steps required to configure a system for triage support would be impossible. EFI firmware interfaces and BIOS configuration utilities are far too varied, with each manufacturer's chosen interface having their own quirks; not to mention the overall loose standardization of the EFI specification imposed by Microsoft. Technicians will have to rely on their own experience and knowledge in order to correctly configure the system for use with triage. Note that in all cases Secure Boot must be disabled and USB boot support must be enabled.
3 User's Guide
The operation and use of a triage drive is fairly straightforward. The unit must be configured such that USB boot option roms are enabled, and Secure Boot if applicable is disabled. Once the flash drive is inserted and booted the technician must simply follow the onscreen instructions. NOTE: Units without storage media will only take a few seconds Units with SSDs may take several minutes Units with HDDs may take 1/2 an hour or more
3.1 Overview
The testing process proceeds roughly as follows
- Boot the USB drive
- Gather information about the system
- Connect to the internet
- Test the CPU
- Gather disk information
- Test disks
- Sanitize disks
- Test LCD and keyboard
- Submit report to database
Minimal involvement from the technician is required at any of these steps save for the LCD and keyboard testing, and the only one which will consistently generate errors of any kind is disk sanitization. This is due to restrictions and caveats inherent to the SATA command set, alternative methods of logical sanitization are only available for mechanical media. See NIST Special Publication 800-88 Rev. 1 for more information regarding the methods available.
3.2 Detailed Instructions
3.2.1 Step 1 Boot the USB Drive
As stated above it wouldn't be possible to write a comprehensive guide for UEFI/BIOS configuration which would allow for Linux based media to boot. In all cases however Secure boot must be disabled USB boot must be enabled The technician will have to rely on their own experience and judgment for information on how to accomplish this. It will be different on a system-by-system basis.
3.2.2 Step 2 Gather information about the system
This process is entirely automated and the technician will only be notified if something goes wrong.
3.2.3 Step 3 Connect to the internet
This is the first section which requires any manual intervention on the part of the user. Triage will first attempt to detect if the system has any wireless hardware installed, the procedure for wireless devices vs hardwired ethernet is different and will be detailed separately.
- Wireless
If a wireless card is detected the system will place the NIC in scanning mode and the unit will attempt to search for available wireless networks. This is not a foolproof process and a non exhaustive list of what may go wrong is detailed below
- Wireless connection procedure
Assuming the card is activated and functioning as expected the iwd daemon and iwctl interface will be shown on screen. The wireless NIC is placed into a 'scanning' mode to locate and display all available wireless access points. In order to continue the system in use must be connected to the internet. Simply type out the SSID of the local wireless network, followed by the password. Triage flash drives will remember the passwords of networks which they have connected to previously so only the SSID will be necessary after the first time a given drive has been used
- Troubleshooting
If the SSID has been input but the device still refuses to connect (simply dumping one back to the SSID prompt), the appropriate response is to type 'forget' in the input field, followed by the SSID. This will clear the network configuration from memory. Press Ctrl-C to exit the triage script, then restart the shell. There are a number of ways to do this but I prefer typing 'bash' at the command prompt, at which point the .bashrc configuration will take over and the script will be restarted. Attempt to reconnect to the network, if it still fails the same way the wireless NIC may be damaged or defective.
- RFKILL is enabled
rfkill (radio frequency killswitch) is a technology mandated by the FCC for portable electronic devices capable of wireless radio transmission. In this event the killswitch has been enabled and you'll need to disable it. This may be a literal physical switch that has to be toggled as it is on many Toughbooks or a logical killswitch controlled by a hardware shortcut, in this event you should look for a wireless communications icon on the keyboard, usually alongside the 'F' keys. It may look like a wireless antenna or a wifi-radio symbol, occasionally it may also appear as an airplane. Once the killswitch has been disabled you may proceed as normal.
- Lack of driver support
The mainline linux kernel ships with literally thousands of kernel modules for as many drivers as the community can reasonably supply. Occasionally however a piece of hardware slips through the cracks, and these pieces can be split into three broad categories
- Older Broadcom chips
For whatever reason it's particularly difficult to maintain and support these chips and as a result they've largely been left out of kernel support. These are fortunately quite rare and you should only run into one on rare occasions, unless you're dealing with mid-2010's apple macbooks. I advise either replacing the NIC, or removing it and doing the testing via ethernet. A manual fallback mode will be added in an upcoming version of the software (3.16.4 at time of writing)
- Apple macbooks
Apple loves their proprietary interfaces, and there are many unsupported cards built in to their macbook line of machines. While 3rd-party drivers are available these are often unstable and prone to causing Kernel Panics, the Linux equivalent of a blue screen of death. Avoid running large numbers of these machines but if you must do so and the chips aren't supported out of the box, support can be installed to each flash drive; though this may compromise system stability. Remove damaged or defective wireless cards when appropriate
- Brand new cards without support
I seriously doubt you'll ever encounter one of these but it is possible that driver support for a given which has just hit the consumer market may not yet have builtin support. We run a rolling release distribution with bleeding-edge support for this sort of hardware, but the core system itself is only updated once every few months barring manual intervention on the part of the technicians. In the event this occurs, or is believed to have occurred see the Administrator's Guide for instructions on updating the triage drives.
- Older Broadcom chips
- RFKILL is enabled
- Troubleshooting
- Ethernet
Ethernet configuration is, particularly in comparison to the fragile methods of wireless connection, very simple by comparison (though it is slightly less robust as you shall see shortly). The system polls the kernel device list for ethernet devices and attempts to connect using the first one that it finds. If a cable is not connected the user will be notified about a damaged or defective ethernet port and instructed to inspect the connection before pressing return. Connections are automatic and mostly seamless, so long as the network the systems are connecting to has a functioning DHCP server. Please note in the event that a given unit (typically a desktop) has multiple ethernet ports the system will only be listening on the first one detected. Due to the nature of internal device representation the selection of which port to use is effectively random. If a device is connected with link lights active and the technician is still receiving warnings about a damaged or defective port please check for additional ports either added via USB interface devices or expansion cards connected via PCI.
3.2.4 Step 4 Test the CPU
This process happens automatically in the background. All CPU cores will be stressed to their 100% maximum capacity for a few seconds to ensure system stability under heavy load.
3.2.5 Step 5 Gather disk information
Much like system info in step 2, the only time a technician would need to know about this function is when something has gone wrong.
3.2.6 Step 6 Test disks
Again an automated process, this time the parallelized SMART testing of internal storage media. Non-fatal SMART codes are logged as an 8-bit mask detailed in an appendix
3.2.7 Step 7 Sanitize disks
This is where the real fun starts. By far the most complicated and intricate section of the entire triage program suite, this is also the most important in terms of data security and the one where things are most likely to go wrong. Due to the complexity of the task at hand both the wiping process itself and the guide must be broken down into phases.
- Phase 1: Preparation
Technically speaking this phase was already started way back in step 5, but to summarize a lot of jargon in order to correctly wipe these disks the system or technician were this done manually has to understand a LOT about their hardware; that information has already been gathered and stored in an internal hash table. All the technician has to worry about at this phase is if the disk is mechanical or not, as mechanical Hard Disk Drives can take hours to properly sanitize. Accordingly in the event a mechanical HDD is detected the user is warned ahead of time. You may wish to stop the process at this point and set this computer aside for later processing, or even remove the HDD and proceed as normal. In either case the technician must respond by stating 'y' or 'n' respectively and the wipe will continue.
- Phase 2: Security (ATA Only)
This process has been largely automated as well, but it serves us well to cover it in detail here as it allows for much more effective troubleshooting in the event that any step of this process fails to work as anticipated. The ATA command set has a robust and flexible suite for internally managed security of ATA devices, this system is also an ineffective mess that really only gets in the way of what we're trying to do but we have to work within its boundaries anyway. In order for us to actually sanitize the drives we need access to the full ATA command set, and in order for us to get that access we have to disable one security feature and enable another.
- Thawing disks
When disks are first spun up on system boot they initialize themselves in a 'Frozen' state, this is to preclude the possibility of a boot-sector virus or compromised firmware from locking the disks or otherwise causing them to behave in a self-destructive manner. This security feature is entirely pointless and serves to do nothing but make the exact job we are attempting to do much more complicated. As we are about to demonstrate it's trivial for a piece of code, malicious or otherwise, to bypass this "security feature". The official method is to power-cycle the drive, physically, by disconnecting it from the system power supply unit and then reconnecting it. Upon being connected to a system already in operation the drive will be "Unfrozen" and ready to respond to commands from the user. What we do is a bit different, by happy coincidence placing the computer system into an S2 state (deep sleep) the internal power supply will temporarily stop supplying electricity to the disk, and once the system returns from S2 the drives will likely be unfrozen. This is not 100% effective, but is effective enough to be used as standard practice. I don't have recorded metrics but I'd give this step a 98% success rate.
- Locking the disk
The method we use for sanitizing SATA media is the 'Security Erase Unit' command detailed in the ATA specification published by the T3 workgroup. However 'Security Erase Unit' (heretofore referred to as Secure Erase) requires authentication from a password previously set by the user. Now there is no password set on 99% of SATA drives out there, so we need to set one. Attempting to do so will put the drive in a 'Locked' state, effectively rendering unusable without said password, but the password will be disabled again by the end of this process. (Sidenote: We always use the same password, it's just 'geek' all lower case) Attempting to set this password may not work if there has already been a password set, in which case unless that password is known to the user and can be manually disabled this drive is effectively bricked and needs to be disposed of. Once the password has been set the process can proceed as normal.
- Thawing disks
- Phase 3A: Secure Erase (ATA Only)
At this point our drive wiping utility hdparm is invoked, granted our password set in Phase 2 and the process begins properly. What the drives do specifically and how they do so is entirely up to manufacturer specification (see NIST SP 800-88 Rev. 1) but the internal mechanism is out of our hands. Here are some caveats to keep in mind however. Firstly the drives are going to be entirely unresponsive during this period, if a technician attempted to use hdparm to check disk information or SMART info during this time, the checking process would hang. This is expected behavior. While you can be patient and wait it out though a USB disk controller will interpret an unresponsive disk as non-functional and power cycle the device in an attempt to bring it back to functionality. This will permanently brick the device and should be avoided at all costs. The reasons behind this are complex but in a nutshell once the Secure Erase command has been sent the drive blanks out its internal password buffer, but doesn't unlock until the process has been completed. Meaning you now have a locked drive with no password set. No password means we can't use a password to unlock the drive and it's been permanently sealed. This is intended behavior, its to keep disks which failed to securely erase due to a power failure from being reused, but its also a massive pain in the neck for us who have to deal with the fallout.
Do NOT run Secure Erase on a disk connected via USB EVER, At All, You will break the disk. Connect the drive directly to an ATA interface or face the consequences, You have been warned.
In the event that a disk fails to unfreeze this is usually why. You'll likely get a scary looking error message about an SG/IO error, this doesn't mean the drive is broken, but it does mean the drive is Frozen, and a drive which failed to unfreeze is usually broken. This also doesn't mean that you were the one who broke the drive but someone did and now you have to remove it.
- Phase 3B: NVMe Sanitize (NVMe Only)
A much cleaner and far less error prone sanitization method is that provided by nvme-cli. I don't fully understand this one, it's been convenient and reliable enough that I haven't had to build up the same level of familiarity as I have with ATA. Basically the process is issue an nvme-format command with the –ses flag set to 1, and the disks internal electronics take care of the wiping for us. I've only seen it fail once and the disk in question was dead on arrival anyway.
- Phase 4: Cleanup and Verification
The disk will quickly be checked for remaining partition structures before being given a thorough check with our homebrew sampler utility (Note that Sampler was disabled for the 3.0 release-candidate version of triage and may be skipped depending on the version you happen to be running)
3.2.8 8. Test LCD and keyboard
After the disk has been verified our in-house LCD/Keyboard testing utility will be launched. endscreen is the absolute latest in hacked-together late 80's terminal garbage, pressing keys will echo them to the display. Pressing the Tab Key will cause the display to cycle between eight different colors allowing the technician to examine the screen for defects. The mouse cursor will also be enabled at this point so that trackpads can be tested.
3.2.9 9. Submit report to database
Home stretch. The system is going to ask the user to input a cosmetic grade (see relevant R2 documentation for full details on the grading system) as well as asking for additional information about the device which may be relevant (such as cracked corners, missing keys, etc). This information will be included in the final report generated by the server so be thorough. Pressing the enter key will exit the prompt and submit the information to the central server.
4 Appendices
4.1 SMART return codes
Excerpted from the smartctl manual page: The exit statuses of smartctl are defined by a bitmask. If all is well with the disk, the exit status (return value) of smartctl is 0 (all bits turned off). If a problem occurs, or an error, potential error, or fault is detected, then a non-zero status is returned. In this case, the eight different bits in the exit status have the following meanings for ATA disks; some of these values may also be returned for SCSI disks.
Bit 0: Command line did not parse.
Bit 1: Device open failed, device did not return an IDENTIFY DEVICE structure, or device is in a low-power mode (see '-n' option above).
Bit 2: Some SMART or other ATA command to the disk failed, or there was a checksum error in a SMART data structure (see '-b' option above).
Bit 3: SMART status check returned "DISK FAILING".
Bit 4: We found prefail Attributes <= threshold.
Bit 5: SMART status check returned "DISK OK" but we found that some (usage or prefail) Attributes have been <= threshold at some time in the past.
Bit 6: The device error log contains records of errors.
Bit 7: The device self-test log contains records of errors. [ATA only] Failed self-tests outdated by a newer successful extended self- test are ignored.
4.1.1 Example
A smart mask of 80 would mean bits 4 and 6 were both enabled, meaning in turn that the disk is not currently failing, but may begin to do so soon (Bit 4), and that there had been error messages logged during device operation prior to this point (Bit 6).