Linux and Your USB Flash Drive
One day when I was frustrated with Linux, my primary Operating System, I wrote of it, "The Linux Penguin version 4 just doesn't understand nor never really fully understood USB attached storage. The kernel is USB dumb." The truth is that Windows is rather USB dumb also considering that every time I insert the flash drive windows takes time to, according to the message, install device drivers. I just used the flash drive 10 minutes ago, where did the drivers go?
Flash memory: An EPROM, or erasable programmable read-only memory, is a type of computer memory chip that retains its data when its power supply is switched off. From this technology we have the modern solid state storage devices known as flash memory or memory cards, including thumb drives and flash drives.
The Thumb Drive / The Pen Drive / The Flash Drive
Most popular among the types of EPROM removable memory is the thumb drive, a flash drive that you plug directly into the computer USB port. Replacing the floppy disk from a time long ago, the modern thumb drive can hold magnitudes more data. The floppy drive is no longer practical for most applications. A thumb drive can be made to be bootable and is often used to install an operating system on a computer.
Contents
Troubleshooting
Installing Linux from a thumb drive onto a PC is probably the only time that the USB memory seems to function correctly. While working in a Linux desktop environment on a daily basis I have found the use of USB flash memory very problematic. Problems include:
- File copy progress misreported, progress indicator in the file manager (all of them) shows completed in seconds while you wait for many minutes for the process to actually complete and the dialog to go away.
- File drive rarely cleanly ummounts
- Process frequently hangs for over a minute after the i/o activity completes. What is it doing? We have to wait for the dialog to go away even though its done!
- Process fails on copying large files. Files of 1GB or greater often result in the progress interface showing complete but hanging up and not releasing the drive indefinitely. How long? I once let it sit overnight and it was still hung up in the morning. Linux would not even property shut down!
Why is USB flash memory support so flaky in Linux? It is difficult to determine the problem and cooperate with developers on a resolution due to factors including:
- Linux zealots defend the kernel and deny the existence of a problem rather blaming the end user
- Microsoft poorly documents FAT elements such as bit flags relating to corruption of the file system making it difficult for kernel developers to ensure compatibility
- Linux desktop file managers failing to cope with the USB file system and relying on buggy kernel support
- A new industry focus on so called cloud storage has demoted focus on attached / portable mass storage
- Something different happens every time I connect my thumb drive: sometimes the file manger opens, sometimes it asks if I should open it in the file manager, sometimes nothing there is an audio sound, sometimes not --- all with the exact same thumb drive!
There are a number of lovely error messages. I think this one gets to the point...
"The files have been correctly transferred to the USB drive, however the program could not communicate to the kernel that the writing is completed."
Microsoft Windows Says Linux Corrupted My USB Drive
In ordinary cases and under circumstances where the file copy process went (as smooth as possible) on the Linux system, the flash drive being removed and attached to the Windows system may complain that it is necessary to "Repair this drive." Typically the drive does NOT need to be repaired even though Windows complains. The problem has to do with lack of Linux Kernel support for specific flags of a FAT32, FAT16, or NTFS filesystem. The Microsoft people won't provide clear documentation on specific flags and the Linux kernel developers are too stubborn to take the time and properly reverse engineer until full compatibility is realized.
Yes, it is a fact that Microsoft won't share, and even when they do, they provide bad information. However, flags such as ClnShutBitMask and HrdErrBitMask are not that difficult to reverse engineer based on simple observation of behavior. Linux kernel developers are too busy adding candy and crap to the kernel rather than stepping back and fixing long standing irritation problems such as USB drive support.
According to Microsoft:
- ClnShutBitMask: If bit is 1, the volume is “clean”. The volume can be mounted for access. If bit is 0, the volume is “dirty” indicating that a FAT file system driver was unable to dismount the volume properly (during a prior mount operation). The volume contents should be scanned for any damage to file system metadata.
- HrdErrBitMask: If this bit is 1, no disk read/write errors were encountered. If this bit is 0, the file system driver implementation encountered a disk I/O error on the volume the last time it was mounted, which is an indicator that some sectors may have gone bad. The volume contents should be scanned with a disk repair utility that does surface analysis on it looking for new bad sectors.
Bottom line - ignore the error from Microsoft Windows and keep using the drive, your files are probably fine. I say probably because there are a number of other things like can do to butcher your files beyond the kernel not understanding a couple bit flags.
Linux doesn't see my thumb drive
You can refer to USB Device Diagnostics in Linux for help using commands such as lsusb to diagnose why your thumb drive is not coming up when you connect it. These are console commands. There should be a graphical way to accomplish these tasks in Linux, however, that would require someone taking the time to create a good desktop environment tool for Linux users. I am a big fan of the command line, however, I am an anachronism and the modern Linux user demands graphical tools to do these things. Good desktop utilities do not seem to exist!
Warning when attempting to eject or file corruption data loss after drive ejected
This issue is also covered in the next section "Gnome, nautilus copy files to USB stops at 100% or near" because the same issue is at play. You may have just copied a large amount of data to the USB drive as an example, and even though the operating system reports the copy process is complete, and you have closed any file manager or terminal accessing the drive, you are still unable to eject the drive as it reports "busy." This is due to the same kernel issue discussed below. The operating system thinks the copy is complete because the data is actually buffered and not yet written. The stupid linux kernel tells you the data is written while it warns you against unmounting the USB media because it is not actually finished writing to it. Your Linux system just lied to you! Do not eject the media or the data will be corrupted.
Just because your file manager aka Nautilus or whatever you use, tells you the file transfer from your hard disk to the USB device is complete does not really indicate a damn thing, as it is not really complete until the buffer is clear and the kernel releases the lock on the drive. Trust Nautilus and you trash the data on the USB drive.
IT IS GOOD to have one of those USB Flash Drives with the flashing light <- the light that indicates i/o activity! Often the file system reports to you that the file copy is completed, while you observe the flashing light still very active, indicating that your operating system just lied to you.
Gnome, nautilus copy files to USB stops at 100% or near
The recommendations below relate to how much system RAM you have. Some of these fixes I believe are specific to people with 16MB of RAM. If you have a different amount you may have to make adjustments.
source (StackExchange) : http://unix.stackexchange.com/questions/180818/gnome-nautilus-copy-files-to-usb-stops-at-100-or-near
The reason it happens that way is that the program says "write this data" and the linux kernel copies it into a memory buffer that is queued to go to disk, and then says "ok, done". So the program thinks it has copied everything. Then the program closes the file, but suddenly the kernel makes it wait while that buffer is pushed out to disk.
So, unfortunately the program can't tell you how long it will take to flush the buffer because it doesn't know.
If you want to try some power-user tricks, you can reduce the size of the buffer that Linux uses by setting the kernel parameter vm.dirty_bytes to something like 15000000 (15 MB). This means the application can't get more than 15MB ahead of its actual progress. (You can change kernel parameters on the fly with sudo sysctl vm.dirty_bytes=15000000 but making them stay across a reboot requires changing a config file like /etc/sysctl.conf which might be specific to your distro.)
A side effect is that your computer might have lower data-writing throughput with this setting, but on the whole, I find it helpful to see that a program is running a long time while it writes lots of data vs. the confusion of having a program appear to be done with its job but the system lagging badly as the kernel does the actual work. Setting dirty_bytes to a reasonably small value can also help prevent your system from becoming unresponsive when you're low on free memory and run a program that suddenly writes lots of data.
But, don't set it too small! I use 15MB as a rough estimate that the kernel can flush the buffer to a normal hard drive in 1/4 of a second or less. It keeps my system from feeling "laggy".
source (AskUbuntu) : http://askubuntu.com/questions/397249/system-freezes-unresponsive-unusable-when-copying-large-file-to-usb/685293#685293
vm.dirty_background_ratio = 5 vm.dirty_ratio = 10
into /etc/sysctl.conf and running
sudo sysctl -p
source (AskUbuntu) : https://askubuntu.com/questions/734827/transfer-freezes-when-writing-big-files-on-usb-drive
If your running a x64 bit Ubuntu or other Linux and find USB transfers hang at the end apply this fix:
sudo echo $((16*1024*1024)) > /proc/sys/vm/dirty_background_bytes sudo echo $((48*1024*1024)) > /proc/sys/vm/dirty_bytes
I suggest you edit your /etc/rc.local file to make this change persistent across reboots.
sudo vi /etc/rc.local
Go to the bottom of the file and leave a space then paste in those two lines.
Don't like the changes??? To revert the changes enter this in console and remove the lines in /etc/rc.local
sudo echo 0 > /proc/sys/vm/dirty_background_bytes sudo echo 0 > /proc/sys/vm/dirty_bytes
Terminal Mount
First, find the usb drive if it does not automount. (skip ahead to lsblk if you know drive is mounted)
fdisk -l
The command lsblk or LIST BLOCK DEVICES will help you determine if and where your Thumb drive is mounted. On Ubuntu and Mint the system likes to put your flash drive under /media/username - which is kinda lame. We can use grep to cut out clutter:
lsblk|grep media
- note: You won't see the flash drive assignment using this method if the drive is not mounted. As an alternative I like to type: "fdisk -l|grep sd.."
Ah there's our USB thumb drive (based an an actual system example under Linux Mint)
nicole@cat$ lsblk|grep media └─sdc1 8:33 1 14.6G 0 part /media/nicole/STORE N GO
Automount will mount the thumb drive when inserted as long as it has an acceptable recognized file system. You can manually mount it.
udisksctl mount -b /dev/sdc1
When you're done you should make sure you have nothing running in the path of the file drive, including your own console
cd ~
And dismount the thumb drive
udisksctl unmount -b /dev/sdc1
Linux didn't auto mount my FAT32 flash drive
Determine where the flash drive is assigned. Create and choose a mount point and execute in console
mkdir /mnt/flashdrive mount -t msdos /dev/sdc1 /mnt/flashdrive
dstat
This is useful for specifically looking at flash media connected to a USB port. Sometimes I am trying to write data to a flash drive and nothing seems to be happening.
dstat - versatile tool for generating system resource statistics. I can look at where my USB drive is mounted and monitor activity with dstat. Say, for example, my USB flash drive is at /dev/sda2
dstat -D /dev/sda2
will show i/o activity there. The -d and -D are both ways to show disk statistics.
udevadm real-time monitoring of usb ports
To see if a device is plugged into USB or unplugged, real time monitoring showing when a USB device is connected or disconnected.
udevadm monitor --subsystem-match=usb
Does not require sudo
As soon as you connect any USB device, be it a USB flash drive or USB keyboard, mouse, etc details will be displayed. Also when disconnected.
sysstat
Install sysstat. Using iostat from the sysstat package provides a single snapshot of results since startup. Use of the interval parameter will append the results for only the last interval to the output. Run at 1-second intervals with:
iostat -d 1
OR with 0.1-second intervals with:
watch -n 0.1 iostat
nmon
nmon - systems administrator, tuner, benchmark tool. It can display the CPU, memory, network, disks (mini graphs or numbers), file systems, NFS, top processes, resources (Linux version & processors) and on Power micropartition information.
has a nice ascii line drawn interface that does not scroll off your screen.
Partition and Format a USB Flash Drive
This page is part of Linux and Your USB Flash Drive and applies to Solid State Removable Storage. Although the new USB Flash Drive (pendrive pen drive flash stick) that you just took out of the bubble pack is already formatted and ready to use, I would suggest you delete the existing partition, and start over. This is a good idea to protect you from getting a virus. In the past users have had to cope with companies like Sandisk and their U3 Flash Drive Virus. They really did intentionally install a virus on all of the flash drives they sold people! Other times rogue employees have been behind factory new flash media having a virus pre-installed. If you partition and format your flash drive before use then you know what's on it.
You can read about Recommend File System for Linux Use on your USB Flash Drive for some background. NTFS or FAT32 is recommended due to its compatibility with Windows however if you only plan to use the flash drive on linux systems then you can also use EXT4. Also read USB Flash Drive Formats for more information on partition and format options for this type of flash memory.
You will have to insert the flash drive then make sure it is not mounted, if your system uses automount then you will have to manually unmount the flash drive first.
Partition and Format USB Drive via Terminal
If your flash drive already has a partition table and you simply wish to reformat the drive then skip to step #3. In our example we will partition and format a 128GB flash drive attached at sdc.
1. Determine the assignment of your flash drive
fdisk -l|grep sd..
In the absence of having a partition on the flash drive you will observe only the block device node assignment:
Disk /dev/sdc: 119.6 GiB, 128446365696 bytes, 250871808 sectors
2. Partition the flash drive
echo 'start=2048, type=83' | sudo sfdisk /dev/sdc
- remember to replace "sdc" with the assignment of your usb flash drive, which might be something else like sdb sdd etc...
3. Format the flash drive
If you created only a single partition on the flash drive (recommended) then you can assume that will be partition #1 or in our example sdc1
Format the flash drive with the file system type you prefer
- FAT32
sudo mkfs.vfat /dev/sdc1
- NTFS
sudo mkfs.ntfs /dev/sdc1
- EXT4
sudo mkfs.ext4 /dev/sdc1
- exFAT ***
sudo mkfs.exfat /dev/sdc1
- remember to replace "sdc" with the assignment of your usb flash drive, which might be something else like sdb sdd etc...
- *** note on exFAT: requires exfat-utils (sudo apt install exfat-utils) read exFAT on Linux for details. Both exFAT and NTFS are Microsoft proprietary file systems.
Using this method and applying a FAT32 file system is verified to allow you to read and write to the flash drive under Linux and Windows 7. There is a file size limitation with FAT32. You cannot transfer a file that is greater than 4GB in size to a FAT32 file system. Files larger than 4GB can NOT be stored on a FAT32 volume. Formatting the flash drive as exFAT or NTFS will resolve this issue.
Format USB Drive via GParted GUI
If you're a graphical user interface sorta fella then this is an easy way to get the job done.
- GParted -> Device -> Create Partition Table -> Select new partition table type "msdos" - use this even if you plan to make an EXT4 partition and format EXT4
- GParted -> Partition -> New
- Choose Partition type xxxx
- GParted -> Partition -> Format to...
Format USB Drive via USB Stick Formatter
mintstick - A utility included with the Mint Linux distribution that allows for the quick format of a USB flash drive from an easy to use graphical interface within the cinnamon desktop. There is also USB Image Writer as well as USB Stick Formatter. It is located in the MENU under:
- Menu -> Accessories -> USB Stick Formatter
$ /bin/sh /usr/bin/mintstick
Usage: mintstick [--debug] -m [format|iso] : mode (format usb stick or burn iso image)
mintstick [--debug] -m iso [-i|--iso] iso_path
mintstick [--debug] -m format [-u|--usb] usb_device
[-f|--filesystem] filesystem
Linux Mint Package Install - https://community.linuxmint.com/software/view/mintstick
This utility is very easy to operate and works well. You will be prompted for elevated privileges to perform actual format.
Writing an ISO Image to USB Stick
There are a number of Bootable USB flash drive utilities that you can use for this purpose, even if it is not necessary to have a bootable image. Some things to keep in mind include the tip, don't use USB Image Writer to burn a windows ISO image to flash media. For this purpose WoeUSB is recommended. Even the popular Etcher aka balenaEtecher is not recommended for creating bootable USB media from a Windows ISO image.
swappiness and cache
Arn't the dev's cheeky? I wish they would spend more time actually fixing things than making humor.
The kernel divides the memory on your system into a number of “zones” which are related to the address range and what NUMA node they’re associated with. You can view information about the zones in the file /proc/zoneinfo. The kernel does attempt to reserve a certain amount of memory in each zone. You can view the sum of the reserved memory in the file /proc/sys/vm/min_free_kbytes, but you can also view the per-zone statistics in /proc/zoneinfo. The linux kernel tunable parameter vm.swappiness (/proc/sys/vm/swappiness) can be used to define how aggressively memory pages are swapped to disk and also impacts invocation of kswapd to attempt to reclaim memory by killing certain types of processes. Linux moves memory pages that have not been accessed for some time to the swap space even if there is enough free memory available. By changing the percentage in /proc/sys/vm/swappiness you can control the swapping behavior.
A high swappiness value means that the kernel will be more apt to unmap mapped pages. Changing the swappiness value to a low number or zero will do the opposite.
sudo sysctl vm.swappiness=0
This will cause linux to drop the cache before libraries etc. are written to the swap. Linux, like Windows, doesn't do all copying in real time. Instead, they build up a queue of write operations, and then work through the queue in the background so that the actual copy / cut / delete operation can "complete" much faster from the user's perspective of allowing them to continue working with the computer. It is a perception thing. Microsoft started it to trick people. Linux developers spending too much time trying to emulate Windows which makes no sense since we the user are trying to avoid the pitfalls of Microsoft Windows.
swappiness
A value from 0 to 100 which controls the degree to which the system favors anonymous memory or the page cache. A high value improves file-system performance, while aggressively swapping less active processes out of physical memory. A low value avoids swapping processes out of memory, which usually decreases latency, at the cost of I/O performance. The default value is 60.
A swappiness setting of zero means that the disk will be avoided unless absolutely necessary (you run out of memory), while a swappiness setting of 100 means that programs will be swapped to disk almost instantly.
Ubuntu system comes with a default of 60, meaning that the swap file will be used fairly often if the memory usage is around half of my RAM. You can check your own system's swappiness value by running:
cat /proc/sys/vm/swappiness
- dirty_ratio - Defines a percentage value. Writeout of dirty data begins (via pdflush) when dirty data comprises this percentage of total system memory. The default value is 20.
- dirty_background_ratio - Defines a percentage value. Writeout of dirty data begins in the background (via pdflush) when dirty data comprises this percentage of total memory. The default value is 10.
- dirty_expire_centisecs - Specifies the number of centiseconds (hundredths of a second) dirty data remains in the page cache before it is eligible to be written back to disk. You should not mess with this.
- dirty_writeback_centisecs - Specifies the length of the interval between kernel flusher threads waking and writing eligible data to disk, in centiseconds (hundredths of a second). Setting this to 0 disables periodic write behavior.
- drop_caches - Setting this value to 1, 2, or 3 causes the kernel to drop various combinations of page cache and slab cache. (1) will invalidate and free all page cache, (2) will free up all unused slab cache and (3) will free all page cache and slab cache.This is a non-destructive operation. Since dirty objects cannot be freed, running sync before setting this parameter's value is recommended.
Example:
echo 50 > /proc/sys/vm/swappiness
To set this value persistently, you will need to use the sysctl command.
Example:
sysctl vm.swappiness=10
Changes it while the system is running but change is lost on restart.
Example:
sudo sysctl -p
If you are using a solid state drive as your primary operating system drive then swapiness is very important to you. Too much swap will wear out your SSD. See Solid State Drive- Linux for more information.
Has my USB Flash Drive bit the dust?
Discussion on testing to see if a USB stick has gone bad - there is no S.M.A.R.T. data to go on when trying to determine if your USB memory stick bit the dust. We will discuss some ways to test the USB flash memory drive here.
First DETERMINE where you USB Flash is mounted so you don't wipe out your primary drive!!!
I simply use the command "fidsk -l|grep sd.." before I insert the flash and again after. It wont always be the same depending on the system and what other media is connected but as an example my flash is on /dev/sdc
badblocks
This is a data destructive test. Don't do this if you have any plans to attempt data recovery on the drive. This test will help you determine if the USB flash drive can still be used or if it needs to go in the rubbish bin.
- determine where the USB flash is mounted (for this example sdc)
- run the command:
sudo badblocks -w -s -o usbstick.log /dev/sdc
Notice there is no partition number, just the device sdc (not sdc1 or sdc2 etc...)
The badblocks command here will destroy all data on the flash drive. Failures may be due to the flash drive memory controller or capacity has been maxed on space to remap failed blocks which is another way of saying "too many failed blocks." Either of these issues indicates the drive is toast.
The driver descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes
A Parted / GParted error message that is displayed when a failing USB flash drive has been inserted.
GParted is showing a 16MB flash drive is something odd like 58.27GB. Errors pop up when GParted is opened complaining about the driver descriptor.
This could indicate either that (A) your flash drive has a bad memory controller or other physical problem and is toast or (B) it is physically fine however someone used a low-level device tool and mistakenly wrote the incorrect block size onto the drive. You can check if the drive is bad after you attempt to correct the block size. (my examples use sdc but your assignment might be different!)
sudo dd if=/dev/zero of=/dev/sdc bs=2048 count=32 && sync
This will clear the Master Boot Record.
Result on a 16GB drive:
# fdisk -l /dev/sdc Disk /dev/sdc: 14.6 GiB, 15640600576 bytes, 30548048 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes
