Left Arrow Back to Main page of KG

dd_rescue

Data recovery and data protection tool

Like dd, dd_rescue does copy data from one file or block device to another. You can specify file positions (called seek and Skip in dd). There are several differences:

Purpose of dd_rescue

The latter three features make it suitable for rescuing data from a medium with errors, i.e. a hard disk with some bad sectors.
Why? More info can be found in the README file included in the package.

Just one note: It does work. I unfortunately did not just create this program for fun ...

History / News

Latest version is 1.46 (2014-08-09)
ddr_hash now supports calculating HMACS instead of plain hashes. The hash calculation has been cleaned up a bit. When a seed val of 0 is passed on the command line, additional randomness is created using the rdrand() command on x86/x86-64 (if available).
Note that I switched to creating .bz2 instead of .gz tarballs and using my new stronger GnuPG key 2bffc5bf instead of the old 1c98774e. This time, I offer the old .gz with old key in addition, but I won't do this in the future.

(2014-06-27) A vulnerability in most implementations of lzo decompression has been reported. The liblzo2 library (up to and including v 2.06) used by the ddr_lzo plugin (until dd_rescue-1.45) is affected. You need i to feed specially crafted compressed data in blocks of 16MB or larger to the decompressor on 32bit platforms to exploit it, see the report for more details. (This issue has ID LMS-20140616-1/ CVE-2014-4607.)
The man page ddr_lzo advises to be careful when feeding data from untrusted sources to the decompressor; it seems that this advice has been wise. Fortunately, ddr_lzo does not normally feed such large blocks to the decompressor; you'd need to manually increase the soft block size to at least 8MB and ignore a warning to trigger this issue with dd_rescue. But it is possible.
So here's my advice:

Previous version was 1.45 (2014-05-27)
ddr_hash received a bugfix (sha512/sha384 could overflow a buffer). It gained support for sha1 hash. ddr_hash can now conveniently retrieve (and check) and store hashes in xattrs and md5sum/sha256sum/... style files. A new null plugin (ddr_null) was added.

Previous version was 1.44 (2014-05-23)
The plugin libddr_MD5.so (short ddr_MD5) has been renamed to ddr_hash, reflecting that we also support sha1, sha256, sha224, sha512, sha384 now. Checks have been added to the test suite and the documentation been updated accordingly.

Previous version was 1.43 (2014-05-20)
The main feature of 1.43 is the new lzo plugin. It de/compresses data using the lzo algorithms, which are very fast to decompress and most versions are also fast to compress (at somewhat moderate compression levels). The plugin supports many of dd_rescue's features, such as skipping bad blocks (encoding sparseness/holes into the output) as well as appending. It also continues on errors (skipping a whole block if nodiscard is NOT given) and allows to search for valid lzo block headers if sync is lost. fuzz testing has been done to support reliability. A man page ddr_lzo(1) has been created.
The plugin interface has been enhanced to support ddr_lzo; the MD5 plugin has also seen some work beyond just refactoring: It supports the parameter output/outfd= now and supports all type of holes that can be generated in a chain with ddr_lzo now.
Some minor improvements (docu, messages) and bug fixes have been applied. There also is a new ARMv8 (AArch64 aka ARM64) optimized routine to detect zero-blocks.

Previous version was 1.42.1 (2014-03-06)
1.42.1 contains a fix for a sublety how we set up a handler for SIGILL and return with longjmp to detect the supported instruction sets of the CPU -- we need to manually reset the process' signal mask, otherwise a second failed probe would abort.

Previous version is 1.42 (2014-03-03)
1.42 brings the possibility to load plugins to analyze or transform data before it's written to the output file(s). A plugin to calculate the MD5 hash is provided. posix_fadvise() is used if available (optimization) and dd_rescue now only provides a short usage info rather than the long help text on wrong parameters.

Previous version was 1.41 (2014-02-25)
There has been a lot of internal refactoring that improves the detection of CPU features (at runtime) and libc/compiler features (at build time). One result is that this version supports building against the Android NDK. (armv7l binaries built against Android API 17 (aka 4.2) libc can be found below in the download section.) Another consequence is that AVX2 support is now enabled (for saving CPU cycles on sparse block detection). A few minor bugs have been addressed (the most serious one a harmless off-by-one on determining the size of a block device). Number formatting is more consistent now. There also a new option -u/--rmvtrim that deletes the created file again and issues a fstrim on the filesystem -- good if you filled the empty space of a filesystem with zeros for data protection and SSD refreshment.

Previous version was 1.40.1 (2014-02-15)
It just has one patch to fix the SSE2 detection on i386 -- the old code would end in an endless loop ...

Previous version was 1.40 (2013-08-18)
It brings copying of extended attributes (with -p/--preserve). It doubles the default soft block size for buffered IO, but brings sparse write optimization for half-empty blocks. It also optimizes copying by using the first write to get rid off odd file offsets. It also adds a lot more test cases to make check.

Previous version was 1.39 (2013-08-08)
It fixes an issue where a copied file could be appended zeros if hardblocksize copy was used (e.g. b/c hardbs==softbs, bnc #833765). There's also a bit better ARM asm optimization, yielding a ~15% performance increase. There's also a help/manpage clarification that syncfreq actually is a size. And we use autoconf now to determine the target system features. Default build target now uses libdl.

Previous version was 1.38 (2013-08-02); improving SSE sparse detection performance (by 40%), adding a testcase for the 1.35/1.36 bug and run it in make check. There's even an AVX version, but it's not enabled by default, as it's untested. --force/-f now allows to ignore a non-zero output position on non-seekable output and the curr.rate and ETA calculations have improved a bit.

Version 1.37 (2013-08-01) was fixing an issue with SSE2 sparse detection, which could spuriously detect zero-filled blocks and thus result in corrupted copies if option -a was used. (This would happen for blocks that had no bytes with the uppermost bit set, such as e.g. ASCII text.) Embarassing! Also fixed issues on big-endian machines (although these were inconsequential for dd_rescue).

Older version history ...

The previous version is 1.36, released 2013-07-24.
It fixes an overflow issue with the number output for long running dd_rescue processes. SSE2 is now also enabled in x86 (32bit, with runtime detection) and an optimized ARM version (assembler yeah!) to find zero blocks was added.
The previous version was 1.35, released 2013-07-17.
It had some improvements on the output that it prints -- beyond internal improvements it introduces colours to the output unless the terminal type is clearly dumb; there is also an option to control this. Numbers are highlighted for readability. Output is rate limited (10/s). 1.35 also brings a simple rewrite logic for handling write errors. There's an SSE2 optimized version to find zero blocks for sparse writing.
The previous version was 1.34, released 2013-07-04, It brought better support for various *nix systems (specifically had a few fixes for FreeBSD), better compatibility with compilers (clang and g++ and clang++). It can now also load libfallocate at runtime (libdl) and detects a few more fatl write errors as such.
Previous version was 1.33 (2013-03-30) and brought long options, a new double overwrite mode (-2) and a man page.
Version 1.32 has a new option -x to append to the output file and you can specify -Y (multiple times if you wish so) to write the same data to secondary output files.
Version 1.31 (2013-02-03) brought a few tiny improvements in the output (such as displaying the total elapsed time in the summary as opposed to ETA of 0, and the amount of data really written with option -W). But importantly, it has the new mode of triple overwriting of data (options -3 and -4), with random numbers, inverse random numbers, new random numbers (only for -4) and zeros, this way allowing paranoia-safe deletion of information.
Version 1.30 (2013-01-25) brought a fix for outputting data to stdout and a fix for a possible double free operation (introduced in 1.29). The message formatting has been streamlined a bit. The PRNG can now be initialized from a file (e.g. -Z /dev/urandom). The program now can also avoid writing to a target block if the target block already has the same data (option -W). Think of SSDs or other devices where you want to avoid writes.
In Version 1.29 (2013-01-22) a bug was fixed, where the last bytes where not copied corrected if hardbs == softbs. 1.29 also brings a number of new features; the ability to write the same (softbs sized) block again and again (option -R, automatically set if infile is /dev/zero), the ability to limit transfer size such that the outfile won't be enlarged (-M) and the possibility to use userspace random numbers (libc/frandom) to fill files with random data (options -z and -Z). Last not least, OBS also builds .deb binaries for Ubu12.04 / Deb6 now.
Version is 1.28 (released 2012-05-19) uses better defaults for hard and softblock sizes (4k/64k for buffered I/O, 512/1M for direct IO), as suggested by Jan Kara. Also the copying of access times with the option -p was fixed.
Version 1.27 allowed to do 512b direct IO (which is possible in latest Linux kernels) -- idea and patch from Jan Kara. Change posix_memalign() variable assignment. It has a number of fixes from Valentin Lab; most importantly, when exiting because of an error, it updates the variables that are output. dd_rescue now avoid specal characters in the logfile. It handles situations gracefully, where wrong positions resulted in the progress graph causing faults. Some come from illegal input (negative offset ...), which is nw detected.
Version 1.25 contains a fix for spurious "Success" messages that resulted from overwritten (cleared) errno. Bad blocks are formatted in a way that they are not overwritten on screen and block numbers are output as unsigned.
Version 1.24 contains a compile fix for Linux versions that contain the splice syscall but not the other definitions. I also allows for specifying a directory (such as ".") as output filename in which case dd_rescue just appends the input file basename to it, just like cp does. Maybe most importantly, the RPM now contains the latest version of dd_rhelp (0.1.2).
In version 1.23, the default to not sync every 512 soft blocks is now really changed to only sync at the end (documented for 1.18, but mostly ineffective, ouch). More importantly, a short read because of EOF is now not considered an error any more and thus neither triggers messages (with quiet) nor does it cause a confusing exit code any more.
Version 1.22 contains a number of little improvements: Display of compiler, compile time, enabled options, don't repeat copy attempts if hardbs == softbs, fix FPE in progress bar for non-seekable input reverse copy, and make the code more digestable for non-GCC compilers, optional fallbacks for strsignal, pread, pwrite. Last not least, the documentation has a few notes on the goodies now.
Version 1.21 issues warnings when writing in a sparse mode into existing files or to block devices.
In version 1.20, there's also a progress estimation if input file size is unknown, but transfer size is limited with -m.
Version 1.19 brings the support for fallocate.
In version 1.18, we have changed the default to -y0 (no fsync), the avg. speed is calculated in a meaningful way nevertheless now.
In version 1.17, a progress bar, completion percentage and ETA has been added.
In version 1.16, the work was done to determine and display the total amount of data that will be copied.
Version 1.15 brings support for the Linux splice() mechanism to avoid copying data to a userspace bugffer.
In version 1.14, a bug is fixed that could lead to a target file not having the correct size when copied in sparse (-a) mode.
In version 1.13, a bug with read-errors and non-seekable output has been fixed.
In version 1.12, the ability to log bad blocks to a ASCII file has been added. The option -y can be used to set the fsync frequency which can speed up the copy significantly.
Since 1.10, it has support for non-seekable input and output (so you can write to pipes, e.g. stdout). Since 1.04, dd_rescue uses a larger default soft block size (64k instead of 16k), has support for O_DIRECT, for copying the permissions, and some minor issues fixed.
RPMs since 1.04 also include dd_rhelp.
Since version 1.03, it includes a patch from LAB Valentin which is needed to properly support his dd_rhelp script from his dd_rescue page.
See the RPM change log for a more detailed list of changes.

Roadmap

Here's an unsorted list of things I'd like to do to dd_rescue when I find some time ... I'm open to more suggestions.
I'm even more open to contributions -- please let me know if you intend to work on improving dd_resuce. I'm happy to push the private CVS to some public place (probably converting it to HG along the way) and to provide some guidance on e.g. the plugin API.

Downloads

Please use the RPM and DEB repository public key to validate the RPMs/DEBs. Import it if you trust the packages from my OBS repositories. You can validate the other files (android binaries, source tarballs) using my public gpg keys (fingerprint 94DD 1125 CDFB AB48 6573 28EF C662 E1EA 1C98 774E). Starting from July 2014 (and dd_rescue version 1.46), I started creating .tar.bz2 instead of .tar.gz and using my new, stronger key with fingerprint 6669 F734 0D31 E95E C556 5490 DE4F 1B3A 2BFF C5BF for the .bz2 tarballs and Android binaries. (Note: It's better to get my keys from public key servers ... if someone can attack this site to put bad files up here, s/he will also change the keys and fingerprints ...)

You can download

Version 1.46
Version 1.45
Version 1.44
Version 1.43
Version 1.42.1
Version 1.42
Version 1.41
Version 1.40.1
Version 1.40
Version 1.39
Version 1.38
Version 1.37
Version 1.36
Version 1.35
Version 1.34
Version 1.33
Version 1.32
Version 1.31
Version 1.30
Version 1.29
Version 1.28
Version 1.27
Version 1.25
Version 1.24
Version 1.23
Version 1.22
Version 1.21
Version 1.20
Version 1.18
Version 1.15
Version 1.14
Version 1.13
Version 1.12
Version 1.11
Older versions
Installation and compilation follows the standard ways. If you use Linux, using rpm --recompile on the source RPM may be your best option.

Copyright

The program is protected by the GNU GPL (v2 or v3).
(w) by KG, last changed 2014-08-09