Is a tool available to check the integrity of copied files?

[Robert Heller]

At Fri, 14 Apr 2023 23:15:25 +0200 "Ubuntu user technical support,? not for general discussions" <ubuntu-users at lists.ubuntu.com> wrote:

> 
> Hi,
> 
> I've done some googling for "does linux diff compare data using a cache"
> to figure out what's going on. The first diff of 10 GiB of
> data copied from a SATA3 SSD to an USB 2 stick connected to an USB 3
> port took around a minute, right after the copy finished. A second diff
> needed 3 seconds. Both returned exit status 0.
> 
> It's impossible to read 10 GiB of data in 3 seconds from an USB 2 stick.
> Does diff use cached data instead of comparing the "real" files line by
> line?

Linux file I/O will always use whatever is in the buffer cache.  I suspect 
that (modern) MS-Windows and BSD (read: MacOSX) do the same.

> 
> Google returned "diff isn't doing any caching. The OS is. If you are
> using Linux, you can flush the disk buffers and cache".

Yes, this is correct.

> 
> I expected that diff ensures to compare the "real" files line by line,
> but seemingly diff isn't aimed to check integrity of data.

Diff is just looking for differences in *files*.

> 
> Does a command exist that compares "real" files, not just cached files
> by default?

Not likely.

> 
> I experience weird things with Raptor Lake hardware, especially if USB
> is involved and I want to check the integrity of USB transferred, saved
> files by using a tool, without manually clearing cached data manually.

You really can't do that with *files*, since *files* imply file I/O, which 
always use the buffer cache.  ALL *modern* O/Ses do that for performance 
reasons.  What you *can* do is go below the file system and compare the 
underlying hardware, using a raw "disk" compare or check tool. 

There are some "easy" tricks.  Use dd to read back the whole USB drive (or 
just a partition) to an image file on a known good disk (maybe even on a known 
good computer system).  Then mount the disk image and compare the files there.

Or create a known good disk image and use dd to copy to the USB drive and then 
read it back and compare.

There might be tools that can compare a raw disk with a test image.  And tools 
that can test disk blocks (mkfs.ext[234] can do that).

> 
> Usually I don't sent the same request to more than one mailing list, but
> this time I urgently need help, because I run from one into another
> issue with new hardware. I've made harmless careless mistakes that were
> time-consuming because I'm overworked. Maybe I can't see the forest for
> the trees.
> 
> Regards,
> Ralf
> 

-- 
Robert Heller             -- Cell: 413-658-7953 GV: 978-633-5364
Deepwoods Software        -- Custom Software Services
http://www.deepsoft.com/  -- Linux Administration Services
heller at deepsoft.com       -- Webhosting Services