How to transfer directory-structure diffs to multiple machines

[Luis M]

Well, it certainly seems that you have the problem solved. For as long
as your solution "works" there is no real need to change it... unless
you run into performance/scalability issues.

I'd suggest you take a look into nfsv4. You might simplify your
process like that, but, perhaps a the expense of some bandwith wasted.
Now, if you pair nfsv4 and rsync, you might have something going on.

Say your "master" filesystem is mounted over nfsv4
(secured/encrypted/kerberos/whatever) to the clients. The clients (e/a
in particular), run some cron that essentially rsync's files that
should be kept local (the ones that are used most often) and the rest
(the ones that are not accessed as frequently) could be called
directly from the nfs mounted disk. In short, something like autofs
works:

ls /mnt/auto/non-existing-share/file1

As soon as the kernel sees that you want this file, the share is
mounted for you by autofs and you can read it. The connection then is
dropped in 60 seconds or whatever you choose. you could implement
something like that that would try to open the files from the local
drive if they exists, or fallback to the remote, autofs-mounted disk,
if they don't -- and keep statistics of how many times a file was
attempted and add that to the nightly cron job of files to be
rsync()'ed.

The only thing is that you might have to modify libc6 to implement
this ... ;-) The beauty of open source.

Just a **silly** suggestion...

On 3/18/06, John Schofield <schof at dakim.com> wrote:
> I've been Googling, and haven't been able to come up with a pre-made
> solution for this particular issue.
>
> Here's the problem:
>
> I have a directory structure full of many thousands of files. This
> directory structure is updated on a regular basis. Many of these
> files are audio/video clips. The entire directory structure will
> eventually total several hundred gigs.
>
> I want to mirror this directory structure to a large number of
> computers running Ubuntu. These computers will be outside my local
> network, so the data will travel over the Internet.
>
> I could, of course, simply use rsync from each of the machines to the
> central server. Because of bandwidth concerns, I'm not too crazy
> about this approach.
>
> So I've come up with a hybrid approach.
>
> I have a "deployed" and a "staging" directory structure on my server.
> We make additions to the staging directory structure as they become
> ready. When we're ready to deploy the changes, we do so using the
> procedure outlined below, and then the current staging directory
> becomes the deployed directory. New changes are made to the staging
> directory.
>
> So the procedure I've roughed out is:
>
> 1) We do a diff on the two directory structures, generating a list of
> changes -- based on this list, we generate a collection of new files,
> and a list of deletions and tar/bzip it up.
>
> 2) We distribute the tar archive to the various client machines using
> scripted BitTorrent.
>
> 3) Each client machine untars the archive and follows the
> instructions in it to add new files, delete old files, etc.
>
>
> Looks like "diff -r --brief dir1 dir2" will get me a list of changes
> between two folder structures. (dircmp will do the same, but it
> doesn't seem to be part of Ubuntu or Debian.) I could write a shell
> or perl script to process those diff results, and generate the
> archive file. (And unpack the archive file on the other side.)
>
> So, my questions basically boil down to the following:
>
> Am I smoking crack? Does this seem a reasonable way to solve the
> problems I've outlined? Is there an easier way to do this? A utility
> I could use that already exists?
>
> Thanks very much!
>
>
> --
> ubuntu-users mailing list
> ubuntu-users at lists.ubuntu.com
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-users
>


--
----)(-----
Luis Mondesi
System Administrator
Kiskeyix.org

"We think basically you watch television to turn your brain off, and
you work on your computer when you want to turn your brain on" --
Steve Jobs in an interview for MacWorld Magazine 2004-Feb

No .doc: http://www.gnu.org/philosophy/no-word-attachments.es.html