[ubuntu-us-ma] 10,000+ files in a folder slows down ...

[Brian DeLacey]

The default file manager (Nautilus) seems to work reasonably well with as
many as 1,000 files in a folder. That's a pretty easy number to hit when
working with any kind of real-time data capture (e.g. sensor networks) or
component elements of large files (e.g. video frames of video / images.)

Thanks for all the pointers and information. Lots of good Ubuntu
alternatives and choices for solutions!

Brian


On Thu, Jun 25, 2009 at 1:51 PM, James Gray <jamespgray at gmail.com> wrote:

> If you don't find the command line an attractive option there are
> other file managers (instead of nautilus) that you can install which
> may perform faster.  A few searches in synaptic (or google) will turn
> up some alternatives.
>
> On Thu, Jun 25, 2009 at 1:28 PM, Daniel
> Hollocher<danielhollocher at gmail.com> wrote:
> > I ran into and reported a bug on gedit for large numbers of files.
> > The problem ended up being the data structure and the sort method.  I
> > won't go into it, but basically the design was good for folders with
> > less than 100 files.  There was another bug too, which was multiplying
> > the slow down, and that bug was easy to fix and was fixed.  The data
> > structure / sorting issue was not fixed as of jaunty (AFAIK).
> >
> > So, by my estimates, gedit went from being 10-100 times slower than
> > nautilus and is now on par with nautilus after the simple bug fix.  I
> > suspect that nautilus too is using some poor design, since its now
> > just as fast as gedit.
> >
> > O well, that was my experience.  Quick answer is use the cli.  Its
> > much easier to do just what you want and no more.  Using a GUI, all
> > sorts of extras are done behind the scenes (sometimes that helps, in
> > this case it hurts).
> >
> > http://www.commandlinefu.com/ is a cool website that displays
> > interesting CLI tools, and ways to use them.
> >
> > Dan
> >
> > On Thu, Jun 25, 2009 at 11:22 AM, John Abreau<jabr at blu.org> wrote:
> >> That's really not an Ubuntu issue. A folder with that many files will
> >> have performance issues on any OS. And not just from a GUI, either;
> >> you'll have performance issues on a command line, too.
> >>
> >> Typically, folders get this large when an automated process keeps
> >> adding new files to the folder over long periods of time.  When I write
> >> scripts that need to do this, I avoid the problem by writing the files
> to
> >> a subfolder.  My preferred method is to make yearly/daily folders, e.g.
> >> instead of creating the file
> >>
> >>    /path/to/dir/foo12345.jpg
> >>
> >> I'll make it
> >>
> >>    /path/to/dir/2009/2009-06-25/foo12345.jpg
> >>
> >> The process is unlikely to run for 10,000+ years, and each yearly folder
> >> will have at most 366 daily subfolders.
> >>
> >> I've never run into a situation where I expected the process to generate
> >> 10,000+ files per day, but if the situation ever comes up, I could
> extend
> >> the logic to a third level of hourly or minutely subsubfolders.
> >>
> >>
> >> On Thu, Jun 25, 2009 at 8:29 AM, Brian DeLacey<bdelacey at gmail.com>
> wrote:
> >>> Does anyone have experience with folders containing lots of files in
> Ubuntu?
> >>> Like 10,000+? The files are relatively small - the total disk size is
> under
> >>> 1G.
> >>>
> >>> Ubuntu's GUI seems to have a difficult time managing folders of this
> size
> >>> (eg. attempting to "select all" and copy so many individual/small files
> to
> >>> another folder hangs - or takes incredibly long.)
> >>>
> >>> Any suggestions on how to manage this large number of small files
> >>> efficiently?
> >>>
> >>> Thanks,
> >>> Brian
> >>>
> >>> --
> >>> Ubuntu-us-ma mailing list
> >>> Ubuntu-us-ma at lists.ubuntu.com
> >>> Modify settings or unsubscribe at:
> >>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-us-ma
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> John Abreau / Executive Director, Boston Linux & Unix
> >> AIM abreauj / JABBER jabr at jabber.blu.org / YAHOO abreauj / SKYPE
> zusa_it_mgr
> >> Email jabr at blu.org / WWW http://www.abreau.net / PGP-Key-ID 0xD5C7B5D9
> >> PGP-Key-Fingerprint 72 FB 39 4F 3C 3B D6 5B E0 C8 5A 6E F1 2C BE 99
> >>
> >> --
> >> Ubuntu-us-ma mailing list
> >> Ubuntu-us-ma at lists.ubuntu.com
> >> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-us-ma
> >>
> >
> >
> >
> > --
> > In science and in mind, the impossible and the hasn't-happened-yet are
> > indistinguishable.
> >
> > --
> > Ubuntu-us-ma mailing list
> > Ubuntu-us-ma at lists.ubuntu.com
> > Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-us-ma
> >
>
> --
> Ubuntu-us-ma mailing list
> Ubuntu-us-ma at lists.ubuntu.com
> Modify settings or unsubscribe at:
> https://lists.ubuntu.com/mailman/listinfo/ubuntu-us-ma
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lists.ubuntu.com/archives/ubuntu-us-ma/attachments/20090625/87c3c082/attachment.htm