[ubuntu-us-ma] 10,000+ files in a folder slows down ...

Daniel Hollocher danielhollocher at gmail.com
Thu Jun 25 18:28:41 BST 2009


I ran into and reported a bug on gedit for large numbers of files.
The problem ended up being the data structure and the sort method.  I
won't go into it, but basically the design was good for folders with
less than 100 files.  There was another bug too, which was multiplying
the slow down, and that bug was easy to fix and was fixed.  The data
structure / sorting issue was not fixed as of jaunty (AFAIK).

So, by my estimates, gedit went from being 10-100 times slower than
nautilus and is now on par with nautilus after the simple bug fix.  I
suspect that nautilus too is using some poor design, since its now
just as fast as gedit.

O well, that was my experience.  Quick answer is use the cli.  Its
much easier to do just what you want and no more.  Using a GUI, all
sorts of extras are done behind the scenes (sometimes that helps, in
this case it hurts).

http://www.commandlinefu.com/ is a cool website that displays
interesting CLI tools, and ways to use them.

Dan

On Thu, Jun 25, 2009 at 11:22 AM, John Abreau<jabr at blu.org> wrote:
> That's really not an Ubuntu issue. A folder with that many files will
> have performance issues on any OS. And not just from a GUI, either;
> you'll have performance issues on a command line, too.
>
> Typically, folders get this large when an automated process keeps
> adding new files to the folder over long periods of time.  When I write
> scripts that need to do this, I avoid the problem by writing the files to
> a subfolder.  My preferred method is to make yearly/daily folders, e.g.
> instead of creating the file
>
>    /path/to/dir/foo12345.jpg
>
> I'll make it
>
>    /path/to/dir/2009/2009-06-25/foo12345.jpg
>
> The process is unlikely to run for 10,000+ years, and each yearly folder
> will have at most 366 daily subfolders.
>
> I've never run into a situation where I expected the process to generate
> 10,000+ files per day, but if the situation ever comes up, I could extend
> the logic to a third level of hourly or minutely subsubfolders.
>
>
> On Thu, Jun 25, 2009 at 8:29 AM, Brian DeLacey<bdelacey at gmail.com> wrote:
>> Does anyone have experience with folders containing lots of files in Ubuntu?
>> Like 10,000+? The files are relatively small - the total disk size is under
>> 1G.
>>
>> Ubuntu's GUI seems to have a difficult time managing folders of this size
>> (eg. attempting to "select all" and copy so many individual/small files to
>> another folder hangs - or takes incredibly long.)
>>
>> Any suggestions on how to manage this large number of small files
>> efficiently?
>>
>> Thanks,
>> Brian
>>
>> --
>> Ubuntu-us-ma mailing list
>> Ubuntu-us-ma at lists.ubuntu.com
>> Modify settings or unsubscribe at:
>> https://lists.ubuntu.com/mailman/listinfo/ubuntu-us-ma
>>
>>
>
>
>
> --
> John Abreau / Executive Director, Boston Linux & Unix
> AIM abreauj / JABBER jabr at jabber.blu.org / YAHOO abreauj / SKYPE zusa_it_mgr
> Email jabr at blu.org / WWW http://www.abreau.net / PGP-Key-ID 0xD5C7B5D9
> PGP-Key-Fingerprint 72 FB 39 4F 3C 3B D6 5B E0 C8 5A 6E F1 2C BE 99
>
> --
> Ubuntu-us-ma mailing list
> Ubuntu-us-ma at lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-us-ma
>



-- 
In science and in mind, the impossible and the hasn't-happened-yet are
indistinguishable.



More information about the Ubuntu-us-ma mailing list