Site Spider

[Paul Lemmons]

-------- Original Message  --------
Subject: Re:Site Spider
From: Ant Cunningham <prodigitalson at vectrbas-d.com>
To: kubuntu-users <kubuntu-users at lists.ubuntu.com>
Date: 01/17/2008 09:24 AM
> On 1/17/08 11:14 AM, "Paul Lemmons" <paul at lemmons.name> wrote:
>
>   
>> -------- Original Message  --------
>> Subject: Site Spider
>> From: Ant Cunningham <prodigitalson at vectrbas-d.com>
>> To: kubuntu-users <kubuntu-users at lists.ubuntu.com>
>> Date: 01/17/2008 07:30 AM
>>
>>     
>>> Does anyone know of any good HTTP spider/ripper utilities that will preserve
>>> file structure and names?
>>>
>>> thanx!
>>>
>>>
>>>
>>>       
>> wget
>>     
>
> granted I havent read the man page for all the options but I assumed wget
> worked similar to curl in that you had to supply each url. I don't know the
> url of every page and I don't feel like writing a spider myself - that's why
> I asked.
>
> will wget spider through the site for me and grab everything - or better
> yet, everything I don't exclude with some option?
>
>
>
>   
Ooops... missed the "exclude" requirement...

wget -rc -X/not/this/dir,/and/not/this/one http://www.your-site.com


-- 
Sometimes I wonder.  Were our faith able to stand upright and look around, would it be looking down at the mustard seed or standing in awe of the height and breadth of it.