From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on polar.synack.me X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00 autolearn=ham autolearn_force=no version=3.4.4 X-Google-Language: ENGLISH,ASCII-7-bit X-Google-Thread: 103376,15267b2c375b45c2 X-Google-Attributes: gid103376,public X-Google-ArrivalTime: 2003-10-24 10:31:26 PST Path: archiver1.google.com!news2.google.com!news.maxwell.syr.edu!wn14feed!wn13feed!worldnet.att.net!204.127.198.203!attbi_feed3!attbi_feed4!attbi.com!attbi_s51.POSTED!not-for-mail From: tmoran@acm.org Newsgroups: comp.lang.ada Subject: Re: Ada Component Registry proposal References: X-Newsreader: Tom's custom newsreader Message-ID: NNTP-Posting-Host: 12.234.124.41 X-Complaints-To: abuse@comcast.net X-Trace: attbi_s51 1067016684 12.234.124.41 (Fri, 24 Oct 2003 17:31:24 GMT) NNTP-Posting-Date: Fri, 24 Oct 2003 17:31:24 GMT Organization: Comcast Online Date: Fri, 24 Oct 2003 17:31:24 GMT Xref: archiver1.google.com comp.lang.ada:1614 Date: 2003-10-24T17:31:24+00:00 List-Id: >You see, ultimately we could do a bot of our own crawling the web as per the >list of URLs tha appear in the XML file and write a log (successful or non >successful) in trying to get to the library's webpage. The library authors www.adapower.com/os/finder.html has an early version of a crawler for finding dead links. The current version also handles robots.txt, cookies, and more. It's available.