From: Lawrence D'Oliveiro <ldo@nz.invalid>
Subject: Re: "Red" And The DoD Language Competition
Date: Thu, 12 Sep 2024 22:27:54 -0000 (UTC) [thread overview]
Message-ID: <vbvpta$esm6$8@dont-email.me> (raw)
In-Reply-To: vbtsbq$38je$1@dont-email.me
On Wed, 11 Sep 2024 23:57:33 -0500, Randy Brukardt wrote:
> "Lawrence D'Oliveiro" <ldo@nz.invalid> wrote in message
> news:vbimad$1j26j$6@dont-email.me...
>>
>> Don't you wonder why people insist on returning "403
>> Forbidden" for those using a command-line tool like wget?
>
> Most SEO tools and other
> useless/criminal scrapers like to fake their identification, and WGet is
> a favorite for that task. There are many owners that block that and many
> other abused user-agents.
That doesn’t make any sense, because anybody who knows how to use wget
would know about its “--user-agent” option. So if they really were using
wget to conduct their site abuse, you wouldn’t know, and blocking wget’s
default user-agent setting wouldn’t help.
> (The Ada-Auth.org blocks about 20 user-agents, but not WGet.
Which kind of proves my point.
next prev parent reply other threads:[~2024-09-12 22:27 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-09-06 1:55 “Red” And The DoD Language Competition Lawrence D'Oliveiro
2024-09-07 16:43 ` Luke A. Guest
2024-09-07 23:06 ` Lawrence D'Oliveiro
2024-09-07 23:31 ` Luke A. Guest
2024-09-08 0:25 ` Keith Thompson
2024-09-08 7:48 ` Nioclás Pól Caileán de Ghloucester
2024-09-12 4:57 ` "Red" " Randy Brukardt
2024-09-12 22:27 ` Lawrence D'Oliveiro [this message]
2024-09-12 23:17 ` Bill Findlay
2024-09-13 1:41 ` geodandw
2024-09-13 2:16 ` Paul Rubin
2024-09-13 2:46 ` Lawrence D'Oliveiro
2024-09-14 6:27 ` Randy Brukardt
2024-09-14 7:21 ` Lawrence D'Oliveiro
replies disabled
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox