Re: Credentials in LWP

At 09:53 2002-07-02 -0500, Kenny G. Dubuisson, Jr. wrote:
>[...]I don't understand what it means by NetLoc and Realm[...]

Try hitting http://www.unicode.org/mail-arch/unicode-ml and it'll 
say  "enter your username and password for 
Unicode-MailList-Archives".  That "Unicode-MailList-Archives" string is the 
realm name.  NetLoc is the hostname plus colon plus the port number, by 
default ":80" -- in this case, "www.unicode.org:80".

Here's an extract from chapter 11 of my new book, /Perl and LWP/
(<http://www.amazon.com/exec/obidos/ASIN/0596001789>)
which you might find useful and worth buying:



Authenticating via LWP

To add a username and password to a browser object's key ring, call the 
credentials method on a user agent object:

   $browser->credentials(
     'servername:portnumber',
     'realm-name',
     'username' => 'password'
   );

In most cases, the port number is 80, the default TCP/IP port for HTTP. For 
example:

   my $browser = LWP::UserAgent->new;
   $browser->name('ReportsBot/1.01');
   $browser->credentials(
     'reports.mybazouki.com:80',
     'web_server_usage_reports',
     'plinky' => 'banjo123'
   );
   my $response = $browser->get(
     'http://reports.mybazouki.com/this_week/'
   );

One can call the credentials method any number of times, to add all the 
server-port-realm-username-password keys to the browser's key ring, 
regardless of whether they'll actually be needed. For example, you could 
read them all in from a datafile at startup:

   my $browser = LWP::UserAgent->new( );
   if(open(KEYS, "< keyring.dat")) {
     while(<KEYS>) {
       chomp;
       my @info = split "\t", $_, -1;
       $browser->credential(@info) if @info == 4;
     }
     close(KEYS);
   }


Security

Clearly, storing lots of passwords in a plain text file is not terribly 
good security practice, but the obvious alternative is not much better: 
storing the same data in plain text in a Perl file. One could make a point 
of prompting the user for the information every time,* instead of storing 
it anywhere at all, but clearly this is useful only for interactive 
programs (as opposed to a programs run by crontab, for example). In any 
case, HTTP Basic Authentication is not the height of security: the username 
and password are normally sent unencrypted. This and other security 
shortcomings with HTTP Basic Authentication are explained in greater detail 
in RFC 2617. See the Preface for information on where to get a copy of RFC 
2617.

* In fact, Ave Wrigley wrote a module to do exactly that. It's not part of 
the LWP distribution, but it's available in CPAN as LWP::AuthenAgent. The 
author describes it as "a simple subclass of LWP::UserAgent to allow the 
user to type in username/password information if required for authentication."



An HTTP Authentication Example: The Unicode Mailing Archive

Most password-protected sites (whether protected via HTTP Basic 
Authentication or otherwise) are that way because the sites' owners don't 
want just anyone to look at the content. And it would be a bit odd if I 
gave away such a username and password by mentioning it in this book! 
However, there is one well-known site whose content is password protected 
without being secret: the mailing list archive of the Unicode mailing lists.

In an effort to keep email-harvesting bots from finding the Unicode mailing 
list archive while spidering the Web for fresh email addresses, the 
Unicode.org sysadmins have put a password on that part of their site. But 
to allow people (actual not-bot humans) to access the site, the site 
administrators publicly state the password, on an unprotected page, at 
http://www.unicode.org/mail-arch/, which links to the protected part, but 
also states the username and password you should use.

The main Unicode mailing list (called unicode) once in a while has a thread 
that is really very interesting and you really must read, but it's buried 
in a thousand other messages that are not even worth downloading, even in 
digest form. Luckily, this problem meets a tidy solution with LWP: I've 
written a short program that, on the first of every month, downloads the 
index of all the previous month's messages and reports the number of 
messages that has each topic as its subject.

The trick is that the web pages that list this information are password 
protected. Moreover, the URL for the index of last month's posts is 
different every month, but in a fairly obvious way. The URL for March 2002, 
for example, is:
   http://www.unicode.org/mail-arch/unicode-ml/y2002-m03/

Deducing the URL for the month that has just ended is simple enough:

   # To be run on the first of every month...
   use POSIX ('strftime');
   my $last_month = strftime("y%Y-m%m", localtime(time - 24 * 60 * 60));
   # Since today is the first, one day ago (24*60*60 seconds) is in
   # last month.
   my $url = "http://www.unicode.org/mail-arch/unicode-ml/$last_month/";

But getting the contents of that URL involves first providing the username 
and password and realm name. The Unicode web site doesn't publicly declare 
the realm name, because it's an irrelevant detail for users with 
interactive browsers, but we need to know it for our call to the credential 
method. To find out the realm name, try accessing the URL in an interactive 
browser. The realm will be shown in the authentication dialog box, as shown 
in Figure 11-1.

In this case, it's "Unicode-MailList-Archives," which is all we needed to 
make our request.

   my $browser = LWP::UserAgent->new;
   $browser->credentials(
     'www.unicode.org:80', # Don't forget the ":80"!
     # This is no secret...
     'Unicode-MailList-Archives',
     'unicode-ml' => 'unicode'
   );
   print "Getting topics for last month, $last_month\n",
         " from $url\n";
   my $response = $browser->get($url);
   die "Error getting $url: ", $response->status_line
    if $response->is_error;

If this fails (if the Unicode site's admins have changed the username or 
password or even the realm name), that will die with this error message:

   Error getting http://www.unicode.org/mail-arch/unicode-ml/y2002-m03/:
   401 Authorization Required at unicode_list001.pl line 21.

But assuming the authorization data is correct, the page is retrieved as if 
it were a normal, unprotected page. From there, counting the topics and 
noting the absolute URL of the first message of each thread is a matter of 
extracting data from the HTML source and reporting it concisely.

   my(%posts, %first_url);
   while( ${ $response->content_ref }
     =~ m{<li><a href="(\d+.html)"><strong>(.*?)</strong>}g
     # Like: <li><a href="0127.html"><strong>Klingon</strong>
   ) {
     my($url, $topic) = ($1,$2);
     # Strip any number of "Re:" prefixes.
     while( $topic =~ s/^Re:\s+//i ) {}
     ++$posts{$topic};
     use URI; # For absolutizing URLs...
     $first_url{$topic} ||= URI->new_abs($url, $response->base);
   }
   print "Topics:\n", reverse sort map # Most common first:
     sprintf("% 5s %s\n %s\n",
     $posts{$_}, $_, $first_url{$_}
   ), keys %posts;

Typical output starts out like this:

   Getting topics for last month, y2002-m02
   from http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/
   Topics:
   86 Unicode and Security
    http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/0021.html
   47 ISO 3166 (country codes) Maintenance Agency Web pages move
    http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/0390.html
   41 Unicode and end users
    http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/0260.html
   27 Unicode Search Engines
    http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/0360.html
   22 Smiles, faces, etc
    http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/0275.html
   18 This spoofing and security thread
    http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/0216.html
   16 Standard Conventions and euro
    http://www.unicode.org/mail-arch/unicode-ml/y2002-m02/0418.html

This continues for a few pages.

[end extract]

--
Sean M. Burke    http://www.spinn.net/~sburke/

• Credentials in LWP by Kenny G. Dubuisson, Jr.
• Re: Credentials in LWP by Sean M. Burke