jsonperllandsat

Malformed JSON string when using perl and earthexplorer

I'm currently working with the USGS's EarthExplorer to setup some batch downloads of Landsat scenes based on spatial coordinates. They provide a wonderful handy script to do this at https://earthexplorer.usgs.gov/inventory/example/json-download_data-pl which is great. I'm working on a cluster, and despite installing all perl modules properly, when I run the script I get the following output:

Running Script...


Error: Error: malformed JSON string, neither array, object, number, string or 
atom, at character offset 0 (before "LWP will support htt...") at ./dl.pl line 182

This seems curious. As way of explanation, the script starts out with

#!/usr/bin/perl

#use strict;
use warnings;
use 5.010;
use JSON;
use Scalar::Util qw(looks_like_number reftype dualvar );
use LWP::UserAgent;
use Getopt::Long qw(GetOptions);

my ($username, $password);
$username = "myusername_filled_in";
$password = "mypassword_filled_in";

GetOptions(
    'username=s' => \$username,
    'password=s' => \$password,
) or die "Error retrieving Username and Password\n";

and the offending bit of code is

    $res = $response->{_content};
    $res = decode_json $res;

Following very useful advice in Can't run Perl script on other computer I have done the following:

  1. Changed $response->content to $response->decoded_content( charset => 'none') in the offending area of code.

  2. Ran lwp-request https://google.com/ which just pulled back a full webpage - no error. So, that seems to be working.

  3. Tried to see some debug by inserting print $response->decoded_content( charset => 'none'); which then threw the error

LWP will support https URLs if the LWP::Protocol::https module is installed.

And, indeed, LWP::Protocol::https is installed.

I feel like there must be something simple I'm missing - something like how I defined my username and password (just $username = "myusername"; etc., after the variables are declared) or something else asinine.

Has anyone else run into this?

To add output from the query below:

$ which cpan ; head -n 1 `which cpan` ; echo 'o conf' | cpan | grep -P 'make|mbuild' ; set | grep ^PERL ; which perl ; perl -le'use LWP::Protocol::https; print "ok";'

/share/pkg/perl/5.10.1/bin/cpan
#!/share/pkg/perl/5.10.1/bin/perl
    make               [/usr/bin/make]
    make_arg           []
    make_install_arg   []
    make_install_make_command [/usr/bin/make]
    makepl_arg         []
    mbuild_arg         []
    mbuild_install_arg []
    mbuild_install_build_command [./Build]
    mbuildpl_arg       []
PERL5LIB=/home/jb92b/perl5/lib/perl5:/home/jb92b/perl5/lib/perl5:/home/jb92b/perl5/lib/perl5
PERL_LOCAL_LIB_ROOT=/home/jb92b/perl5:/home/jb92b/perl5:/home/jb92b/perl5
PERL_MB_OPT='--install_base "/home/jb92b/perl5"'
PERL_MM_OPT=INSTALL_BASE=/home/jb92b/perl5
/share/pkg/perl/5.10.1/bin/perl
ok
Solution

  • Your code doesn't check for errors. You should have something like

    $response->is_success()
   or die("Can't fetch X: ".$response->status_line());

    The problem you are encountering is that LWP::Protocol::https isn't installed.

    You claim it is, but Perl is authoritative here :) It wasn't installed by that Perl, you installed it a non-standard directory without tell Perl to look for it there, or there's a permission issue.

    In this case, your script uses /usr/bin/perl, but you installed the module using/for /share/pkg/perl/5.10.1/bin/perl. You need to switch which perl your script uses, or you need to install the module using/for /usr/bin/perl.