How to capture large STDOUT output in Perl when executing an external command

I want to get the list of file names present in the remote location.

I am using the below snippet in my Perl script.

my $command = "sftp -q -o${transferAuthMode}=yes -oPort=$sftpPort ${remoteUsername}\@${remoteHost} 2>\&1 <<EOF\n" . 
       "cd \"${remotePath}\"\n" . 
       "ls -l \n" . 
       "quit\n" . 
       "EOF\n";

my @files = `$command`;

When the number of files in the remote location is large (>500) then not all the file names are captured in @files.

When I manually do SFTP and list the files, all files are getting listed but I'm not getting the same through the script. Each time getting @files size different. It's occurring only when there are large number of files.

I'm unable find the reason behind this. Could you please help?

Solution

This can be achieved without requiring any additional package module/s. I tested this on my CentOS 7 Server (Windows VM).

My remote host details: I got ~2000 files in the remote host dir. A CentOS 6.8 server.

%_gaurav@[remotehost]:/home/gaurav/files/test> ls -lrth|head -3;echo;ls -lrth|tail -2
total 7.9M
-rw-rw-r--. 1 gaurav gaurav 35 Feb 16 23:51 File-0.txt
-rw-rw-r--. 1 gaurav gaurav 35 Feb 16 23:51 File-1.txt

-rw-rw-r--. 1 gaurav gaurav 38 Feb 16 23:51 File-1998.txt
-rw-rw-r--. 1 gaurav gaurav 38 Feb 16 23:51 File-1999.txt
%_gaurav@[remotehost]: /home/gaurav/files/test>

Script output from LocalHost: Please note that I am running your command sans the o${transferAuthMode}=yes part. As seen below, the script is able to gather all results in an array, greater than 500 results.

I am printing the total entries, some particular index numbers from the array to show the results, but give it a try with un-commented Dumper line to see the full result.

%_STATION@gaurav * /root/ga/study/pl> ./scp.pl
Read 2003 lines from SCP command.

ArrayIndex: 2,3,1999,2000 contain:

[-rw-rw-r--    0 501      501           36B Feb 16 23:51 File-58.txt]
[-rw-rw-r--    0 501      501           37B Feb 16 23:51 File-129.txt]
[-rw-rw-r--    0 501      501           38B Feb 16 23:51 File-1759.txt]
[-rw-rw-r--    0 501      501           38B Feb 16 23:51 File-1810.txt]
%_STATION@gaurav * /root/ga/study/pl>

Script and it's working:

#!/usr/bin/perl

use strict ;
use warnings ;
use Data::Dumper ;

my $sftp_port=22 ;
my ($user, $host) = ("gaurav","192.168.246.137") ;
my $remote_path = '/home/gaurav/files/test' ;

my @result ;    # To store result

my $command = "sftp -q -oPort=$sftp_port ${user}\@${host} 2>\&1 <<EOF\n"."cd $remote_path\nls -lrth\nquit\nEOF" ;

# open the command as a file handle, read output and store it.
open FH, "$command |" or die "Something went wrong!!\n" ;
while (<FH>) {
  tr/(?\r|\f|\n)//d ; # Removing any new line, carriage return or form feed.
  push(@result,"\[$_\]") ;
}
close FH ;

#print Dumper @result ;

# Just for printing a little bit of results from
# the array. Following lines can be deleted.
my $total = scalar @result ;
print "Read $total lines from SCP command.\n" ;
print "\nArrayIndex: 2,3,1999,2000 contain:\n
$result[2]
$result[3]
$result[1999]
$result[2000]
" ;

Another way: One could also get around this issue by making a shell script and calling it from the perl script and read its output. As shown below, my shell script which gets called by the perl script and the final output. This can be used as a quick technique when one doesn't have much time to write/formulate commands in perl directly. You can use the qx style(shown below) in earlier script as well.

Shell script "scp.sh"

%_STATION@gaurav * /root/ga/study/pl> cat scp.sh
#!/bin/bash

sftp -oPort=${1} ${2}@${3} 2>&1 <<EOF
cd ${4}
ls -l
quit
EOF

Perl Script "2scp.pl"

%_STATION@gaurav * /root/ga/study/pl> cat 2scp.pl
#!/usr/bin/perl

use strict ;
use warnings ;
use Data::Dumper ;

my $sftp_port=22 ;
my ($user, $host) = ("gaurav","192.168.246.137") ;
my $remote_path = '/home/gaurav/files/test' ;

# Passing arguments to shell script using concatenation.
my $command = './scp.sh '." $sftp_port $user $host $remote_path" ;

my @result = qx{$command} ;     # Runs the command and stores the result.
my $total = scalar @result ;
print "Read $total lines from SCP command.\n" ;
# End.

Output:

%_STATION@gaurav * /root/ga/study/pl> ./2scp.pl
Read 2004 lines from SCP command.
%_STATION@gaurav * /root/ga/study/pl>

Try it out and let us know.

Thanks.