I am using Popen3 to run some Perl scrips and then dump their output into a text file. Once in the text file, I search for the result of the Perl script. I get the error after running for about 40 minutes, which is about 220 files.
ruby/1.8/open3.rb:49:in `pipe': Too many open files (Errno::EMFILE)
from /ruby/1.8/open3.rb:49:in `popen3'
from ./RunAtfs.rb:9
from ./RunAtfs.rb:8:in `glob'
from ./RunAtfs.rb:8
The script is below.
require 'logger'
require 'open3'
atfFolder = ARGV[0]
testResult = ARGV[1]
res = "result.txt"
open('result.txt', 'w') { }
Dir.glob(atfFolder+'/*.pl') do |atfTest|
Open3.popen3("atf.pl -c run-config.pl -t #{atfTest}") do |i, o, e, t|
while line = e.gets
$testFile = testResult + line[/^[0-9]+$/].to_s + "testOutput.txt"
log = Logger.new($testFile)
log.info(line)
end
log.close
end
lastLine = `tail +1 #{$testFile}`
file = File.open(res, 'a')
if(lastLine.include? "(PASSED)")
file.puts("Test #{atfTest} --> Passed")
file.close
File.delete($testFile)
else
file.puts("Test #{atfTest} --> Failed!")
file.close
end
end
This script is processing 4900 Perl files so I don't know if that is just too many files for popen3 or I am not using it correctly.
Thanks for helping me!
I refactored my script after some very helpful pointers! The code is working great!
require 'open3'
atf_folder, test_result = ARGV[0, 2]
File.open('result.txt', 'w') do |file| end
Dir.glob("#{ atf_folder }/*.pl") do |atf_test|
test_file = atf_test[/\/\w+.\./][1..-2].to_s + ".txt"
comp_test_path = test_result + test_file
File.open(comp_test_path, 'w') do |file| end
Open3.popen3("atf.pl -c run-config.pl -t #{ atf_test }") do |i, o, e, t|
while line = e.gets
File.open(comp_test_path, 'a') do |file|
file.puts(line)
end
end
end
last_line = `tail +1 #{comp_test_path}`
File.open('result.txt', 'a') do |file|
output_str = if (last_line.include? "(PASSED)")
File.delete(comp_test_path)
"Passed"
else
"Failed!"
end
file.puts "Test #{ atf_test } --> #{ output_str }"
end
end
Consider this:
require 'logger'
require 'open3'
atf_folder, test_result = ARGV[0, 2]
Dir.glob("#{ atf_folder }/*.pl") do |atf_test|
Open3.popen3("atf.pl -c run-config.pl -t #{ atf_test }") do |i, o, e, t|
while line = e.gets
$testFile = test_result + line[/^[0-9]+$/].to_s + "testOutput.txt"
log = Logger.new($testFile)
log.info(line)
log.close
end
end
lastLine = `tail +1 #{ $testFile }`
File.open('result.txt', 'a') do |file|
output_str = if (lastLine.include? "(PASSED)")
File.delete($testFile)
"Passed"
else
"Failed!"
end
file.puts "Test #{ atf_test } --> #{ output_str }"
end
end
It's untested, of course, since there's no sample data, but it's written more idomatically for Ruby.
Things to note:
atf_folder, test_result = ARGV[0, 2] slices the ARGV array and uses parallel assignment to retrieve both parameters at once. You should test to see that you got values for them. And, as you move to more complex scripts, take advantage of the OptionParser class that comes in Ruby's STDLIB.Ruby lets us pass a block to File.open, which automatically closes the file when the block exits. This is a major strength of Ruby and helps reduce errors like you're seeing. Logger doesn't do that, so extra care has to be take to avoid leaving hanging file-handles, like you're doing. Instead, use:
log = Logger.new($testFile)
log.info(line)
log.close
to immediately close the handle. You are doing it outside the loop, not inside it, so you had a bunch of open handles.
Also consider whether you need Logger, or if a regular File.open would suffice. Logger has additional overhead.
$testFile is questionable. $variables are globals, and their use is generally an indicator you're doing something wrong, at least until you understand why and when you should use them. I'd refactor the code using that.In general, I question whether this is the fastest way to do what you want. I suspect you could write a shell script using grep or tail that would at least keep up, and maybe run faster. You might sit down with your sysadmin and do some brain-pickin'.