rubytimeoutdeadlockopen-uri

How to fix a deadlock caused by open

I have is a deadlock, but I am not using any threads in my program. Plus, the error only happens about once every 1000 to 1500 function calls, making it very difficult to pinpoint and correct.

Here is the complete error message when the issue occurs:

/usr/lib/ruby/2.3.0/timeout.rb:95:in `join': No live threads left. Deadlock? (fatal)
    from /usr/lib/ruby/2.3.0/timeout.rb:95:in `ensure in block in timeout'
    from /usr/lib/ruby/2.3.0/timeout.rb:95:in `block in timeout'
    from /usr/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
    from /usr/lib/ruby/2.3.0/net/http.rb:878:in `connect'
    from /usr/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
    from /usr/lib/ruby/2.3.0/net/http.rb:852:in `start'
    from /usr/lib/ruby/2.3.0/open-uri.rb:319:in `open_http'
    from /usr/lib/ruby/2.3.0/open-uri.rb:737:in `buffer_open'
    from /usr/lib/ruby/2.3.0/open-uri.rb:212:in `block in open_loop'
    from /usr/lib/ruby/2.3.0/open-uri.rb:210:in `catch'
    from /usr/lib/ruby/2.3.0/open-uri.rb:210:in `open_loop'
    from /usr/lib/ruby/2.3.0/open-uri.rb:151:in `open_uri'
    from /usr/lib/ruby/2.3.0/open-uri.rb:717:in `open'
    from /usr/lib/ruby/2.3.0/open-uri.rb:35:in `open'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/utils.rb:85:in `get_pic'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:87:in `page_link'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:116:in `chapter_link'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_download.rb:142:in `chapter'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:57:in `block in MF_manga_missing_chapters'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:45:in `reverse_each'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:45:in `MF_manga_missing_chapters'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/mangafox/MF_update.rb:80:in `MF_update'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:5:in `update_manga'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:15:in `block in update_all'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:14:in `each'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:14:in `update_all'
    from /home/mat/travail_perso/RUBY/MangaScrapp_github/sources/update.rb:22:in `update'
    from ./MangaScrap.rb:28:in `<main>'

The link to the complete program is https://github.com/Hellfire01/MangaScrap

The issue happens to the three different methods that use open. Here is the one that crashed this time:

# conect to link and download picture
def get_pic(link)
  safe_link = link.gsub(/[\[\]]/) { '%%%s' % $&.ord.to_s(16) }
  tries ||= 20
  begin
    page = open(safe_link, "User-Agent" => "Ruby/#{RUBY_VERSION}")
  rescue URI::InvalidURIError => error
    puts "Warning : bad url"
    puts link
    puts "message is : " + error.message
    return nil
  rescue => error
    if tries > 0
    tries -= 1
    sleep(0.2)
    retry
    else
      puts 'could not get picture ' + safe_link + ' after ' + $nb_tries.to_s + ' tries'
      puts "message is : " + error.message
      return nil
    end
  end
  sleep(0.2)
  return page
end

Here is the link to the file: https://github.com/Hellfire01/MangaScrap/blob/master/sources/utils.rb

I would like to know:

Solution

  • You're not catching all exceptions here. When nothing is specified after rescue, it means that you're catching StandardError which is not at the root of Exceptions' hierarchy.

    If you want to make sure you're catching all exceptions and retry opening a URL (or whatever behavior you'd like), what you want to do is:

    rescue Exception => error