ruby-on-railsrubyrakemechanize

Login to amazon partnernet using Ruby Mechanize


I try to login into amazon partnernet, e.g. https://partnernet.amazon.de/ using the Ruby Mechanize gem:

Gemfile:

# https://github.com/sparklemotion/mechanize
gem 'mechanize'

The code below is a rake task. It worked in the past, I think Amazon changed the page html so this code is no longer working, e.g. by changing the submit button of the form name="sign_in" to an image type=input.

desc "Cron Task for Email Notifications"
task :email_amazon_stats => :environment do
  puts "Start: Fetch and send Amazon Sales from yesterday (#{Time.now})"

  # login to Amazon Partnernet
  a = Mechanize.new
  a.user_agent_alias = 'Mac Safari'
  a.follow_meta_refresh = true
  a.redirect_ok = true

  a.get('https://partnernet.amazon.de/') do |page|
    # Submit the login form
    page.form_with(:name => 'sign_in') do |f|
      username_field = f.field_with(:id => "username")
      username_field.value = "email@example.com"
      password_field = f.field_with(:id => "password")
      password_field.value = "somepassword"
    end.submit

    start_date  = Time.now - 1.day
    end_date    = Time.now
    my_page2 = a.get("https://partnernet.amazon.de/gp/associates/network/reports/report.html?ie=UTF8&deviceType=all&endDay=#{(end_date.strftime('%d').to_i).to_s}&endMonth=#{((end_date.strftime('%m').to_i)-1).to_s}&endYear=#{end_date.strftime('%Y').to_i.to_s}&periodType=exact&preSelectedPeriod=monthToDate&program=all&reportType=earningsReport&startDay=#{start_date.strftime('%d').to_i.to_s}&startMonth=#{((start_date.strftime('%m').to_i)-1).to_s}&startYear=#{start_date.strftime('%Y').to_s}")

    form = my_page2.form_with(:name => 'htmlReport')
    button = form.button_with(:name => 'submit.download_XML')
    xml = a.submit(form, button)

    # ASIN="3423347570"
    # Binding="paperback"
    # Category="14"
    # Date="December 01, 2015"
    # DeviceType="BROWSER"
    # EDate="1448928000"
    # Earnings="0,65"
    # LinkType="asn"
    # Price="9,25"
    # Qty="1"
    # Rate="7,03"
    # Revenue="9,25"
    # Seller="Amazon.de"
    # Tag="yx-21"
    # Title="Kopf schlägt Kapital: Die ganz andere Art, ein Unternehmen zu gründen Von der Lust, ein Entrepreneur zu sein (dtv Sachbuch)"/>

    doc = Nokogiri::XML(xml.body)
    @sales = []
    doc.xpath("//Item").each do |item|
      @sales << {
          :sale_itemasin    => item['ASIN'],
          :sale_itemname    => item['Title'].truncate(80),
          :sale_date        => Time.at(item['EDate'].to_i).strftime("%Y-%m-%d %H:%M:%S").to_s,
          :sale_amount      => '%.2f' % item['Revenue'].gsub(',','.').to_f,
          :sale_commission  => '%.2f' % item['Earnings'].gsub(',','.').to_f
      }
    end

    earnings = 0
    @sales.each do |s|
      earnings += s[:sale_commission].to_f
    end
    @total_commission = '%.2f' % earnings
  end

  ReportsMailer.daily_dashboard(@total_commission,@sales).deliver
  puts "Done: Fetch and send Amazon Sales from yesterday (#{Time.now})"
end

Can someone help me in this?

--

I looked for similar questions how to restructure the submit, but so far nothing works. Login is not happening. (Yes, PWD is correct :-) )

Similar question, but does not solve the problem above: Cannot Login to Amazon with Ruby Mechanize


Solution

  • So.. I debugged the code, this version is working now as expected:

    desc "Cron Task for Email Notifications"
    task :email_amazon_stats => :environment do
      puts "Start: Fetch and send Amazon Sales from yesterday (#{Time.now})"
    
      agent = Mechanize.new
      agent.cookie_jar.clear!
      agent.user_agent_alias = 'Mac Firefox'
      agent.follow_meta_refresh = true
      agent.redirect_ok = true
    
      dashboard_url = "https://partnernet.amazon.de/gp/associates/network/reports/report.html?__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&tag=&reportType=earningsReport&program=all&deviceType=all&periodType=preSelected&preSelectedPeriod=yesterday&startDay=1&startMonth=11&startYear=2016&endDay=2&endMonth=11&endYear=2016&submit.display.x=87&submit.display.y=16&submit.display=Auf+der+Seite+anzeigen"
    
      agent.get(dashboard_url)
    
      form = agent.page.form_with(:name => 'sign_in')
      form.username = ENV['AZON_PARTNER_USR']
      form.password = ENV['AZON_PARTNER_KEY']
      form.submit
    
      dashboard = agent.get(dashboard_url)
      form2 = dashboard.form_with(:name => 'htmlReport')
      button = form2.button_with(:name => 'submit.download_XML')
      xml = agent.submit(form2, button)
    
      doc = Nokogiri::XML(xml.body)
      @sales = []
      doc.xpath("//Item").each do |item|
        @sales << {
            :sale_itemasin    => item['ASIN'],
            :sale_itemname    => item['Title'].truncate(80),
            :sale_date        => Time.at(item['EDate'].to_i).strftime("%Y-%m-%d %H:%M:%S").to_s,
            :sale_amount      => '%.2f' % item['Revenue'].gsub(',','.').to_f,
            :sale_commission  => '%.2f' % item['Earnings'].gsub(',','.').to_f
        }
      end
    
      earnings = 0
      @sales.each do |s|
        earnings += s[:sale_commission].to_f
      end
    
      @total_commission = '%.2f' % earnings
    
      ReportsMailer.daily_dashboard(@total_commission,@sales).deliver
      puts "Done: Fetch and send Amazon Sales from yesterday (#{Time.now})"
    end
    

    As you can see yourself this is pretty ugly, because I try to go to the deeplink directly, which redirects me to the login page. There I login and try again to go to the dashboard. This time it works. Why ugly? Because if I try to go to the login page directly the code does not work, I somehow need this redirect. Any idea why? Would be interesting to understand this...