pythondjangofacebookfacebook-graph-api

Facebook not consistently scraping information as per OG meta tags

I automatically post adverts for people's jobs on member's facebook pages using Python's facebook module. I have applied for and got Facebook's authority to do this. Now when I post to most people's pages, the image and description are not being scraped, however when I post with the FB site owner's access token of the business site associated with the application, they are. The code I use to post the pages:

  import facebook
  graph       = facebook.GraphAPI(access_token)
  # record the id of the wall post so we can like it
  fbWallPost  = graph.put_wall_post(message=fbmsg,  attachment=fbpost, profile_id=pageID)

Running this in the python shell posts a URL that gets scraped, but running it from within the application it very frequently does not. The page has the following meta tags:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">

  <title>Fair Work</title>


  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <meta name="author" content="Fair Work">
    <!-- Facebook crawler uses these JRT -->
  <meta property="og:site_name" content="Fair Work"/>
  <meta property="fb:app_id" content="322643981237986" />


  <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate" />
  <meta http-equiv="Pragma" content="no-cache" />
  <meta http-equiv="Expires" content="0" />


  <meta property="og:url" content="http://fairwork.com.au/job/final-job-on-thursday-april-30th/" />
  <meta property="og:description" content="My last test today"/>
  <meta property="og:type" content="article"/>

  <meta property="og:image" content="http://fairwork.com.au/media/cache/cd/b0/cdb05f8dd8885351925bf43076870937.jpg"/>
  <meta property="og:image:width" content="400" />
  <meta property="og:image:height" content="400" />

The application ID is correct. I go to Facebook's debugger, and I enter the URL My URL given to FB debugger
I click on "Debug", and I get an error that the "og:type" meta tag is missing, which it is not as you can see: enter image description here I then click on "Fetch new scrape information" ... enter image description here ...and the image and description are loaded up fine. From this point if I post the link on my site, the image and description display, however all previously posted links do not have the information scraped. The two successive posts of the same link in the next image were done before (bottom) and after (top) running through the debug process mentioned previously: enter image description here

The page linked to is publicaly available, and served by Django. I want to be able to automatically post the ad's (when this is requested by the FB page owner) and have the image and description scraped by FB on posting. Any ideas?

Solution

  • It seems that NGINX is causing the http calls to the Facebook graph API not to work properly. When 

    fbWallPost  = graph.put_wall_post(message=fbmsg,  attachment=fbpost, profile_id=pageID)

    is run through Celery (which runs separately to django, and not through the web server), facebook scrapes the posted link and shows the image and description linked to by the OG meta tags. When django runs put_wall_post, the post occurs, but facebook does not scrape the link. Other calls to the API also don't work when run from a Django view. I can only think that NGINX is interfering with the HTTP request in some way.