pythonscrapyscrapyd

I can't access scrapyd port 6800 from browser

I searched a lot on this, it may have a simple solution that I am missing.

I have setup scrapy + scrapyd on both my local machine and my server. They work both ok when I try as "scrapyd".

I can deploy to local without a problem, and I can access to localhost:6800 as well from the browser and I can run spiders on local.

After running scrapyd on remote, I try to deploy to http://remoteip:6800/ with the same as I did deploy locally,

I get,

Packing version 1500333306
Deploying to project "projectX" in http://remoteip:6800/addversion.json
Deploy failed: <urlopen error [Errno 111] Connection refused>

I also can't access http://remoteip:6800/ from my local PC, but I can access from ssh on remote PC (with curl)

I opened inbound and outbound connections on the remote server, what else I am missing?

Thanks

Solution

  • First check if its running or not, run curl localhost:6800 on the server where ScrapyD is running

    Check if firewall is enabled

    sudo ufw status

    Ideally, just allow tcp connections to 6800instead of disabling firewall, to do so

    sudo ufw allow 6800/tcp
sudo ufw reload

    Check your scrapyd.conf please set

    bind_address=0.0.0.0

    instead of

    bind_address=127.x.x.x

    0.0.0.0 will make scrapyD accessible for incoming connections outside the server/instance, not only localhost.

    Then stop scrapyD, I do killall scrapyd to stop scrapyd

    Then restart scrapyD using command scrapyd

    Note: If you want to keep scrapyd running even after you disconnect from server, do this

    nohup scrapyd >& /dev/null &

    Also see my answer to set ScrapyD as a System Service