pythonurlurlrequest

Why does the url request not give out the response?


I use the following program to get the url response, but only the following url cannot correctly give out the response. I have waited for a long time, but it still cannot finish the running. How can I solve it?

import re
from bs4 import BeautifulSoup
import whois
import urllib
import urllib.request
import requests
from datetime import datetime

try:
    url="http://zozo.jp/shop/bestpackingstore/?price=proper&p_ssy=2015&p_ssm=5&p_ssd=13&p_sey=2015&p_sem=5&p_sed=13&dstk=2"
    header = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}
    response = requests.get(url, headers=header)
    print(response.text)
except:
    response = ""

Solution

  • Your User-Agent is invalid.

    This code shows an improved pattern of use with a valid user agent

    import requests
    from bs4 import BeautifulSoup as BS
    
    url = 'http://zozo.jp/shop/bestpackingstore'
    params = {
        'price': 'proper',
        'p_ssy': 2015,
        'p_ssm': 5,
        'p_ssd': 13,
        'p_sey': 2015,
        'p_sem': 5,
        'p_sed': 13,
        'dstk': 2
    }
    headers = {
        'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.4 Safari/605.1.15'
    }
    
    with requests.get(url, headers=headers, params=params) as response:
        response.raise_for_status()
        soup = BS(response.text, 'lxml')