[SOLVED] Get response 200 instead of <418 I'm a Teapot>, using DDG

Get response 200 instead of <418 I'm a Teapot>, using DDG

I was trying to scrape search results from DDG the other day, but i keep getting response 418. How can i make it response 200 or get results from it? This is my code.

import requests
from bs4 import BeautifulSoup
import urllib
    
while True:
    
    query = input("Enter Search Text: ")

    a = query.replace(' ', '+')

    url = 'https://duckduckgo.com/?q=random' +a
    
    headers = {"User-Agent": "Mozilla/5.0 (Linux; Android 6.0.1; SHIELD Tablet K1 Build/MRA58K; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/55.0.2883.91 Safari/537.36"}
    
    r = requests.get(url)
    print(r)

    soup = BeautifulSoup(r.content,'lxml')
    
    tags = soup.findAll('h2', class_="result__title")
    print(tags)```

Solution

You can use HTML only version of DDG localted on (https://html.duckduckgo.com/html/) to obtain the results.

For example:

import requests
from bs4 import BeautifulSoup


url = 'https://html.duckduckgo.com/html/'
params = {'q': 'python'}
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

soup = BeautifulSoup(requests.get(url, params=params, headers=headers).content, 'html.parser')

for t in soup.select('h2.result__title'):
    print(t.get_text(strip=True))

Prints:

LearnPythonOnline - Start Today & Change Your LifeAdViewing ads is privacy protected by DuckDuckGo. Ad clicks are managed by Microsoft's ad network (more info).
PythonOnline - FreePythonIntro by DataCampAdViewing ads is privacy protected by DuckDuckGo. Ad clicks are managed by Microsoft's ad network (more info).
Welcome toPython.org
How Modulo (%) works inPython: Explained with 6 Examples
Python(programming language) - Wikipedia
python- How do I pass a variable by reference? - Stack ...
PythonTutorial - W3Schools
Python- Basic Operators - Tutorialspoint
DownloadPython|Python.org
PythonFor Beginners |Python.org
Introduction toPython- W3Schools
PythonReleases for Windows |Python.org
PythonReleasePython3.8.2 |Python.org
ThePythonTutorial —Python3.8.3 documentation
Python
Python- Free download and software reviews - CNET ...
ThePythonLanguage Reference —Python3.8.3 documentation
After 19 Years,PythonMay Finally Get a Pattern Matching ...
PythonTutorial - Tutorialspoint
Python| snake group | Britannica
FrontPage -PythonWiki
BeginnersGuide -PythonWiki
syntax - What is :: (double colon) inPythonwhen ...
PythonOperators - W3Schools
PythonTutorial: LearnPythonFor Free | Codecademy
uuid — UUID objects according to RFC 4122 —Python3.8.3 ...
LearnPython- Free InteractivePythonTutorial
PythonString find() Method - W3Schools
PythonPIP - W3Schools
How to Use thePythonor Operator - RealPython
Python2.7.18 documentation
What is the result of % inPython? - Stack Overflow