pythonweb-scrapingbeautifulsoupscreen-scrapingscrapinghub

YouTube Subscriptions List Scraping

I want to scrap my YouTube subscriptions list into one csv file. I typed this code (but I didn't finish coding yet):

import requests
from bs4 import BeautifulSoup
import csv

url = 'https://www.youtube.com/feed/channels'
source = requests.get(url)
soup = BeautifulSoup(source, 'lxml')

I found this error:

File "/Users/hendy/YouTube subscriptions scraping.py", line 7, in soup = BeautifulSoup(source, 'lxml') File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/bs4/init.py", line 312, in init elif len(markup) <= 256 and ( TypeError: object of type 'Response' has no len()

I don't know what's the problem.

Solution

  • What happens?

    You using the whole response object and push it to BeautifulSoup what would not work.

    How to fix?

    To generate a BeautifulSoup object use the content or text of your response:

    BeautifulSoup(source.content, 'lxml')

    Example

    from bs4 import BeautifulSoup
import requests
headers ={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36'}
url = 'https://www.youtube.com/feed/channels'
source = requests.get(url, headers=headers)
soup = BeautifulSoup(source.content, 'lxml')