I'm making a python web scrapper for MyAnimeList
So basically if you go onto the website, and if you hover onto a title of an anime. It'll pop up a hover window which gives out a brief details of this anime.
I'm trying to use request and beautifulsoup4 module to get the html of the hoverwindow. Specifically, I want my program to grab the Genres for each anime on that page and possibly put them into a list.
How do I do that?
The html for a genre will go like this(Every genre is the subclass of detail)
<span class="dark_text">Genres:</span>
The genre for the anime
When I use:
soup.find_all("span",class_="dark_text")
It returns nothing.
Try:
import requests
from bs4 import BeautifulSoup
url = "https://myanimelist.net/topanime.php"
hover_url = "https://myanimelist.net/anime/{}/hover"
soup = BeautifulSoup(requests.get(url).content, "html.parser")
for a in soup.select("h3 > a[id]"):
title = a.text
id_ = a["id"].removeprefix("#area")
print(title)
soup2 = BeautifulSoup(requests.get(hover_url.format(id_)).content, "html.parser")
for span in soup2.select("span.dark_text"):
print(span.text, span.find_next_sibling(string=True).get_text(strip=True))
print("-" * 80)
Prints:
...
--------------------------------------------------------------------------------
Hunter x Hunter (2011)
Genres: Action, Adventure, Fantasy
Demographic: Shounen
Status: Finished Airing
Type: TV
Episodes: 148
Score: 9.03
Ranked: #8
Popularity: #9
Members: 2,853,222
--------------------------------------------------------------------------------
Ginga Eiyuu Densetsu
Genres: Drama, Sci-Fi
Themes: Adult Cast, Military, Space
Status: Finished Airing
Type: OVA
Episodes: 110
Score: 9.02
Ranked: #9
Popularity: #746
Members: 324,593
--------------------------------------------------------------------------------
...