I am trying to scrape the titles of the cards from this link : https://www.axisbank.com/retail/cards/credit-card
Using the below code
from urllib.request import urlopen
from bs4 import BeautifulSoup
import json, requests, re
axis_url = ["https://www.axisbank.com/retail/cards/credit-card"]
html = requests.get(axis_url[0])
soup = BeautifulSoup(html.content, 'lxml')
print(soup.select("#ulCreditCard h3"))
Output is as follows :
[]
My main concern is how shall I specify a css selector tag that I got using the selector gadget when it is in this form #ulCreditCard h3 in the soup.select() function.
As mentioned, the information you are looking for is obtained via Javascript. It uses a slightly different URL to get the JSON data containing all of the card details. If you use this instead, you can easily list all of the card names without needing to use BeautifulSoup. For example:
import requests
import json
axis_url = "https://www.axisbank.com/AjaxService/GetCreditCardsProducts"
data = {"strcategory" : "[]", "strrewardtypes" :"[]"}
r = requests.post(axis_url, data=data)
for entry in json.loads(r.json()[0]):
print(entry['Name'])
Would give you the following cards:
Axis Bank ACE Credit Card
Axis Bank AURA Credit Card
Privilege Easy Credit Card
Axis Bank Reserve Credit Card
Axis Bank Freecharge Plus Credit Card
IndianOil Axis Bank Credit Card
Axis Bank Magnus Credit Card
Flipkart Axis Bank Credit Card
Axis Bank Freecharge Credit Card
Axis Bank MY Zone Credit Card
Axis Bank Neo Credit Card
Axis Bank Vistara Credit Card
Axis Bank Vistara Signature Credit Card
Axis Bank Vistara Infinite Credit Card
Axis Bank Privilege Credit Card
Miles and More Axis Bank Credit Card
Axis Bank Select Credit Card
Axis Bank Pride Platinum Credit Card
Axis Bank Pride Signature Credit Card
Axis Bank MY Zone Easy Credit Card
Axis Bank Insta Easy Credit Card
Axis Bank Signature Credit Card with Lifestyle Benefits
Platinum Credit Card
Titanium Smart Traveler Credit Card
Axis Bank My Wings Credit Card