I'm trying to scrape data from Google Maps but I'm not able to do it successfully as I'm not able to fetch find_elements(by class_name or css_selector) but I'm able to fetch that element by XPATH and the problem is XPATH keeps on changing.
The Class_Name I want to fetch is Io6YTe fontBodyMedium kR99db .
There is more than one of these class_names and I want to fetch all and store data from it but when I do find_elements(by class_name or css_selector) it returns an empty list.
Here's the HTML Part:
<div class="m6QErb " aria-label="Information for Jio Center" role="region">
<div ve-visible="cf" jslog="39448;"></div>
<div ve-visible="cf" jslog="39497;"></div>
<div class="RcCsl fVHpi w4vB1d NOE9ve M0S7ae AG25L ">
<div class="OyjIsf "></div>
<button class="CsEnBe" jsaction="pane.wfvdle118;clickmod:pane.wfvdle118;focus:pane.focusTooltip;blur:pane.blurTooltip" aria-label="Address: HM3V+GF6, Sector No.11, Hiran Magri, Udaipur, Rajasthan 313001 " data-item-id="address" data-tooltip="Copy address" jslog="36622; track:click; mutable:true;">
<div class="AeaXub">
<div class="cXHGnc">
<div class="Eottgc "><img class="Liguzb " alt="" src="//www.gstatic.com/images/icons/material/system_gm/2x/place_gm_blue_24dp.png"></div>
</div>
<div class="rogA2c ">
<div class="Io6YTe fontBodyMedium kR99db ">HM3V+GF6, Sector No.11, Hiran Magri, Udaipur, Rajasthan 313001</div>
<div class="gSkmPd fontBodySmall CuiGbf DshQNd"></div>
</div>
</div>
</button>
<div class="UCw5gc">
<div class="C9yzub">
<div class="etWJQ kdfrQc ">
<button class="g88MCb S9kvJb " jsaction="pane.wfvdle119;keydown:pane.wfvdle119;mouseover:pane.wfvdle119;mouseout:pane.wfvdle119;focus:pane.focusTooltip;blur:pane.blurTooltip" aria-label="Copy address" data-tooltip="Copy address" data-value="Copy address" jslog="38087;track:click;mutable:true;">
<span class="DVeyrd ">
<div class="OyjIsf zemfqc"></div>
<img class="EgL07d" aria-hidden="true" draggable="false" alt="Copy address" src="//www.gstatic.com/images/icons/material/system_gm/2x/content_copy_gm_grey_18dp.png">
</span>
</button>
</div>
</div>
</div>
</div>
<div class="OqCZI fontBodyMedium WVXvdc" jslog="36914;metadata:WyIwYWhVS0V3alJydENGcFlHQkF4VVZabXdHSGVXZkNfd1E4QmNJQlNnQSJd">
<div data-hide-tooltip-on-mouse-move="true" class="OMl5r hH0dDd jBYmhd" aria-expanded="false" role="button" tabindex="0" jsaction="pane.openhours.189.dropdown; keydown:pane.openhours.189.dropdown" jslog="14925;track:click,keydown_click;mutable:true;metadata:WyIwYWhVS0V3alJydENGcFlHQkF4VVZabXdHSGVXZkNfd1E4QmNJQlNnQSJd">
<div class="OyjIsf "></div>
<img class="OdW2qd" src="//www.gstatic.com/images/icons/material/system_gm/2x/schedule_gm_blue_24dp.png" aria-label="Hours">
<div class="MkV9">
<div class="o0Svhf"><span class="ZDu9vd"><span><span style="font-weight: 400; color: rgba(217,48,37,1.00);">Closed</span><span style="font-weight: 400;"> ⋅ Opens 10 pm</span></span></span> <span role="img" class="rh7Scc LaAyid M5ziBd " aria-label="Show open hours for the week"></span></div>
</div>
</div>
<div aria-label="Tuesday, 10 pm to 12 am; Wednesday, 10 pm to 12 am; Thursday, 10 pm to 12 am; Friday, 10 pm to 12 am; Saturday, 10 pm to 12 am; Sunday, Closed; Monday, 10 pm to 12 am. Hide open hours for the week" class="t39EBf GUrTXd">
<div>
<table class="eK4R0e fontBodyMedium">
<tbody>
<tr class="y0skZc">
<td class="ylH6lf fontTitleSmall">
<div>Tuesday</div>
</td>
<td class="mxowUb" role="text" aria-label="10 pm to 12 am">
<ul class="fontTitleSmall">
<li class="G8aQO">10 pm–12 am</li>
</ul>
</td>
<td class="HuudEc">
<button class="mWUh3d" data-hide-tooltip-on-mouse-move="true" data-value="Tuesday, 10 pm–12 am" jsaction="pane.openhours.189.copy; ; focus:pane.focusTooltip; blur:pane.blurTooltip" data-tooltip="Copy open hours" aria-label="Tuesday, 10 pm to 12 am, Copy open hours">
<div class="OyjIsf "></div>
<div class="GYnni"></div>
</button>
</td>
</tr>
<tr class="y0skZc">
<td class="ylH6lf ">
<div>Wednesday</div>
</td>
<td class="mxowUb" role="text" aria-label="10 pm to 12 am">
<ul>
<li class="G8aQO">10 pm–12 am</li>
</ul>
</td>
<td class="HuudEc">
<button class="mWUh3d" data-hide-tooltip-on-mouse-move="true" data-value="Wednesday, 10 pm–12 am" jsaction="pane.openhours.189.copy; ; focus:pane.focusTooltip; blur:pane.blurTooltip" data-tooltip="Copy open hours" aria-label="Wednesday, 10 pm to 12 am, Copy open hours">
<div class="OyjIsf "></div>
<div class="GYnni"></div>
</button>
</td>
</tr>
<tr class="y0skZc">
<td class="ylH6lf ">
<div>Thursday</div>
</td>
<td class="mxowUb" role="text" aria-label="10 pm to 12 am">
<ul>
<li class="G8aQO">10 pm–12 am</li>
</ul>
</td>
<td class="HuudEc">
<button class="mWUh3d" data-hide-tooltip-on-mouse-move="true" data-value="Thursday, 10 pm–12 am" jsaction="pane.openhours.189.copy; ; focus:pane.focusTooltip; blur:pane.blurTooltip" data-tooltip="Copy open hours" aria-label="Thursday, 10 pm to 12 am, Copy open hours">
<div class="OyjIsf "></div>
<div class="GYnni"></div>
</button>
</td>
</tr>
<tr class="y0skZc">
<td class="ylH6lf ">
<div>Friday</div>
</td>
<td class="mxowUb" role="text" aria-label="10 pm to 12 am">
<ul>
<li class="G8aQO">10 pm–12 am</li>
</ul>
</td>
<td class="HuudEc">
<button class="mWUh3d" data-hide-tooltip-on-mouse-move="true" data-value="Friday, 10 pm–12 am" jsaction="pane.openhours.189.copy; ; focus:pane.focusTooltip; blur:pane.blurTooltip" data-tooltip="Copy open hours" aria-label="Friday, 10 pm to 12 am, Copy open hours">
<div class="OyjIsf "></div>
<div class="GYnni"></div>
</button>
</td>
</tr>
<tr class="y0skZc">
<td class="ylH6lf ">
<div>Saturday</div>
</td>
<td class="mxowUb" role="text" aria-label="10 pm to 12 am">
<ul>
<li class="G8aQO">10 pm–12 am</li>
</ul>
</td>
<td class="HuudEc">
<button class="mWUh3d" data-hide-tooltip-on-mouse-move="true" data-value="Saturday, 10 pm–12 am" jsaction="pane.openhours.189.copy; ; focus:pane.focusTooltip; blur:pane.blurTooltip" data-tooltip="Copy open hours" aria-label="Saturday, 10 pm to 12 am, Copy open hours">
<div class="OyjIsf "></div>
<div class="GYnni"></div>
</button>
</td>
</tr>
<tr class="y0skZc">
<td class="ylH6lf ">
<div>Sunday</div>
</td>
<td class="mxowUb" role="text" aria-label="Closed">
<ul>
<li class="G8aQO">Closed</li>
</ul>
</td>
<td class="HuudEc">
<button class="mWUh3d" data-hide-tooltip-on-mouse-move="true" data-value="Sunday, Closed" jsaction="pane.openhours.189.copy; ; focus:pane.focusTooltip; blur:pane.blurTooltip" data-tooltip="Copy open hours" aria-label="Sunday, Closed, Copy open hours">
<div class="OyjIsf "></div>
<div class="GYnni"></div>
</button>
</td>
</tr>
<tr class="y0skZc">
<td class="ylH6lf ">
<div>Monday</div>
</td>
<td class="mxowUb" role="text" aria-label="10 pm to 12 am">
<ul>
<li class="G8aQO">10 pm–12 am</li>
</ul>
</td>
<td class="HuudEc">
<button class="mWUh3d" data-hide-tooltip-on-mouse-move="true" data-value="Monday, 10 pm–12 am" jsaction="pane.openhours.189.copy; ; focus:pane.focusTooltip; blur:pane.blurTooltip" data-tooltip="Copy open hours" aria-label="Monday, 10 pm to 12 am, Copy open hours">
<div class="OyjIsf "></div>
<div class="GYnni"></div>
</button>
</td>
</tr>
</tbody>
</table>
<div class="onOnO">
<div class="">
<button class="M77dve " aria-label="Suggest an edit to open hours" jslog="119438;track:click;mutable:true;metadata:WyIwYWhVS0V3alpoY09QcFlHQkF4VkZTMndHSFZodUFFSVE4QmNJQmlnQiJd" jsaction="pane.wfvdle140">
<div class="BgrMEd cYlvTc VOgY4">
<div class="OyjIsf zemfqc"></div>
<span class="wNNZR fontTitleSmall">Suggest new hours</span>
</div>
</button>
</div>
</div>
</div>
</div>
<div class="z6qSc GUrTXd">
<div>
<div></div>
<div class="uLQcoc"></div>
</div>
</div>
</div>
<div class="RcCsl fVHpi w4vB1d NOE9ve M0S7ae AG25L ">
<div class="OyjIsf "></div>
<button class="CsEnBe" jsaction="pane.wfvdle120;clickmod:pane.wfvdle120" aria-label="Send Jio Center to your phone" data-item-id="send-to-device" jslog="141697; track:click; mutable:true;">
<div class="AeaXub">
<div class="cXHGnc">
<div class="Eottgc "><img class="Liguzb " alt="" src="//maps.gstatic.com/consumer/images/icons/2x/send_to_mobile_alt_gm_blue_24dp.png"></div>
</div>
<div class="rogA2c ">
<div class="Io6YTe fontBodyMedium kR99db ">Send to your phone</div>
<div class="gSkmPd fontBodySmall CuiGbf DshQNd"></div>
</div>
</div>
</button>
<div class="UCw5gc">
<div class="C9yzub"></div>
</div>
</div>
<div class="RcCsl fVHpi w4vB1d NOE9ve M0S7ae AG25L ">
<div class="OyjIsf "></div>
<a class="CsEnBe" jsaction="pane.wfvdle121;clickmod:pane.wfvdle121" aria-label="Claim this business" data-item-id="merchant" jslog="17658; track:click; mutable:true;metadata:WyIwYWhVS0V3alJydENGcFlHQkF4VVZabXdHSGVXZkNfd1FfMWtJSnlnUSJd" href="https://business.google.com/create?fp=9749311200610629361&hl=en&authuser=0&gmbsrc=in-en-et-ip-z-gmb-s-z-l~mrc%7Cclaimbz%7Cu&ppsrc=GMBMI&utm_campaign=in-en-et-ip-z-gmb-s-z-l~mrc%7Cclaimbz%7Cu&utm_source=gmb_mrc81&utm_medium=et&getstarted&lis=1">
<div class="AeaXub">
<div class="cXHGnc">
<div class="Eottgc "><img class="Liguzb " alt="" src="//www.gstatic.com/images/icons/material/system_gm/2x/verified_user_gm_blue_24dp.png"></div>
</div>
<div class="rogA2c ">
<div class="Io6YTe fontBodyMedium kR99db ">Claim this business</div>
<div class="gSkmPd fontBodySmall CuiGbf DshQNd"></div>
</div>
</div>
</a>
<div class="UCw5gc">
<div class="C9yzub"></div>
</div>
</div>
</div>
Code I used with CSS_SELECTORCSS_SELECTOR and CLASS_NAMECLASS_NAME both:
allDetailsXPATH = driver.find_element(By.XPATH,'//*[@id="QA0Szd"]/div/div/div[1]/div[3]/div/div[1]/div/div/div[2]/div[7]')
allDetails = allDetailsXPATH.find_elements(By.CSS_SELECTOR,'Io6YTe fontBodyMedium kR99db ')
print(allDetails)
The correct CSS_SELECTOR would be:
allDetails = allDetailsXPATH.find_elements(By.CSS_SELECTOR, 'div.Io6YTe.fontBodyMedium.kR99db')
Full code:
from selenium.webdriver import Chrome
from selenium.webdriver import ChromeOptions
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
options = ChromeOptions()
options.add_argument("--start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
driver = Chrome(options=options)
wait = WebDriverWait(driver, 10)
url = "https://www.google.co.in/maps/place/Jio+Center/@24.5537982,73.6833263,15z/data=!4m10!1m2!2m1!1sJio+Center!3m6!1s0x3967ef99640710a3:0x874c82e2052236f1!8m2!3d24.5537986!4d73.693626!15sCgpKaW8gQ2VudGVyIgOIAQGSARBjb3Jwb3JhdGVfb2ZmaWNl4AEA!16s%2Fg%2F11h9y566w1?entry=ttu"
driver.get(url)
allDetailsXPATH = wait.until(EC.presence_of_element_located((By.XPATH,'//*[@id="QA0Szd"]/div/div/div[1]/div[3]/div/div[1]/div/div/div[2]/div[7]')))
allDetails = allDetailsXPATH.find_elements(By.CSS_SELECTOR, 'div.Io6YTe.fontBodyMedium.kR99db')
print(allDetails)
print(len(allDetails))
output:
[<selenium.webdriver.remote.webelement.WebElement (session="5b2e1a6dc7d9b93befdd48746027a868", element="5B45CCB6FF176A5259BAD24EE4A9C1D0_element_574")>, <selenium.webdriver.remote.webelement.WebElement (session="5b2e1a6dc7d9b93befdd48746027a868", element="5B45CCB6FF176A5259BAD24EE4A9C1D0_element_591")>, <selenium.webdriver.remote.webelement.WebElement (session="5b2e1a6dc7d9b93befdd48746027a868", element="5B45CCB6FF176A5259BAD24EE4A9C1D0_element_597")>]
3