I'm working on scrapper with chromedp.
To get what i want (page html), i have to click a specific button.
So I used chromedp.click, and chromedp.outerhtml, but i only got html of page before click, not the html of page after click have done.
Can you see my code and advice me how to fix it?
func runCrawler(URL string, lineNum string, stationNm string) {
// settings for crawling
opts := append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", false))
// create chrome instance
contextVar, cancelFunc := chromedp.NewExecAllocator(context.Background(), opts...)
defer cancelFunc()
contextVar, cancelFunc = chromedp.NewContext(contextVar)
defer cancelFunc()
var htmlContent string
err := chromedp.Run(contextVar,
chromedp.Navigate(URL),
chromedp.WaitVisible(".end_footer_area"),
chromedp.Click(".end_section.station_info_section > div.at_end.sofzqce > div > div.c10jv2ep.wrap_btn_schedule.schedule_time > button"),
chromedp.OuterHTML("html", &htmlContent, chromedp.ByQuery),
)
fmt.Println("html", htmlContent)
checkErr(err)
i also give you homepage and button i need to click
Page URL: https://pts.map.naver.com/end-subway/ends/web/11321/home
Button Area I need to click:
Thank you very much
The page you want to get is open in a new tab (target).
In this case, we can use chromedp.WaitNewTarget to create a chan from where we can receive the target id of the new tab. Then create a new context with the chromedp.WithTargetID option so that we can connect to the new tab. From here everything is what you are already familiar with.
package main
import (
"context"
"fmt"
"strings"
"github.com/chromedp/cdproto/target"
"github.com/chromedp/chromedp"
)
func main() {
opts := append(chromedp.DefaultExecAllocatorOptions[:],
chromedp.Flag("headless", false),
)
ctx, cancel := chromedp.NewExecAllocator(context.Background(), opts...)
defer cancel()
ctx, cancel = chromedp.NewContext(ctx)
defer cancel()
var htmlContent string
ch := chromedp.WaitNewTarget(ctx, func(i *target.Info) bool {
return strings.Contains(i.URL, "/timetable/web/")
})
err := chromedp.Run(ctx,
chromedp.Navigate("https://pts.map.naver.com/end-subway/ends/web/11321/home"),
chromedp.WaitVisible(".end_footer_area"),
chromedp.Click(".end_section.station_info_section > div.at_end.sofzqce > div > div.c10jv2ep.wrap_btn_schedule.schedule_time > button"),
)
if err != nil {
panic(err)
}
newCtx, cancel := chromedp.NewContext(ctx, chromedp.WithTargetID(<-ch))
defer cancel()
if err := chromedp.Run(newCtx,
chromedp.WaitReady(".table_schedule", chromedp.ByQuery),
chromedp.OuterHTML("html", &htmlContent, chromedp.ByQuery),
); err != nil {
panic(err)
}
fmt.Println("html", htmlContent)
}