[SOLVED] Scraping all possible tags and putting them into one variable using Go Colly

Scraping all possible tags and putting them into one variable using Go Colly

I need to scrape different tags from a list of sites, put in variable and then put them in a .csv list. For example, all lines where the author of the article is mentioned (div.author, p.author etc). On all sites, the location of this line and the tags are different, so I need to create a conditional and regular expression to filter that tags.

This is my code, where I find 1 possible author tag and append it to articleCollection. I tried if and for conditions, but can't put right variant it into author_name variable.

c.OnHTML("body", func(e *colly.HTMLElement) {
    author_name := e.DOM.Find("div.author").Text()

    if author_name == "" {
        log.Println("Author not found \n")
    }

    author := Authors{
        Author: author_name,
    }

    articleCollection = append(articleCollection, author)
})

Also, I tried implement condition like this for find all

with author class, but it didn't work, because author_name declared and not used :

if author_name == "" {
    author_name := e.DOM.Find("p.author").Text()
}

Thank you.

Solution

Use

if author_name == "" {
    author_name = e.DOM.Find("p.author").Text()
}

instead of

if author_name == "" {
    author_name := e.DOM.Find("p.author").Text()
}

Using := will allocate a new variable, and in your case, it is author_name, a new variable that is only valid within that if block. and you are not using it on anything after declaring the variable, that is why the error comes up