gogo-colly

how to ignore printing Max depth limit reached go colly


i have a go colly crawler that i am trying to crawl many sites . on my terminal it prints a lot of :

2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached
2023/05/30 02:22:56 Max depth limit reached

it makes it hard for me to read some prints that i have placed . i wanted to know if there is any way to ignore printing this in termial . thanks


Solution

  • Max depth limit reached is colly.ErrMaxDepth. There must be codes like this in your project:

    c := colly.NewCollector(colly.MaxDepth(5))
    
    // ...
    
    if err := c.Visit("http://go-colly.org/"); err != nil {
        log.Println(err)
    }
    

    If you don't want to log this error, add a simple check to exclude it:

    c := colly.NewCollector(colly.MaxDepth(5))
    
    // ...
    
    if err := c.Visit("http://go-colly.org/"); err != nil {
        // log the error only when the error is not ErrMaxDepth.
        if err != colly.ErrMaxDepth {
            log.Println(err)
        }
    }
    

    Another option is to redirect the output to a file:

    go run . 2>&1 >log.txt
    

    or copy the output to a file and also to standard output with tee:

    go run . 2>&1 | tee log.txt