gostatic-analysis

golangci-lint - want to "really" ignore a go file not simply analyse it and suppress warnings - my current ignore approach eats memory and is slow

With golangci-lint I know I can use skip-dirs or skip-files settings but these just stop reporting at the end and I think the tool still does the "work", quoting docs :

... which files to skip: they will be analyzed, but issues from them won't be reported. ...

Is there way, say with a variant of //nolint to prevent some files getting analysed by golangci-lint in the first place so we don't waste build-time resources - memory/cpu - on them them ?

I also tried lines at the top like // Code generated by XXX. DO NOT EDIT. and these like //nolint stop reporting but I think golangci-lint is still doing the analysis - i.e. tool runs slow with high memory when big "ignored" files are present. From https://golangci-lint.run/usage/false-positives/#nolint-directive I can do

> //nolint:unparam    
> package pkg

And indeed no lint errors are reported BUT I still get excessive memory usage e.g

13:14 $ golangci-lint run --timeout 30m -v 
INFO [config_reader] Config search paths: [ REMOVED FOR STACKOVERFLOW POST] 
INFO [config_reader] Used config file .golangci.yml 
INFO [lintersdb] Active 17 linters: [deadcode dogsled errcheck gocyclo golint gosimple govet ineffassign misspell nakedret scopelint staticcheck structcheck typecheck unconvert unused varcheck] 
INFO [lintersdb] Active 17 linters: [deadcode dogsled errcheck gocyclo golint gosimple govet ineffassign misspell nakedret scopelint staticcheck structcheck typecheck unconvert unused varcheck] 
INFO [loader] Go packages loading at mode 575 (deps|imports|name|types_sizes|compiled_files|exports_file|files) took 15.003932542s 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 831.549179ms 
INFO [runner/unused/goanalysis] analyzers took 15.395924718s with top 10 stages: U1000: 13.12191427s, buildssa: 2.274010448s 
INFO [runner/goanalysis_metalinter/goanalysis] analyzers took 8m25.344883444s with top 10 stages: golint: 6m29.017594865s, ineffassign: 50.003663795s, vrp: 34.694968113s, misspell: 5.91752885s, buildssa: 4.09136586s, SA2003: 1.003697913s, copylocks: 660.780161ms, SA4000: 601.476823ms, SA4018: 510.871677ms, SA4004: 476.466048ms 
INFO [runner] Issues before processing: 28876, after processing: 0 
INFO [runner] Processors filtering stat (out/in): skip_files: 28876/28876, skip_dirs: 28876/28876, identifier_marker: 53/53, path_prettifier: 28876/28876, autogenerated_exclude: 53/28876, exclude: 0/53, filename_unadjuster: 28876/28876, cgo: 28876/28876 
INFO [runner] processing took 60.293318ms with stages: autogenerated_exclude: 33.592255ms, path_prettifier: 18.17708ms, skip_dirs: 4.421717ms, filename_unadjuster: 1.497968ms, cgo: 1.333012ms, identifier_marker: 1.036647ms, exclude: 220.939µs, nolint: 8.996µs, max_same_issues: 1.448µs, uniq_by_line: 592ns, exclude-rules: 544ns, max_from_linter: 467ns, skip_files: 386ns, diff: 362ns, path_shortener: 331ns, source_code: 309ns, max_per_file_from_linter: 265ns 
INFO [runner] linters took 6m47.422489108s with stages: goanalysis_metalinter: 6m28.298523575s, unused: 19.033707326s 
INFO File cache stats: 2 entries of total size 40.4MiB 
INFO Memory: 4114 samples, avg is 7797.4MB, max is 8629.2MB 
INFO Execution took 7m3.337365705s

When I remove the big files golangci-lint behaves fine.

Solution

  • Android's syzkaller uses a dedicated build tag to exclude files for golangci-lint. Using that approach prevents parsing of the files, and significantly reduced our memory consumption during build time.

    At top of the Go files that need to be efficiently skipped add // !codeanalysis e.g.

     // AUTOGENERATED FILE
 
 // +build !codeanalysis

 package foo

    In the .golangci.yml file ...

    run:
  deadline: 8m
  skip-dirs:
    - pkg/kd
  # Autogenerated files take too much time and memory to load,
  # even if we skip them with skip-dirs.
  # So we define this tag and use it in the autogenerated files.
  build-tags:
    - codeanalysis

    I tested this approach in our project, and the linting is no longer degrading performance.

    The following approaches appeared to work & avoided printing the lint errors, but they were actually still doing the unwanted linting so they were no go for us: