visual-studio-2017nugetnuget-server

Nuget local feed slow search


I have setup a local Nuget folder \\servername\packages and configured Visual Studio to use it as source instead of https://api.nuget.org/v3/index.json. It works fine. However, search for a package (Example: Automapper) using "Manage NuGet packages for Solution" is very very slow. It eventually brings back the result though.

Is there anyway I can speed up the search? (Example by adding some kind of index).

Example of local Nuget folder:

\\servername\packages\automapper
\\servername\packages\microsoft.aspnetcore

Solution

  • When you push a package to nuget.org, or any other NuGet server, you may notice that the package is not available immediately. This is because the server has some kind of ingestion process. It might include virus scanning or other checks, but it also includes search indexing. So, when you upload a package, the server pre-processes search data, so that when it gets a search query, it can look up in a search index very quickly for results.

    local feed explanation

    When you put files in a "local" feed, there is no search index. When doing a search, NuGet has to look at every version of every package in the feed. Even though the "v3" folder layout has the nuspec file extracted on disk, I'm not sure whether it's actually used. But certainly for local folders which just contain nupkg files, NuGet has to: Open the nupkg (zip) file, and start scanning the file backwards to find the zip file "central directory". Once the central directory is found, scan it forwards to find the metadata for the nuspec file (hopefully the operating system kept this data in its file cache). Once the zip metadata (data position, size, compression algorithm) for the nuspec file is found, seek to the data location of the file, and read the compressed bytes. Decompress the nuspec data. Parse the nuspec file as an XML file. Finally look for your search keyword in the parsed nuspec. Reminder, this is done for every version of every package in the feed.

    Although NuGet can (and I'm sure does) do this in parallel, your CPU has limited cores, and maybe your IO speed is a factor, so if you have many packages, it's always just going to take time. As mentioned about IO, when the "local" files are on a network share, then every IO operation has increased latency from network IO. If you're not convinced that IO latency is a big issue, I suggest writing a little program that has a class implementing Stream that does nothing more than write to the console every time read, seek, position, etc is called, and then call the same method on the real stream. Use this LoggingStream with ZipArchive to open a zip, find one specific file in the zip, and then read the zip. See how many methods on the Stream class are called, in particular Seek, since this drastically reduces the operating system's ability to pre-fetch data.

    solution/workaround

    Anyway, there is a solution, or workaround, depending on how much you (dis)like it. Use a local server. There are several listed on https://learn.microsoft.com/nuget/hosting-packages/overview.

    I know in some corporate environments that getting a web server is much more difficult than getting a network share, but it is what it is. Search can't be fast without a search index, and in my opinion the largest benefit of local feeds is the simplicity of just dropping in a file. If you have to run some tool to generate a search index for a local feed, then it's no simpler than pushing to a private server instead.