Build ASP.NET Core on Windows Server (without Visual studio) for CI/CD

This is not duplicate of this thread, although goal is the same, but the error is not.

I want to build my ASP.NET Core on machine without Visual Studio installed.

I need this, because I am setting GitLab CI/CD.

Just for test purpose I open console I execute dotnet restore and dotnet build in folder where .sln file is. I get the following error in both cases:

C:\Program Files\dotnet\sdk\2.1.4\NuGet.targets(103,5): error : The local source 'C:\Users\matjaz.cof\.dotnet\NuGetFallbackFolder' doesn't exist.

Full log:

C:\GitLab-Runner\builds\7cab42e4\0\web\app>dotnet restore
  Restoring packages for C:\GitLab-Runner\builds\7cab42e4\0\web\app\App.Web.Core\App.Web.Core.csproj...
  Restoring packages for C:\GitLab-Runner\builds\7cab42e4\0\web\app\App.Web.Library.Tests\App.Web.Library.Tests.csproj...
  Restoring packages for C:\GitLab-Runner\builds\7cab42e4\0\web\app\App.Web.Core.Tests\App.Web.Core.Tests.csproj...
  Restoring packages for C:\GitLab-Runner\builds\7cab42e4\0\web\app\App.Web.Library\App.Web.Library.csproj...
  Restoring packages for C:\GitLab-Runner\builds\7cab42e4\0\web\app\App.Web.Hosting\App.Web.Hosting.csproj...
  Restore completed in 93,72 ms for C:\GitLab-Runner\builds\7cab42e4\0\web\app\App.Web.Hosting\App.Web.Hosting.csproj.
C:\Program Files\dotnet\sdk\2.1.4\NuGet.targets(103,5): error : The local source 'C:\Users\matjaz.cof\.dotnet\NuGetFallbackFolder' doesn't exist. [C:\GitLab-Runner\builds\7cab42e4\0\web\app\App.Web.sln]

Answer in provided thread says that all I need is Dotnet SDK, which I installed:

C:\GitLab-Runner\builds\7cab42e4\0\web\app>dotnet --version
2.1.4

Folder really doesn't exists on Windows server, but it exists on my development machine (with VS 2017 installed).
I search entire c: on Windows Server for folder NuGetFallbackFolder and found it in c:\Program Files\dotnet\sdk\NuGetFallbackFolder.

Why dotnet restore/build search for folder under username and what do I need to change?

Edited: (@MartinUllrich)

This is the content of c:\Users\matjaz.cof\AppData\Roaming\NuGet\NuGet.Config:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
  </packageSources>
</configuration>

I simply solved it by coping c:\Program Files\dotnet\sdk\NuGetFallbackFolder\ to c:\Users\matjaz.cof\.dotnet\NuGetFallbackFolder\.

I couldn't think of anything else. Maybe uninstalling old .NET Core runtimes and SDKs would also do the trick.