Why can't I debug my nuget package with sourcelink?

I've enabled source link in my csproj and used github actions to publish my nuget, but adding PublishRepositoryUrl property and referencing Microsoft.SourceLink.GitHub package.

However, when I reference the package, I cannot step into the code nor can I go to definition:

<Project Sdk="Microsoft.NET.Sdk.Razor">

  <PropertyGroup>
    <TargetFramework>net5.0</TargetFramework>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <PackageProjectUrl>https://github.com/Liero/vNext.BlazorComponents</PackageProjectUrl>
    <PublishRepositoryUrl>true</PublishRepositoryUrl>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0" PrivateAssets="all" />
  </ItemGroup>

</Project>

What am I missing? All the source codes can be found here https://github.com/Liero/vNext.BlazorComponents

EDIT: When I run a project that references the package, it looks like symbos are loaded based on debug output:

'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\5.0.3\Microsoft.AspNetCore.Components.Web.dll'. Symbol loading disabled by Include/Exclude setting.
'iisexpress.exe' (CoreCLR: clrhost): Loaded 'C:\Projects\liero\BlazorComponents\BlazorApp1\bin\Debug\net5.0\vNext.BlazorComponents.dll'. 

I have also disabled just my code debugging in Visual Studio.

If somebody wanted to try, here are steps to reproduce:

1. Create new Blazor project in Visual Studio

2. Add package reference to vNext.BlazorComponents

3. modify index.razor:

    @page "/"
    @using vNext.BlazorComponents.Grid
   
    <SimpleGrid TRow="object" @ref="grid"></SimpleGrid>
   
    @code {
        SimpleGrid<object> grid;
   
            protected override void OnAfterRender(bool firstRender)
            {
                base.OnAfterRender(firstRender);
                grid.Refresh(); // put breakpoint here and step into
            }
    }
   

4. Run the app

<PublishRepositoryUrl>true</PublishRepositoryUrl>

and

<PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0" PrivateAssets="all" />

is not enough. I had to add also:

<PropertyGroup>
  <EmbedUntrackedSources>true</EmbedUntrackedSources>
  <DebugType>embedded</DebugType>
</PropertyGroup>  

<PropertyGroup Condition="'$(GITHUB_ACTIONS)' == 'true'">
  <ContinuousIntegrationBuild>true</ContinuousIntegrationBuild>
</PropertyGroup>

then I was able to step into the nuget sources downloaded from github