I am trying to implement OAuth2 authentication for an App in a Spring Boot backend using Spring Authorization Server. My problem is that I have custom logic that uses its own AuthenticationProviders, yet Spring Authentication Server automatically adds a whole bunch of default providers which can't handle my custom logic, and then crash. What I want is to figure out how I can get rid of the default AuthenticationProviders so that no crashes will occur.

Now, this is very abstract, so here's the concrete thing. In my own refresh token AuthenticationProvider, I have a bit that checks if the device_id (a custom field we use) matches the saved device_id and throws an OAuth2AuthenticationException if it doesn't.

        if (!providedDeviceId.equals(storedDeviceId)) {
            String errorMessage = "Provided device ID '" + providedDeviceId
                + "' did not match stored device ID '" + storedDeviceId
                + "' for refresh token '" + refreshTokenValue + "'";
            log.warn(errorMessage);
            throw new OAuth2AuthenticationException(
                new OAuth2Error(OAuth2ErrorCodes.INVALID_GRANT, errorMessage, null)
            );
        }

That logic works just fine. However, in the framework class org.springframework.security.authentication.ProviderManager, the authentication method works like this:

    @Override
    public Authentication authenticate(Authentication authentication) throws AuthenticationException {
        [...]
        for (AuthenticationProvider provider : getProviders()) {
            if (!provider.supports(toTest)) {
                continue;
            }
            [...]
            try {
                result = provider.authenticate(authentication);
                if (result != null) {
                    copyDetails(authentication, result);
                    break;
                }
            }
            catch (AccountStatusException | InternalAuthenticationServiceException ex) {
                prepareException(ex, authentication);
                // SEC-546: Avoid polling additional providers if auth failure is due to
                // invalid account status
                throw ex;
            }
            catch (AuthenticationException ex) {
                lastException = ex;
            }
        }

That is, it iterates over all AuthenticationProviders that support an authentication type, and only breaks if it succeeds, or an AccountStatusException or InternalAuthenticationServiceException is thrown. Note that it does not break on an AuthenticationException, which is how I would have expected it to work. Instead, it continues iterating over the remaining AuthenticationProviders that support an authentication type. And that leads me to my problem:

getProviders() returns not one but two providers that nominally support OAuth2RefreshTokenAuthenticationToken: My own custom OAuth2RefreshTokenAuthProvider, and the default org.springframework.security.oauth2.server.authorization.authentication.OAuth2RefreshTokenAuthenticationProvider. On the plus side, my own provider is called first and correctly throws its OAuth2AuthenticationException if the device_id mismatches. Unfortunately, however, the Spring OAuth2RefreshTokenAuthenticationProvider is called next, and runs into this error:

java.lang.IllegalArgumentException: value cannot be null
    at org.springframework.util.Assert.notNull(Assert.java:181)
    at org.springframework.security.oauth2.server.authorization.token.OAuth2TokenContext$AbstractBuilder.put(OAuth2TokenContext.java:219)
    at org.springframework.security.oauth2.server.authorization.token.OAuth2TokenContext$AbstractBuilder.principal(OAuth2TokenContext.java:152)
    at org.springframework.security.oauth2.server.authorization.authentication.OAuth2RefreshTokenAuthenticationProvider.authenticate(OAuth2RefreshTokenAuthenticationProvider.java:171)
    at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:182)
    at org.springframework.security.authentication.ObservationAuthenticationManager.lambda$authenticate$1(ObservationAuthenticationManager.java:54)
    at io.micrometer.observation.Observation.observe(Observation.java:564)
    at org.springframework.security.authentication.ObservationAuthenticationManager.authenticate(ObservationAuthenticationManager.java:53)

I could get into detail as to why this Assert.notNull fails. The short version is that the Spring OAuth2RefreshTokenAuthenticationProvider can't check the device_id because it's a custom field, and that causes issues further down the line. But really, that is not the point. The main issue in my eyes here is that the Spring OAuth2RefreshTokenAuthenticationProvider is called at all. What I want is that my custom OAuth2RefreshTokenAuthProvider is called instead, and not in addition to the default provider, since that's what I wrote it for.

And that wraps us back to my initial question: How can I configure Spring Authorization Server to not add the default providers?

Addendum

My OAuth2Config looks like this:

@Configuration
public class OAuth2Config {
    @Bean
    public SecurityFilterChain oAuth2FilterChain(
        HttpSecurity http,
        OAuth2PasswordGrantAuthProvider passwordAuthProvider,
        OAuth2RefreshTokenAuthProvider refreshTokenAuthProvider,
        PasswordGrantAuthenticationConverter passwordGrantAuthenticationConverter,
        RefreshTokenAuthenticationConverter refreshTokenAuthenticationConverter
    ) throws Exception {
        OAuth2AuthorizationServerConfigurer configurer =
            new OAuth2AuthorizationServerConfigurer();

        configurer.tokenEndpoint(token -> token
            .accessTokenRequestConverter(
                new DelegatingAuthenticationConverter(List.of(
                    passwordGrantAuthenticationConverter,
                    refreshTokenAuthenticationConverter
                )))
            .authenticationProvider(passwordAuthProvider)
            .authenticationProvider(refreshTokenAuthProvider)
        );

        http
            .securityMatcher("/oauth2/**")
            .with(configurer, (authorizationServer) ->
                authorizationServer.oidc(Customizer.withDefaults()))
            .authorizeHttpRequests(
                auth -> auth
                    .requestMatchers("/oauth2/token").permitAll()
                    .anyRequest().authenticated())
            .csrf(csrf -> csrf.ignoringRequestMatchers("/oauth2/**"));

        return http.build();
    }

    @Bean
    public RegisteredClientRepository registeredClientRepository(
        OAuth2Properties oAuth2Properties
    ) {
        TokenSettings tokenSettings = TokenSettings.builder()
            .accessTokenTimeToLive(Duration.ofMinutes(
                oAuth2Properties.getFirstScope().getAccessTokenTimeToLiveInMinutes()))
            .refreshTokenTimeToLive(Duration.ofDays(
                oAuth2Properties.getFirstScope().getRefreshTokenTimeToLiveInDays()))
            .reuseRefreshTokens(false)
            .build();
        RegisteredClient firstClient = RegisteredClient.withId(
                UUID.randomUUID().toString())
            .clientId(MyOAuth2.FIRST_CLIENT_ID)
            .clientAuthenticationMethod(ClientAuthenticationMethod.NONE)
            .authorizationGrantType(AuthorizationGrantType.PASSWORD)
            .authorizationGrantType(AuthorizationGrantType.REFRESH_TOKEN)
            .scope(MyOAuth2.FIRST_SCOPE)
            .tokenSettings(tokenSettings)
            .build();
        return new InMemoryRegisteredClientRepository(firstClient);
    }

    @Bean
    public OAuth2AuthorizationService authorizationService(
        JdbcTemplate jdbcTemplate,
        RegisteredClientRepository registeredClientRepository
    ) {
        return new JdbcOAuth2AuthorizationService(
            jdbcTemplate, registeredClientRepository);
    }
}

Thanks to the comment of @ChinHuang I was able to figure out how to do it. Here is how the `OAuth2Config` needs to be configured in order to exclude a default provider:

@Configuration
public class OAuth2Config {
    @Bean
    public SecurityFilterChain oAuth2FilterChain(
        HttpSecurity http,
        OAuth2PasswordGrantAuthProvider passwordAuthProvider,
        OAuth2RefreshTokenAuthProvider refreshTokenAuthProvider,
        PasswordGrantAuthenticationConverter passwordGrantAuthenticationConverter,
        RefreshTokenAuthenticationConverter refreshTokenAuthenticationConverter
    ) throws Exception {
        OAuth2AuthorizationServerConfigurer configurer = new OAuth2AuthorizationServerConfigurer();

        configurer.tokenEndpoint(token -> token
            .accessTokenRequestConverter(
                    new DelegatingAuthenticationConverter(List.of(
                            passwordGrantAuthenticationConverter,
                            refreshTokenAuthenticationConverter
                    )))
            .authenticationProviders(providers -> {
                providers.removeIf(OAuth2RefreshTokenAuthenticationProvider.class::isInstance);
                providers.add(passwordAuthProvider);
                providers.add(refreshTokenAuthProvider);
            })
        );

        http.securityMatcher("/oauth2/**")
            .with(configurer, (authorizationServer) ->authorizationServer.oidc(Customizer.withDefaults()))
            .authorizeHttpRequests(auth -> auth
                .requestMatchers("/oauth2/token").permitAll()
                .anyRequest().authenticated())
            .csrf(csrf -> csrf.ignoringRequestMatchers("/oauth2/**"));

        return http.build();
    }
}

The key part is the providers.removeIf inside the authenticationProviders.