Model Validation .Net Core 2.0 Web Api Not working

I have a .Net Core 2.0 Web Api. I have various models with validation attributes on properties as such:

[Required]
public short? Quantity { get; set; }

I have an ActionFilter that checks model state:

if (!context.ModelState.IsValid)
     context.Result = new BadRequestObjectResult(context.ModelState);

No matter what I do the ModelState is always coming back as valid when I purposely omit the required properties. My controllers are marked as:

[Produces("application/json")]

The models are getting deserialized correctly and I have the models parameters in my action methods marked with [FromBody]. It just doesn't seem to be running any validation (standard or custom). I've looked at this answer and this one and several others but I just can't figure out what I'm missing. My APIs are protected with IdenityServer 4 so not sure if that plays into it but at this point I have to validate every action method myself which is not what I want to be doing. Anyone have suggestions?

So my issue appears to be because I'm using services.AddMvcCore() and not services.AddMvc() I have to explicitly set .AddDataAnnotations() where this is baked into AddMvc(). AddMvcCore() gives you bare bones and you add what you need where AddMvc() gives you everything whether you need it or not.

services
    .AddMvcCore()
    .AddAuthorization()
    .AddJsonFormatters()
    .AddApiExplorer()
    .AddDataAnnotations()
    .AddMvcOptions(opt =>
        opt.Filters.Add<RequestFilterAttribute>());