How do I control the alpha of a geom_point fill?

I have a dataframe with two variables, that I want to make a scatterplot of in ggplot. I would like the color and fill to be controlled by "A" and the alpha of only the fill by "B" is that possible? If I include alpha in the aes like below it makes both the color (outline) and fill transparent.

set.seed(123)
data <- data.frame(
  X = rnorm(100),
  Y = rnorm(100),
  A = sample(letters[1:5], 100, replace = TRUE),
  B = sample(letters[1:2], 100, replace = TRUE
)

ggplot(data, aes(x = X, y = Y, color = A, fill = A, alpha=B)) +
  geom_point(shape = 21, size = 4) +  # Use shape 21 for filled circles
  theme_minimal()

Alpha is working for both fill and colour, try to replot without fill:

ggplot(data, aes(x = X, y = Y, color = A, fill = A, alpha = B)) +
  geom_point(shape = 21, size = 4) +
  geom_point(aes(X, Y, color = A), shape = 21, size = 4,
             data = data, inherit.aes = FALSE) +
  theme_minimal()