geom_scatterpie with non-numeric axes

I want to have a species x sample (both strings/factors) scatterplot with piecharts instead of points. The size of the points shall be correlated to the abundance of the species in each sample. This can be easily done with just points as this:

d <- data.frame(Tax=c("A", "B", "C"), Sample=c("01", "02", "03"))
d$A <- abs(rnorm(3, sd=1))
d$B <- abs(rnorm(3, sd=2))
d$size=c(0.1,0.2,0.3)
library(ggplot2)
ggplot(d,aes(x=Tax, y=Sample, size=size)) + geom_point()

To replace the points with piecharts can be achieved with geom_scatterpie of the scatterpie package (available on CRAN)

However, this does not work with factors in the x/y aesthetics:

library(scatterpie)
ggplot() + geom_scatterpie(aes(x=Tax, y=Sample, r=size), data=d, cols=c("A", "B")) 

Warning:
Removed 6 rows containing non-finite values (stat_pie). 

The panel is drawn, but stays empty. Note that scatterpie works well with numeric x/y aesthetics:

d <- data.frame(x=c(1,2,3), y=c(1,2,3))
d$A <- abs(rnorm(3, sd=1))
d$B <- abs(rnorm(3, sd=2))
d$size=c(0.1,0.2, 0.3)
ggplot() + geom_scatterpie(aes(x=x, y=y, r=size), data=d, cols=c("A", "B")) + coord_fixed()

How can i change geom_scatterpie to accept non-numeric axes?

This should work:

d2 <- d %>% 
  mutate(tax_num = as.numeric(as.factor(Tax)), 
         sample_num = as.numeric(as.factor(Sample)))

ggplot() + geom_scatterpie(data=d2, aes(x=tax_num, y=sample_num, r=size), cols=c("A", "B")) + 
  scale_x_continuous(breaks=c(1,2,3), labels=c("A", "B", "C")) + 
  scale_y_continuous(breaks=c(1,2,3), labels=c("01", "02", "03")) + 
  labs(x="Tax", y="Sample") + 
  coord_fixed()