ggplot accessing data labels rather than data values

I've imported some SPSS data into R, the data is labelled and I want to plot the label names, rather than the values. I've tried the below, but it prints the values

library(labelled)
library(tidyverse)

cols <- c("White", "Red", "Black", "Green", "Brown", "Pink", "Orange")
letters <- c("D", "E", "F", "G", "H", "I", "J")

# add labels to the color values
tmp <- diamonds %>% 
 mutate(color = as.character(color)) %>%
 set_value_labels(color = setNames(letters, cols)) 

ggplot(tmp) + geom_bar(aes(x=print_labels(color)))

The graph still uses the color values as the y-axis, rather than the labels. How can we plot the dataframe labels?

library(haven)

cols <- c("White", "Red", "Black", "Green", "Brown", "Pink", "Orange")
letters <- c("D", "E", "F", "G", "H", "I", "J")

tmp <- diamonds %>% 
  mutate(color = as.character(color)) %>%
  set_value_labels(color = setNames(letters, cols)) 

ggplot(tmp) + geom_bar(aes(x=as_factor(color)))

Using the haven library, we can use the as_factor(x) command, which converts the labels into factors