Change default colors in ggpairs using variable assignments

I have code to create a scatterplot matrix with density curves that I want to be colored based on a categorical variable in the data set. I need it to match a specific color, but I can't seem to get the colors to update from the default.

Below is an example of the concept I'm trying to accomplish with a well-known dataset (since my data has sensitive information and can't be posted).

For example if I wanted to create this with the crabs dataset in R, I would be assigning categorical variables to a color and symbol like:

species <- ifelse(crabs$sp == "B", "blue", "orange")
gender <- ifelse(crabs$sex == "M", "O", "+")

Then I would want to have the exact same symbols and colors in my matrix and desity plots:

ggpairs(crabs, columns=4:8, aes(color=species, shape=gender),
        lower=list(continuous="smooth"), diag=list(continuous="densityDiag"))

However, this outputs the following:

But the coral color should be blue, and the teal color should be true orange.

You may need to specify the colours for each group using scale_color_manual() and scale_fill_manual(). Also the colour values must be same as the number of categories and not the length of the data:

species <- ifelse(crabs$sp == "B", "blue", "orange")
gender <- ifelse(crabs$sex == "M", "O", "+")

ggpairs(crabs, columns=4:8, aes(colour=species, shape=gender),
lower=list(continuous="smooth"), diag=list(continuous="densityDiag"))+
scale_color_manual(values = unique(species))+
scale_fill_manual(values = unique(species))