stack bars by an ordering variable which is numeric ggplot

I am trying to create a swimlane plot of different subjects doses over time. When I run my code the bars are stacked by amount of dose. My issue is that subjects doses vary they could have 5, 10 , 5 in my plot the 5's are stacked together. But I want the represented as they happen over time. In my data set I have the amount of time each patient was on a dose for ordered by when they had the dose. I want by bars stacked by ordering variable called "p" which is numeric is goes 1,2,3,4,5,6 etc which what visit the subject had that dose.

ggplot(dataset,aes(x=diff+1, y=subject)) +
  geom_bar(stat="identity", aes(fill=as.factor(EXDOSE)))  +
  scale_fill_manual(values = dosecol, name="Actual Dose in mg")

I want the bars stacked by my variable "p" not by fill

I tried forcats but that does not work. Unsure how to go about this the data in the dataset is arranged by p for each subject

example data

dataset <- data.frame(subject = c("1002", "1002", "1002", "1002", "1034","1034","1034","1034"),
                         exdose = c(5,10,20,5,5,10,20,20),
                         p= c(1,2,3,4,1,2,3,4), 
                      diff = c(3,3,9,7,3,3,4,5)
)


ggplot(dataset,aes(x=diff+1, y=subject)) +
  geom_bar(stat="identity", aes(fill=as.factor(exdose)),position ="stack")  +
  scale_fill_manual(values = dosecol, name="Actual Dose in mg")

If you want to order your stacked bar chart by p you have to tell ggplot2 to do so by mapping p on the group aesthetic. Otherwise ggplot2 will make a guess which by default is based on the categorical variables mapped on any aesthetic, i.e. in your case the fill aes:

Note: I dropped the scale_fill_manual as you did not provide the vector of colors. But that's not important for the issue.

library(ggplot2)

ggplot(dataset, aes(x = diff + 1, y = subject, group = p)) +
  geom_col(aes(fill = as.factor(exdose)))

EDIT And to get the right order we have to reverse the order of the stack which could be achieved using position_stack(reverse = TRUE):

Note: To check that we have the right order I added a geom_text showing the p value.

ggplot(dataset, aes(x = diff + 1, y = subject, group = p)) +
  geom_col(aes(fill = as.factor(exdose)), position = position_stack(reverse = TRUE)) +
  geom_text(aes(label = p), position = position_stack(reverse = TRUE))

Second option would be to convert p to a factor which the order of levels set in the reverse order:

ggplot(dataset, aes(x = diff + 1, y = subject, group = factor(p, rev(sort(unique(p)))))) +
  geom_col(aes(fill = as.factor(exdose))) +
  geom_text(aes(label = p), position = "stack")