Need to plot line plot across one factor variable with two levels

I have a dataset that records details in milliseconds with a maximum of 20 seconds. I need to plot a value "Buffer" across a one-factor variable with two levels - A and B. I'm trying to plot a geom_line() with x as the time, y as the Buffer and two lines for A and B. My problem is that it plots a line for every observation and does not aggregate it by factor. Here is the code I was using:

ggplot(DT, aes(x = Real_Time_Stamp, y = Buffer)) + geom_line(aes(color = FVN))

And here is the plot it is generating:

The dataset I'm dealing with has 49,999 rows and 3 columns. Here is an example:

structure(list(FVN = c("A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A", "A"), Real_Time_Stamp = c(0.015233039855957, 0.0325429439544678, 0.0483760833740234, 0.0653512477874756, 0.0819132328033447, 0.0988430976867676, 0.11584997177124, 0.132710218429565, 0.148336172103882, 0.165808200836182, 0.182291269302368, 0.199646949768066, 0.215576171875, 0.233185052871704, 0.248784303665161, 0.266969203948975, 0.282114028930664, 0.299488067626953, 0.315442323684692, 0.332358121871948), Buffer = c(1.984, 1.968, 1.952, 1.936, 1.936, 1.952, 1.936, 1.92, 1.904, 1.888, 1.872, 1.856, 1.856, 1.872, 1.856, 1.856, 1.84, 1.84, 1.824, 1.824)), row.names = c(NA, 20L), class = "data.frame")

I've used the above code before to generate line plots based on factor levels. What am I doing wrong?

Based on @stefan's suggestion, I binned the real-time stamp variable. As they mentioned, there is a lot of variation in the Buffer variable and real-time stamp.

Here is the code I used:

ggplot(DT, aes(x = Real_Time_Stamp, y = Buffer)) + 
  geom_line(aes(color = FVN, group = FVN), stat = "summary") + 
  scale_x_binned(name = "\nTime (s)",n.breaks = 100, limits = c(0,20), breaks = seq(0,20, by = 1))+
  scale_y_continuous(name = "\nBuffer Values", limits = c(-0.5,2.5),breaks = seq(0,2, by = 1)))

And here is the plot I got: