Add direct labels to geom_smooth rather than geom_line

I recognize that this question is a close duplicate of this one, but the solution there no longer works (using method="last.qp"), so I'm asking it again.

The basic issue is that I'd like to use directlabels (or equivalent) to label smoothed means for each group (from stat_smooth()), rather than the actual data. The example below shows as close as I've gotten, but the labels aren't recognizing the grouping, or even the smoothed line. Instead, I'm getting the label at the last point. What I'd like is colour-coordinated text at the end of each stat_smooth(), rather than the legend on the right of the plot. This post provides an approach for labelling the last data point (the behaviour I'm seeing), but I'm looking for an approach to label automatically-generated summaries, if possible.

Here's an example:

library(ggplot2)
library(directlabels)

## Data
set.seed(10)
d <- data.frame(x=seq(1,100,1), y=rnorm(100, 3, 0.5))
d$z <- ifelse(d$y>3,1,0)

## Plot
p <- ggplot(d, aes(x=x, y=y, colour=as.factor(z))) +
  stat_smooth(inherit.aes=T, se=F, span=0.8, show.legend = T) +
  geom_line(colour="grey50") +
  scale_x_continuous(limits=c(0,110)) +
  geom_dl(label="text", method="maxvar.points", inherit.aes=T)
p

which makes this plot:

A solution using ggrepel package based on this answer

library(tidyverse)
library(ggrepel)

set.seed(123456789)

d <- data.frame(x = seq(1, 100, 1), y = rnorm(100, 3, 0.5))
d$z <- ifelse(d$y > 3, 1, 0)

labelInfo <-
  split(d, d$z) %>%
  lapply(function(t) {
    data.frame(
      predAtMax = loess(y ~ x, span = 0.8, data = t) %>%
        predict(newdata = data.frame(x = max(t$x)))
      , max = max(t$x)
    )}) %>%
  bind_rows

labelInfo$label = levels(factor(d$z))
labelInfo

#>   predAtMax max label
#> 1  2.538433  99     0
#> 2  3.293859 100     1

ggplot(d, aes(x = x, y = y, color = factor(z))) + 
  geom_point(shape = 1) +
  geom_line(colour = "grey50") +
  stat_smooth(inherit.aes = TRUE, se = FALSE, span = 0.8, show.legend = TRUE) +
  geom_label_repel(data = labelInfo, 
                   aes(x = max, y = predAtMax, 
                       label = label, 
                       color = label), 
                   nudge_x = 5) +
  theme_classic()
#> `geom_smooth()` using method = 'loess' and formula 'y ~ x'

Created on 2018-06-11 by the reprex package (v0.2.0).