I'm trying to do simple metric/observability in my Spring Boot application using Micrometer, Prometheus and visualizing that metric on Grafana. I have created minimal example on GitHub. Essentially, I will have a scheduled job, which will be making HTTP request to some other service. I need to do 20k of those requests during single hour. I want to let it run, and measure what was the mean/average rate of request duration during that single hour.

As I understood from reading the docs, Grafana is natively capable to issue PromQL queries to Prometheus, which scrapes data from Actuator. Not sure exactly what Micrometer's role is, but from my understanding it acts as a facade in case we use deferent type telemetry systems, other the Prometheus.

So there are two problems:

Problem 1

I added following bean and annotated method which makes HTTP to external service. Will be using Feign (as I see, Micrometer also "tracks" Feign, need to look further for that config):

@Bean
public TimedAspect timedAspect(MeterRegistry registry) {
    return new TimedAspect(registry);
}

and my service method:

@Service
@Slf4j
public class MyService {

@Timed("email.time")
public EmailMessageResponseDto sendEmail(EmailMessageRequestDto emailMessageRequestDto){
        //TODO: This is the place where Feign Client will be used to send email
        return null;
    }
}

When I make a request to my endpoint (http://localhost:8089/sendEmails) which invokes sendEmail method, I don't see email_time on Prometheus's metric at http://localhost:8089/management/prometheus.

What I don't also understand is how I can visualize email_time on Grafana?

Problem 2

I cannot see anything on Grafana's dashboard. I can tell that datasource in Grafana is correctly set. Grafana is on http://localhost:3000/ (credentials: admin admin). I see different dashboards are available but there's no any metrics. So the file is in docker/grafana/provisioning/dashboards/spring-boot-statistics.json is being picked up. I found it on Grafana marketplace.

If is of any significance, Boot version is 2.4.4 and Java is 1.8. Our project is currently on those versions.

This sample project might have everything you want: https://github.com/jonatan-ivanov/teahouse

It contains 3 Spring Boot apps that talk to each other, they use Prometheus, it has Grafana dashboards and a load generator to generate traffic, you can see it in action here: https://www.youtube.com/watch?v=HQHuFnKvk_U

Problem 1

In newer versions of Boot (3.1+) you don't need to add a TimedAspect @Bean but you need spring-boot-starter-aop. Also, I would recommend using @Observed instead.

Problem 2

I think Prometheus is not scraping your service, if you open the Prometheus UI go to the "targets" menu and see if your app is there and OK. You need to change the Prometheus config to add your app, you can check the Prometheus docs or the Teahouse example above.