gomicroservicesgrpcconsul

consul health check (All service checks failing)


I have read the similar question related to this problem. It ended up with frustration.

  1. I installed consul with docker. I run the following command. docker run --name consul -d -p 8500:8500 consul, then I tested if the registration of service works using postman PUT method, but didn't enable health check, the service register into consul successfully in the consul web UI. For now everything works as expected.
  2. I have a local GRPC service written in go. The local service address is 127.0.0.1:8880. The address of consul server is 127.0.0.1:8500. Then the service is registered into consul, but health check didn't work, gets All service failing. Then I tried following:
check := &api.AgentServiceCheck{
      HTTP:                           "http://127.0.0.1:1010/health",
      Timeout:                        "5s",
      Interval:                       "5s",
      DeregisterCriticalServiceAfter: "15s",
   }
Node          Address         Status  Type    Build   Protocol  DC   Segment
aae2e6ac1ff8  127.0.0.1:8301  alive   server  1.10.3  2         dc1  <all>

Updates:

check := &api.AgentServiceCheck{
      GRPC:                           "127.0.0.1:8880",
      Timeout:                        "5s",
      Interval:                       "5s",
      DeregisterCriticalServiceAfter: "10s",
   }

The code register grpc service to consul server and enable health check is following:

func main() {
   IP := flag.String("IP", "127.0.0.1", "IP address")
   Port := flag.Int("Port", 8880, "Port")
   flag.Parse()


   // initialize logger, configuation file and database
   initialize.InitLogger()
   initialize.InitConfig()
   initialize.InitDB()
   zap.S().Info(global.ServerConfig)
   zap.S().Info("IP: ", *IP)
   zap.S().Info("Port: ", *Port)

  
   server := grpc.NewServer()
   proto.RegisterUserServer(server, &handler.UserServer{})
   l, err := net.Listen("tcp", fmt.Sprintf("%s:%d", *IP, *Port))
   if err != nil {
      panic("failed to listen" + err.Error())
   }

   // register health check
   grpc_health_v1.RegisterHealthServer(server, health.NewServer())

   cfg := api.DefaultConfig()
  
  // get cfg.address from configuration file, in which host is 127.0.0.1 port 8880
   cfg.Address = fmt.Sprintf("%s:%d", global.ServerConfig.ConsulInfo.Host,
      global.ServerConfig.ConsulInfo.Port)
   client, err := api.NewClient(cfg)
   if err != nil {
      panic(err)
   }

   // generate health check instance
   check := &api.AgentServiceCheck{
      GRPC:                           "127.0.0.1:8880/health",
      Timeout:                        "5s",
      Interval:                       "5s",
      DeregisterCriticalServiceAfter: "15s",
   }
   registration := new(api.AgentServiceRegistration)
   registration.Name = global.ServerConfig.Name
   registration.ID = global.ServerConfig.Name
   registration.Address = "127.0.0.1"
   registration.Port = 8880
   registration.Tags = []string{"user-srv", "user"}
   registration.Check = check

   err = client.Agent().ServiceRegister(registration)
   if err != nil {
      panic(err)
   }

   err = server.Serve(l)
   if err != nil {
      panic("failed to start grpc" + err.Error())
   }

Solution

  • Based on the documentation of the Consul API (https://www.consul.io/api-docs/agent/check#grpc), it seems that the GRPC field that you're using

    supports the standard gRPC health checking protocol.

    Therefore, you only need to define the gRPC endpoint there, and the protocol will handle the health checks.

    You can opt-out from this to use an HTTP endpoint for health checks by defining the HTTP field in your Agent Service Check config. You can find more information about that in the documentation (https://www.consul.io/api-docs/agent/check#http)