I am using logrus
to do all the logging my golang application. However, I also want to integrate this with Elastic Search such that all the logs are also flushed to elastic search when I create a logrus log entry. Currently all logs are created in a file as shown in the snippet below. How could I integrate with elastic search?
type LoggerConfig struct {
Filename string `validate:regexp=.log$`
AppName string `validate:regexp=^[a-zA-Z]$`
}
type AppLogger struct {
Err error
Logger logrus.Entry
}
func Logger(loggerConfig LoggerConfig) AppLogger {
response := new(AppLogger)
// validate the schema of the logger_config
if errs := validator.Validate(loggerConfig); errs != nil {
response.Err = errs
// this sets up the error on the the response struct
}
logrus.SetFormatter(&logrus.JSONFormatter{})
f, err := os.OpenFile(loggerConfig.Filename, os.O_WRONLY|os.O_CREATE, 0755)
if err != nil {
response.Err = err
}
multipleWriter := io.MultiWriter(os.Stdout, f)
logrus.SetOutput(multipleWriter)
contextLogger := logrus.WithFields(logrus.Fields{
"app": loggerConfig.AppName,
})
//logrus.AddHook(hook)
response.Logger = *contextLogger
//response.Logger.Info("adele")
return *response
}
I had tried elogrus
which adds a hook but I am not sure how to use it. Here is the method which attempts to create elastic search client. How could I integrate this with logrus instance?
func prepareElasticSearchClient() *elastic.Client {
indexName := "my-server"
client, _ := elastic.NewClientFromConfig(&config.Config{
URL: os.Getenv("ELASTIC_SEARCH_URL_LOGS") + ":" + os.Getenv("ELASTIC_SEARCH_PORT_LOGS"),
Index: indexName,
Username: os.Getenv("ELASTIC_SEARCH_USERNAME_LOGS"),
Password: os.Getenv("ELASTIC_SEARCH_PASSWORD_LOGS"),
})
return client
}
Earlier I have used modules like Winston
where it was super easy to setup elastic search logging but somehow, I find little documentation with golang on how to integrate Golang logging with elastic search
With elogrus
you first create Elastic client and pass it to elogrus
hook when creating it with elogrus.NewAsyncElasticHook()
. Hook just wraps sending message to Elastic. Then you add this hook to logrus
log
. Every time you log message using log
it will fire your hook and send message (if log level filter passes) to Elastic.
log := logrus.New()
client, err := elastic.NewClient(elastic.SetURL("http://localhost:9200"))
// ... handle err
hook, err := elogrus.NewAsyncElasticHook(client, "localhost", logrus.DebugLevel, "testlog")
// ... handle err
log.Hooks.Add(hook)
Signature of NewAsyncElasticHook
is (client *elastic.Client, host string, level logrus.Level, index string)
where:
client
is pointer to Elastic.Client
you obtained before using elastic
host
is string that denotes from which host you are sending the log trace (it's a string - hostname of host where program that logs is running)level
is the maximum logrus log level you want messages to be sent (e.g. if you want to see DEBUG messages locally but only send only ERROR and below to Elastic)index
is name of Elastic Search index you want to add messages from log
toFrom here you can use log
normally as you would do with logrus
and all messages will get passed to Elastic.
Another part of issue was a bit more tricky and rooted in (not only) Golang elastic
client node sniffing behavior. We debugged it in chat and summary was posted as my answer to OP's another question regarding that: Cannot connect to elastic search : no active connection found: no Elasticsearch node available