I am building a new filebeat module for a custom application log and I wish to collaborate on it with a colleague of mine. I understood that a clone is a local copy for me only, without a chance for making my changes visible to my colleague without having them first merged to the https://github.com/elastic/beats/ repo, so I opted to fork the beats repository instead of cloning it and then cloned that fork.
I set up my dev environment by following the instructions. Running make
under filebeat gives no output so I assume everything is a-ok.
I generate the module, fileset, and set up my ingest pipeline for the fileset. My pipeline.json
looks like this:
{
"description": "Pipeline for parsing CA Service Desk Manager stdlogs",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIMESTAMP:casdm.stdlog.timestamp} %{HOSTNAME:casdm.stdlog.hostname} %{PROCESS:casdm.stdlog.process.name} %{PID:casdm.stdlog.process.id} %{LOGLEVEL:casdm.stdlog.level} %{FILENAME:casdm.stdlog.file.name} %{POSINT:casd.stdlog.file.line} %{DATA:casdm.stdlog.message}"
],
"pattern_definitions": {
"TIMESTAMP": "%{MONTHNUM2}/%{MONTHDAY} %{HOUR}:%{MINUTE}:%{SECOND}",
"PROCESS": "%{USERNAME}",
"PID": "%{POSINT}",
"LOGLEVEL": "(FATAL|EXIT|SIGNIFICANT|SEVERE_ERROR|ERROR|WARNING|INFORMATION|MILESTONE|TRACE|VERBOSE)",
"FILENAME": "(?:[A-Za-z0-9_. -]+)"
}
}
}
],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
It's not all the way there but being new to filebeat module creation I am excluding all the special cases of the log format for now.
I am then instructed to generate the fields based on the pipeline configuration. Everything looks good:
~/go/src/github.com/jvalkonen/beats/filebeat $ make create-fields MODULE=casdm FILESET=stdlog
Fields.yml generated for casdm/stdlog
However, no fields.yml
(regardless of the case) is found under the module/casdm/stdlog/_meta/
and the module level module/casdm/_meta/fields.yml
doesn't contain any of my fields set in the module/casdm/stdlog/ingest/pipeline.json
. This already indicates there is something wrong, but I don't know what as the output indicates all is well. If I then run the make update
to generate documentation and configuration I get an error, which doesn't seem to indicate issues with the pipeline.json
but the development setup itself and possibly the fact that I forked the project instead of just cloning it:
~/go/src/github.com/jvalkonen/beats/filebeat $ make update
mage update
Error: failed to find github.com/elastic/beats/dev-tools/mage in the project's vendor
failed to find github.com/elastic/beats/dev-tools/mage in the project's vendor
make: *** [update] Error 1
Something along the build pipeline is referring to the elastic github path instead of my forked repo. I could go search and replace all of those references but am I trying to do this completely backwards and is there an easier way to achieve what I'm trying to do?
My questions are:
Turns out my issue was a combination of being a git and go noob and either failing to read some instructions on how to setup the dev environment properly, or those instructions not being there.
In any case, this is what I needed to do (as instructed on this Elastic community thread:
After this I no longer get errors and so far all the commands I've needed to work have worked, including building filebeat from scratch.