jsonloggingawkjsonpathblackboard

Need help in finding specific error in a BlackBoard log file


I am troubleshooting a problem with a clients installation of our BlackBoard plugin. Per our request, we have been given some json formatted log files. These are very hard to read.

I tried opening in an IDE, like VS Code, and they are too large for the application. I also downloaded the Apache Log Viewer, as I noticed some references to Tomcat in the logs. The viewer seems to be able to parse the files, but opening a file expect me to select a format, and I don't know enough about how to determine the log file format.

Here is an example line from a log file:

{"tags":["plugin
s"],"path":"/usr/local/blackboard/logs/plugins/bbgs-mbs/application.log","host":"ip-xx-xxx-xxx-xxx","message":"2019-10-07 05:02:16 | ERROR | 234:org.hibernate.util.JDBCExceptionReporter  | ERROR: relation \"bbgs_mbs_alerts\" does not exist","type":"plugins","@version":"1","@timestamp":"2019-10-07T09:02:16.715Z","clientid":"xxxxxxxxxxxxx"}

Ideally, all I am after is an easy way to load up the 4 files I have, sort by date, and try and find a correlation to the errors the client is reporting.

help me SO, you're my only hope


Solution

  • I adjusted the log files with a node script. Specifically, I did some string replacement on the file, parsed it to JSON, and wrote it back to a new file.

    I also used jsonpath npm package to write out just the property I need to a txt file.

    Here is the full solution if someone else may find it helpful (I know it is ugly, but was just after something that would work one time :/)

    // Make sure we got a filename on the command line.
    if (process.argv.length < 3) {
      console.log('Usage: node ' + process.argv[1] + ' FILENAME');
      process.exit(1);
    }
    
    // Read the file and print its contents.
    var  fs = require('fs')
      , jp = require('jsonpath')
      , filename = process.argv[2]
      , property = process.argv[3]
      , debug_mode = process.argv[4]
      , only_messages = process.argv[5];
    
    fs.readFile(filename, 'utf8', (err, data) => {
        if (err) throw err;
        const _newNameArr = filename.split('/');
        const _filename = _newNameArr[_newNameArr.length - 1];
        const replaced = data.replace(/[\n\r]/g, '');
        const replacedAgain = replaced.replace(/\}[\s]*\{/g, '},{');
        const _destFileName = (only_messages == 'true')
          ? `messages_${_filename}`
          : `${property || 'full'}_${_filename.replace('.txt', '.json')}`;
    
        if (debug_mode == 'true') {
          fs.writeFile('DEBUG-replaced_' + _destFileName, replacedAgain, (err) => {
            if (err) throw err;
            console.log('The file has been saved!');
          });
        }
        const parsed = JSON.parse(replacedAgain);
    
        const _fileContent = (only_messages == 'true')
          ? jp.query(parsed, '$..message').join('\n')
          : JSON.stringify(parsed, (property ? [property] : null), 1);
    
        fs.writeFile(_destFileName, _fileContent, (err) => {
          if (err) throw err;
          console.log(`The file, ${_destFileName}, has been saved!`);
        });
    
    });