javascriptros

Decoding ROS messages from ROSBridge

How can I decode rosbridge data in the browser?

So far I've been able to decode the following types:

My Problem now is decoding compressed depth and PointCloud2 data. As far as my understanding goes, the data is encoded as base64. The depth image has been compressed to a mono16 PNG. I have tried many different approaches, but none seem to work. The depth image is supposed to contain 307200 depth values of 16 bits each.

I do not want to display this data in something like ros3djs or webviz (cloud not figure out how they do the decoding). I want to decode the data and use it in my own analysis.

Steps to reproduce:

Here is a sample file. It contains the data field of the JSON message: https://drive.google.com/file/d/18ZPpWrH9TKtPBbevfGdceZVpmmkiP4bh/view?usp=sharing

OR

  1. Make sure you have a device publising or a rosbag playing
  2. roslaunch rosbridge_server rosbridge_websocket.launch
  3. Launch your web page and make sure you added roslibjs in your script tag

The JS of my web page is simplified to this:

var ros = new ROSLIB.Ros({
    url: 'ws://127.0.0.1:9090'
});


var depthListener = new ROSLIB.Topic({
    ros: ros,
    name: '/camera/color/image_raw/compressedDepth',
    messageType: 'sensor_msgs/CompressedImage'
});

var pointCloudListener = new ROSLIB.Topic({
    ros: ros,
    name: '/camera/depth/color/points',
    messageType: 'sensor_msgs/PointCloud2'
});



depthListener.subscribe(function (message) {
    console.log(message);
    depthListener.unsubscribe();
});

pointCloudListener.subscribe(function (message) {
    console.log(message);
    pointCloudListener.unsubscribe();
});

I have set the two topics to unsubscribe after the first message so that my console does net get flooded.

Provided a screenshot of the console logs for depth image

Screenshot of depth image output

and for pointcloud

screenshot of pointcloud console log

This is what I have so far, but the onload function is never triggered.

image = new Image();
    image.src = "data:image/png;base64, " + message.data

    image.onload = function(){
        image.decode().then(() =>{
            if(image.width != 0 && image.height != 0){
                canvas.width = image.width;
                canvas.height = image.height;
                ctx = canvas.getContext('2d');
                ctx.drawImage(image, 0, 0);
                image_data = canvas.getContext('2d').getImageData(0,0, 640,480).data;
            }
        });
    }

I think this OpenCV code was used to compress the image. The message can essentially be thought of as a 16 bit gray scale image.

Please comment if I can update the question with specific information

Solution

  • According to libpng a PNG starting signature is

     89  50  4e  47  0d  0a  1a  0a

    As pointed here, the signature is present after a header (few initial 00 bytes in your case). This would solve your problem:

    function extractPng(base64) {
  // const signature = '\x89\x50\x4e\x47\x0d\x0a\x1a\x0a';
  const signature = '\x89PNG\r\n\x1a\n';
  let binary = window.atob(base64);
  let ix = binary.indexOf(signature);
  ix = ix > 0 ? ix : 0;
  return 'data:image/png;base64,' + window.btoa(binary.substring(ix));
}

    Your image

    [640 x 480]

    As you can see there are some very dark gray areas

    ROSBRIDGE depth image

    Contrasted version

    enter image description here

    Complete Example

    This is a complete example that displays you the image too:

    function openDataImage(data) {
  const image = new Image();
  image.src = data;
  const w = window.open("");
  w.document.write(image.outerHTML);
}

openDataImage(extractPng(`...`));

    Complete Decoding

    Unfortunately <canvas> has a 8bit fixed color depth, hence it can not be used to access your 16bit grayscale data. I suggest using pngjs. Pngjs is not available (at least I have not found it) as a compiled browser-ready library so you will need to package your 'website' somehow (like with Webpack).

    The function will need to extract the png binary data as a Buffer:

    function extractPngBinary(base64) {
  const signature = Buffer.from("\x89PNG\r\n\x1a\n", "ascii");
  let binary = Buffer.from(base64, "base64");
  let ix = binary.indexOf(signature);
  ix = ix > 0 ? ix : 0;
  return binary.slice(ix);
}

    Then to decode the png:

    const PNG = require("pngjs").PNG;
const png = PNG.sync.read(extractPngBinary(require("./img.b64")));

    To read values out of the PNG then (encoding is BE):

    function getValueAt(png, x, y) {
  // Check is Monotchrome 16bit
  if (png.depth !== 16 || png.color || png.alpha) throw "Wrong PNG color profile";
  // Check position
  if (x < 0 || x > png.width || y < 0 || y > png.height) return undefined;
  // Read value and scale to [0...1]
  return (
    png.data.readUInt16BE((y * png.width + x) * (png.depth / 8)) /
    2 ** png.depth
  );
}

    To read a region of data then:

    function getRegion(png, x1, x2, y1, y2) {
  const out = [];
  for (let y = y1; y < y2; ++y) {
    const row = [];
    out.push(row);
    for (let x = x1; x < x2; ++x) {
      row.push(getValueAt(png, x, y));
    }
  }
  return out;
}

    All the image:

    getRegion(png, 0, png.width, 0, png.height);

    Here a complete example (with source code)

    The button "Decode Image" will decode the depths of a small area.

    enter image description here