pythonpoint-cloudsopen3d

Show an image at some location on a point cloud using Open3D


So I have a point cloud which I can show on the screen using Open3D's examples. However, now I have an image and I need to show that image on a specific coordinate on the point cloud. I can't find any example about how to do this. Does anyone know how it can be done ? Thank you very much.


Solution

  • One way to display an image is by converting the image to a mesh. Here, we can use TriangleMesh or TetraMesh. I have used TriangleMesh here.

    A TriangleMesh is basically a list of vertices in 3D with a list of faces/traingles. Each traingle is a set of 3 vertex indices defining which vertex to connect.

    Now, imagine an image hanging in a 3D world. Each pixel of this image is a vertex for the mesh. And you can define triangles by connecting positions (i,j), (i+1,j) and (i,j+1) to make one triangle and (i, j+1), (i+1,j) and (i+1,j+1) to make another triangle. Just make sure you skip the last row and last column, since there is no i+1 or j+1 for that.

    See below answer, which will download a Bunny Mesh and an Image from open3d public assets and generate a mesh from image.

    import numpy as np
    import open3d as o3d
    
    mesh = o3d.io.read_triangle_mesh(o3d.data.BunnyMesh().path)
    pcd1 = mesh.sample_points_poisson_disk(10000)
    
    img = o3d.io.read_image(o3d.data.JuneauImage().path)
    
    def convert_image_to_mesh(img: np.ndarray):
        img = np.asarray(img)
    
        # Instead of backprojecting, just convert img to an actual 3D plane with Z=0
    
        # Create 3D vertex for each pixel location
        xvalues = np.arange(img.shape[1])
        yvalues = np.arange(img.shape[0])
        x_loc, y_loc = np.meshgrid(xvalues, yvalues)
        z_loc = np.zeros_like(x_loc)
    
        # Scale down before making 3D vertices
        x_loc = x_loc / xvalues.shape[0]
        y_loc = y_loc / xvalues.shape[0]  # Keep aspect ratio same by dividing with same denominator. Now image width is 1 meter in 3d.
    
        vertices = np.stack((x_loc, y_loc, z_loc), axis=2).reshape(-1, 3)
    
        vertex_colors = img.reshape(-1, 3)/255.0
    
        # Create triangles between each pair of neighboring vertices
        # Connect positions (i,j), (i+1,j) and (i,j+1) to make one triangle and (i, j+1), (i+1,j) and (i+1,j+1) to make
        # another triangle.
        # Pixel (i,j) is in vertices array at location i + j*xvalues.shape[0]
    
        vertex_positions = np.arange(xvalues.size * yvalues.size)
        # Reshape into 2D grid and discard last row and column
        vertex_positions = vertex_positions.reshape(yvalues.size, xvalues.size)[:-1, :-1].flatten()
    
        # Now create triangles (keep vertices in anticlockwise order when making triangles)
        top_triangles = np.vstack(((vertex_positions+1, vertex_positions, vertex_positions+xvalues.shape[0]))).transpose(1, 0)
        bottom_triangles = np.vstack(((vertex_positions+1, vertex_positions, vertex_positions+xvalues.shape[0]))).transpose(1, 0)
    
        triangles = np.vstack((top_triangles, bottom_triangles))
    
        mesh: o3d.geometry.TriangleMesh = o3d.geometry.TriangleMesh(
            o3d.utility.Vector3dVector(vertices), o3d.utility.Vector3iVector(triangles)
        )
    
        mesh.vertex_colors = o3d.utility.Vector3dVector(vertex_colors)
        """
        Flip the y and z axis according to opencv to opengl transformation.
        See - https://stackoverflow.com/questions/44375149/opencv-to-opengl-coordinate-system-transform
        """
        mesh.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
        return mesh
    
    visualizer = o3d.visualization.Visualizer()
    visualizer.create_window()
    
    # Enable back face as we want to see image backside as well
    rendering_options = visualizer.get_render_option()
    rendering_options.mesh_show_back_face = True
    
    visualizer.add_geometry(pcd1)
    
    mesh = convert_image_to_mesh(img)
    mesh.scale(scale=0.1, center=pcd1.get_center())
    visualizer.add_geometry(mesh)
    visualizer.run()