Show an image at some location on a point cloud using Open3D

So I have a point cloud which I can show on the screen using Open3D's examples. However, now I have an image and I need to show that image on a specific coordinate on the point cloud. I can't find any example about how to do this. Does anyone know how it can be done ? Thank you very much.

One way to display an image is by converting the image to a mesh. Here, we can use TriangleMesh or TetraMesh. I have used TriangleMesh here.

A TriangleMesh is basically a list of vertices in 3D with a list of faces/traingles. Each traingle is a set of 3 vertex indices defining which vertex to connect.

Now, imagine an image hanging in a 3D world. Each pixel of this image is a vertex for the mesh. And you can define triangles by connecting positions (i,j), (i+1,j) and (i,j+1) to make one triangle and (i, j+1), (i+1,j) and (i+1,j+1) to make another triangle. Just make sure you skip the last row and last column, since there is no i+1 or j+1 for that.

See below answer, which will download a Bunny Mesh and an Image from open3d public assets and generate a mesh from image.

import numpy as np
import open3d as o3d

mesh = o3d.io.read_triangle_mesh(o3d.data.BunnyMesh().path)
pcd1 = mesh.sample_points_poisson_disk(10000)

img = o3d.io.read_image(o3d.data.JuneauImage().path)

def convert_image_to_mesh(img: np.ndarray):
    img = np.asarray(img)

    # Instead of backprojecting, just convert img to an actual 3D plane with Z=0

    # Create 3D vertex for each pixel location
    xvalues = np.arange(img.shape[1])
    yvalues = np.arange(img.shape[0])
    x_loc, y_loc = np.meshgrid(xvalues, yvalues)
    z_loc = np.zeros_like(x_loc)

    # Scale down before making 3D vertices
    x_loc = x_loc / xvalues.shape[0]
    y_loc = y_loc / xvalues.shape[0]  # Keep aspect ratio same by dividing with same denominator. Now image width is 1 meter in 3d.

    vertices = np.stack((x_loc, y_loc, z_loc), axis=2).reshape(-1, 3)

    vertex_colors = img.reshape(-1, 3)/255.0

    # Create triangles between each pair of neighboring vertices
    # Connect positions (i,j), (i+1,j) and (i,j+1) to make one triangle and (i, j+1), (i+1,j) and (i+1,j+1) to make
    # another triangle.
    # Pixel (i,j) is in vertices array at location i + j*xvalues.shape[0]

    vertex_positions = np.arange(xvalues.size * yvalues.size)
    # Reshape into 2D grid and discard last row and column
    vertex_positions = vertex_positions.reshape(yvalues.size, xvalues.size)[:-1, :-1].flatten()

    # Now create triangles (keep vertices in anticlockwise order when making triangles)
    top_triangles = np.vstack(((vertex_positions+1, vertex_positions, vertex_positions+xvalues.shape[0]))).transpose(1, 0)
    bottom_triangles = np.vstack(((vertex_positions+1, vertex_positions, vertex_positions+xvalues.shape[0]))).transpose(1, 0)

    triangles = np.vstack((top_triangles, bottom_triangles))

    mesh: o3d.geometry.TriangleMesh = o3d.geometry.TriangleMesh(
        o3d.utility.Vector3dVector(vertices), o3d.utility.Vector3iVector(triangles)
    )

    mesh.vertex_colors = o3d.utility.Vector3dVector(vertex_colors)
    """
    Flip the y and z axis according to opencv to opengl transformation.
    See - https://stackoverflow.com/questions/44375149/opencv-to-opengl-coordinate-system-transform
    """
    mesh.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
    return mesh

visualizer = o3d.visualization.Visualizer()
visualizer.create_window()

# Enable back face as we want to see image backside as well
rendering_options = visualizer.get_render_option()
rendering_options.mesh_show_back_face = True

visualizer.add_geometry(pcd1)

mesh = convert_image_to_mesh(img)
mesh.scale(scale=0.1, center=pcd1.get_center())
visualizer.add_geometry(mesh)
visualizer.run()