So I have a point cloud which I can show on the screen using Open3D's examples. However, now I have an image and I need to show that image on a specific coordinate on the point cloud. I can't find any example about how to do this. Does anyone know how it can be done ? Thank you very much.
One way to display an image is by converting the image to a mesh. Here, we can use TriangleMesh or TetraMesh. I have used TriangleMesh here.
A TriangleMesh is basically a list of vertices in 3D with a list of faces/traingles. Each traingle is a set of 3 vertex indices defining which vertex to connect.
Now, imagine an image hanging in a 3D world. Each pixel of this image is a vertex for the mesh. And you can define triangles by connecting positions (i,j), (i+1,j) and (i,j+1) to make one triangle and (i, j+1), (i+1,j) and (i+1,j+1) to make another triangle. Just make sure you skip the last row and last column, since there is no i+1 or j+1 for that.
See below answer, which will download a Bunny Mesh and an Image from open3d public assets and generate a mesh from image.
import numpy as np
import open3d as o3d
mesh = o3d.io.read_triangle_mesh(o3d.data.BunnyMesh().path)
pcd1 = mesh.sample_points_poisson_disk(10000)
img = o3d.io.read_image(o3d.data.JuneauImage().path)
def convert_image_to_mesh(img: np.ndarray):
img = np.asarray(img)
# Instead of backprojecting, just convert img to an actual 3D plane with Z=0
# Create 3D vertex for each pixel location
xvalues = np.arange(img.shape[1])
yvalues = np.arange(img.shape[0])
x_loc, y_loc = np.meshgrid(xvalues, yvalues)
z_loc = np.zeros_like(x_loc)
# Scale down before making 3D vertices
x_loc = x_loc / xvalues.shape[0]
y_loc = y_loc / xvalues.shape[0] # Keep aspect ratio same by dividing with same denominator. Now image width is 1 meter in 3d.
vertices = np.stack((x_loc, y_loc, z_loc), axis=2).reshape(-1, 3)
vertex_colors = img.reshape(-1, 3)/255.0
# Create triangles between each pair of neighboring vertices
# Connect positions (i,j), (i+1,j) and (i,j+1) to make one triangle and (i, j+1), (i+1,j) and (i+1,j+1) to make
# another triangle.
# Pixel (i,j) is in vertices array at location i + j*xvalues.shape[0]
vertex_positions = np.arange(xvalues.size * yvalues.size)
# Reshape into 2D grid and discard last row and column
vertex_positions = vertex_positions.reshape(yvalues.size, xvalues.size)[:-1, :-1].flatten()
# Now create triangles (keep vertices in anticlockwise order when making triangles)
top_triangles = np.vstack(((vertex_positions+1, vertex_positions, vertex_positions+xvalues.shape[0]))).transpose(1, 0)
bottom_triangles = np.vstack(((vertex_positions+1, vertex_positions, vertex_positions+xvalues.shape[0]))).transpose(1, 0)
triangles = np.vstack((top_triangles, bottom_triangles))
mesh: o3d.geometry.TriangleMesh = o3d.geometry.TriangleMesh(
o3d.utility.Vector3dVector(vertices), o3d.utility.Vector3iVector(triangles)
)
mesh.vertex_colors = o3d.utility.Vector3dVector(vertex_colors)
"""
Flip the y and z axis according to opencv to opengl transformation.
See - https://stackoverflow.com/questions/44375149/opencv-to-opengl-coordinate-system-transform
"""
mesh.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
return mesh
visualizer = o3d.visualization.Visualizer()
visualizer.create_window()
# Enable back face as we want to see image backside as well
rendering_options = visualizer.get_render_option()
rendering_options.mesh_show_back_face = True
visualizer.add_geometry(pcd1)
mesh = convert_image_to_mesh(img)
mesh.scale(scale=0.1, center=pcd1.get_center())
visualizer.add_geometry(mesh)
visualizer.run()