I want to render depth from mesh file, and camera parameters, to do so I tried RaycastingScene
from open3d with this mesh file as follows:
#!/usr/bin/env python3
import numpy as np
import open3d as o3d
import matplotlib.pyplot as plt
def render_depth(
intrins:o3d.core.Tensor,
width:int,
height:int,
extrins:o3d.core.Tensor,
tmesh:o3d.t.geometry.TriangleMesh
)->np.ndarray:
"""
Render depth from mesh file
Parameters
----------
intrins : o3d.core.Tensor
Camera Intrinsics matrix K: 3x3
width : int
image width
height : int
image height
extrins : o3d.core.Tensor
camera extrinsics matrix 4x4
tmesh : o3d.t.geometry.TriangleMesh
TriangleMesh
Returns
-------
np.ndarray
Rendred depth image
"""
scene = o3d.t.geometry.RaycastingScene()
scene.add_triangles(tmesh)
rays = scene.create_rays_pinhole(
intrinsic_matrix=intrins,
extrinsic_matrix=extrins,
width_px=width, height_px=height
)
ans = scene.cast_rays(rays)
t_hit = ans["t_hit"].numpy() / 1000.0
return t_hit
if __name__=="__main__":
import os
mesh_path = f"{os.getenv('HOME')}/bbq_sauce.ply"
mesh = o3d.t.io.read_triangle_mesh(mesh_path)
mesh.compute_vertex_normals()
# camera_info[k].reshape(3, 3)
intrins_ = np.array([
[606.9275512695312, 0.0, 321.9704895019531],
[0.0, 606.3505859375, 243.5377197265625],
[0.0, 0.0, 1.0]
])
width_ = 640 # camera_info.width
height_ = 480 # camera_info.height
# root2cam 4x4
extrens_ = np.eye(4)
#
intrins_t = o3d.core.Tensor(intrins_)
extrins_t = o3d.core.Tensor(extrens_)
rendered_depth = render_depth(
intrins=intrins_t,
width=width_,
height=height_,
extrins = extrins_t,
tmesh=mesh
)
plt.imshow(rendered_depth)
plt.show()
but I'm getting a depth image which doesn't seem to be correct!
Can you please tell me how can I fix that? thanks.
The raycasting scene is working correctly. However, your extrinsic matrix is setting the camera inside the mesh and thus you are seeing how mesh looks from inside.
I roughly measured the mesh bbq_sauce
in meshlab and it is of width ~53 units. You can either move the camera away from the mesh, using either:
extrens_[2, 3] = 200
or instead of moving the object, just move the camera pose.
rays = o3d.t.geometry.RaycastingScene.create_rays_pinhole(
fov_deg=90, # Not computed from intrinsic parameters.
center=[0, 0, 0], # Look towards
eye=[100, 100, 0], # Look from
up=[0, 1, 0], # This helps in orienting the camera so object is not inverted.
width_px=640,
height_px=480,
)