I am attempting to minimize an energy function between a rendered 3d scene and an image with opendr as in the example given in the OpenDR Paper
I have essentially copied the code from the paper, but used my own image and 3d object. Here is the code to create the renderer:
V = ch.array(m.v)
rn = TexturedRenderer()
rn.frustum = {'near': 1., 'far': 1., 'width': 350, 'height': 500}
rn.camera = ProjectPoints(v=m.v, t=np.array([0, 0, 1]), rt=np.zeros(3), f=[450, 450],
c=[350/2, 500/2], k=np.zeros(5))
rn.set(v=m.v, f=m.f, texture_image =img, ft=m.ft, vt=m.vt, bgcolor=ch.zeros(3))
A = SphericalHarmonics(vn=VertNormals(v=V, f=rn.f), components=ch.array([4,0,0,0]),
light_color=ch.array([1,1,1]))
rn.set(vc = A)
where m is the mesh of the object loaded with opendr's load_mesh. And then I calculate the energy and use ch.minimize just as in the paper:
translation, rotation = ch.array([0,0,4]), ch.zeros(3)
rn.v = translation + m.v.dot(Rodrigues(rotation))
# Create the energy
difference = rn - load_image(img_file)
E = gaussian_pyramid(difference, n_levels=5, normalization='SSE')
# Minimize the energy
light_parms = A.components
ch.minimize(E, x0=[translation])
ch.minimize(E, x0=[translation, rotation, light_parms])
I've confirmed the renderer and the image shape are both 500x350x3. It always returns a dimension mismatch which occurs in a scipy sparse matmul operation that occurs when the state is updated in the minimize_dogleg function of chumpy. I've modified the file to output the shapes of the matrices in the matmul operation through the iterations and with n=5 the shapes it returns are
(129978, 519912) (519912, 525000)
(31992, 127452) (127452, 129978)
(7686, 30744) (30744, 31992)
(1800, 7080) (7080, 7686)
(378, 1512) (1512, 1800)
(256, 256) (105279, 3)
with the last one being where it returns the error. I've altered the number of pyramids and the results are the same and the final shapes of the two matrices are also always the same (unless I go too high then it gives a different error because the matrix becomes too small). One thing I notice is that the error always occurs at n_levels + 1 iterations which seems odd to me as I would think it stops at n_levels, but I have little understanding of what's actually happening in the minimization. For a little extra context the mesh has 35093 vertices E has a length of 686970. Is there anyone that could assist me as to what exactly is happening here or how the minimize is functioning and why the error is occurring on n_levels + 1 iterations.
The problem was fixed by instead using the opendr version from https://github.com/polmorenoc/opendr as opposed to the mattloper version