swifttransformtrigonometryprojectionperspectivecamera

How to account for yaw and pitch tilt for real world landmark object labeled on iOS camera view perspective

Have an iOS app that labels landmarks (e.g. buildings, mountains the user) in the camera view in realtime, i.e. what the user sees in front of him. When it is straight on, it is no problem and accurate (within reasonable GPS error ranges). When the user tilts the phone, either in yaw or pitch motion, the landmark labels should also adjust accordingly, but I can't quite figure out the best way to do this.

See image tilt rotations

I have tried using the in-built rotateBy transformation on the image context, where midX and midY are the center location of the screen width and height respectively:

UIGraphicsPushContext(ctx)
ctx.saveGState()
ctx.translateBy(x: midX, y:  midY)
ctx.rotate(by: -self.deviceYawAngle)
ctx.translateBy(x: -midX, y: -midY)
...
...
ctx.restoreGState()
UIGraphicsPopContext()

This works ok for handling the "yaw" rotation but I'm not sure how to incorporate the pitch part as well, as intuitively it is a 3d not just a 2d projection on screen.

The other thing I have tried, is to imagine it being a view cone and doing the trigonometry manually. E.g. for the pitch rotation, I can do original dist. to landmark x sin(pitch angle) to figure out the offset based on the pitch angle. Again where I stumble is how to incorporate this in combination with the yaw rotation.

See image trig for perspective change

Any insights would be greatly appreciated, thanks!

Solution

  • Turns out my frame of reference was the main issue because iOS swift typically takes the gravity vector on Z (phone lying flat) whereas I needed it to be on Y (phone held upright with power button on right). This post helped a lot How to create a new CMAttitude Reference Frame to make the gravity be on the Y axis to find the actual yaw value I needed from the new reference frame.

    The updated code for the final transformation (translate & rotate) to achieve the effect looks like:

    let tilttranslation = midY * sin(degreesToRadians(degrees: -self.deviceTiltAngle))
ctx.translateBy(x: 0, y: tilttranslation)
        
ctx.translateBy(x: midX, y:  midY)
ctx.rotate(by: -self.deviceYawAngle)
ctx.translateBy(x: -midX, y: -midY)

    where the tilt and yaw angles are in relation to the upright phone like in this diagram