I'm trying to use JNA to bridge between Leap Motion's C API (LeapC) and Java since Leap Motion has deprecated their official Java bindings. However, I'm unfamiliar with C and I haven't used JNA before. So far I have managed to get some things to work (like connecting to my Leap Motion controller and polling it for status events), but I'm stuck when it comes to receiving the actual tracking data.
The relevant struct from LeapC is defined like so in LeapC.h:
/** \ingroup Structs
* A snapshot, or frame of data, containing the tracking data for a single moment in time.
* The LEAP_FRAME struct is the container for all the tracking data.
* @since 3.0.0
*/
typedef struct _LEAP_TRACKING_EVENT {
LEAP_FRAME_HEADER info;
int64_t tracking_frame_id;
/** The number of hands tracked in this frame, i.e. the number of elements in
* the pHands array.
*/
uint32_t nHands;
/**
* A pointer to the array of hands tracked in this frame.
*/
LEAP_HAND* pHands;
/**
* Current tracking frame rate in hertz.
*/
float framerate;
} LEAP_TRACKING_EVENT;
My attempts to implement it in JNA have only been partially successful: The info, tracking_frame_id and nHands fields are read properly and contain the expected information. pHands refuses to work.
First, I tried treating pHands as a pointer to the start of the LEAP_HAND array:
@FieldOrder({ "info", "tracking_frame_id", "nHands", "pHands", "framerate" })
public static class LEAP_TRACKING_EVENT extends Structure
{
public LEAP_FRAME_HEADER info;
public long tracking_frame_id;
public int nHands;
public Pointer pHands;
public float framerate;
}
Attempting to read anything from the address the pointer points to, whether a full struct (new LEAP_HAND(pHands)) or even just a single byte (pHands.getByte()) always gives an Invalid Memory Access error from JNA.
Based on reading various other StackOverflow questions and answers it looks like I should be doing something like this.
Secondly, I tried treating pHands as a straight up array:
@FieldOrder({ "info", "tracking_frame_id", "nHands", "pHands", "framerate" })
public static class LEAP_TRACKING_EVENT extends Structure
{
public LEAP_FRAME_HEADER info;
public long tracking_frame_id;
public int nHands;
public LEAP_HAND[] pHands = new LEAP_HAND[1]; //Size-limited to 1 here while testing.
public float framerate;
}
Given a situation where exactly 1 hand is present (as defined in the JNA code above), the array is populated by an instance of LEAP_HAND but it contains garbage data (mostly zeroes). Curiously, the value of pHands[0].palm.position.z does change when I move my hand along the x-axis, so it appears to be partially overlapping the correct memory section here but is not aligned correctly.
I must be doing something wrong with how I'm approaching reading this array. Anyone have an idea of what I'm missing?
They call LeapPollConnection like so: result = LeapPollConnection(connectionHandle, timeout, &msg);. If msg contains a tracking event they then call handleTrackingEvent(msg.tracking_event);. This function is defined as follows:
//File: ExampleConnection.c
static void handleTrackingEvent(const LEAP_TRACKING_EVENT *tracking_event){
if(ConnectionCallbacks.on_frame){
ConnectionCallbacks.on_frame(tracking_event);
}
}
ConnectionCallbacks.on_frame is bound to the following OnFrame function:
//File: CallbackSample.c
/** Callback for when a frame of tracking data is available. */
static void OnFrame(const LEAP_TRACKING_EVENT *frame){
printf("Frame %lli with %i hands.\n", (long long int)frame->info.frame_id, frame->nHands);
for(uint32_t h = 0; h < frame->nHands; h++){
LEAP_HAND* hand = &frame->pHands[h];
printf(" Hand id %i is a %s hand with position (%f, %f, %f).\n",
hand->id,
(hand->type == eLeapHandType_Left ? "left" : "right"),
hand->palm.position.x,
hand->palm.position.y,
hand->palm.position.z);
}
}
// Define an object to receive an event message into.
LEAP_CONNECTION_MESSAGE.ByReference messageRef = new LEAP_CONNECTION_MESSAGE.ByReference();
while (true)
{
// Poll LeapC for an event. A status code for success is returned,
// and messageRef is populated with the event message.
LeapC.INSTANCE.LeapPollConnection(leapConnection.getValue(), 500, messageRef);
// If the event is a tracking event, get the event data.
if (messageRef.type == eLeapEventType.Tracking.getShortValue())
{
LEAP_TRACKING_EVENT event = messageRef.union.tracking_event;
}
// Sleep a moment before polling again.
try
{
Thread.sleep(100);
}
catch (InterruptedException e)
{
}
}
PollLeapConnection and LEAP_CONNECTION_MESSAGE.LeapPollConnection (LeapC API ref)
// Original C signature:
// LeapPollConnection(LEAP_CONNECTION hConnection, uint32_t timeout, LEAP_CONNECTION_MESSAGE* evt);
public eLeapRS LeapPollConnection(Pointer hConnection, int timeout,
LEAP_CONNECTION_MESSAGE.ByReference message);
LEAP_CONNECTION_MESSAGE (LeapC API ref)
// Original C signature:
// typedef struct _LEAP_CONNECTION_MESSAGE {
// /**
// * The size of this message struct. @since 3.0.0
// */
// uint32_t size;
//
// /**
// * The message type. @since 3.0.0
// */
// eLeapEventType type;
//
// /**
// * A pointer to the event data for the current type of message. @since 3.0.0
// */
// union {
// /** An untyped pointer. @since 3.0.0 */
// const void* pointer;
// /** A connection message. @since 3.0.0 */
// const LEAP_CONNECTION_EVENT* connection_event;
// /** A connection lost. @since 3.0.0 */
// const LEAP_CONNECTION_LOST_EVENT* connection_lost_event;
// /** A device detected message. @since 3.0.0 */
// const LEAP_DEVICE_EVENT* device_event;
// /** A device's status has changed. @since 3.1.3 */
// const LEAP_DEVICE_STATUS_CHANGE_EVENT* device_status_change_event;
// /** A policy message. @since 3.0.0 */
// const LEAP_POLICY_EVENT* policy_event;
// /** A device failure message. @since 3.0.0 */
// const LEAP_DEVICE_FAILURE_EVENT* device_failure_event;
// /** A tracking message. @since 3.0.0 */
// const LEAP_TRACKING_EVENT* tracking_event;
// /** A log message. @since 3.0.0 */
// const LEAP_LOG_EVENT* log_event;
// /** A log messages. @since 4.0.0 */
// const LEAP_LOG_EVENTS* log_events;
// /** A get config value message. @since 3.0.0 */
// const LEAP_CONFIG_RESPONSE_EVENT* config_response_event;
// /** A set config value message. @since 3.0.0 */
// const LEAP_CONFIG_CHANGE_EVENT* config_change_event;
// const LEAP_DROPPED_FRAME_EVENT* dropped_frame_event;
// /** A streaming image message. @since 4.0.0 */
// const LEAP_IMAGE_EVENT* image_event;
// /** A point mapping message. @since 4.0.0 */
// const LEAP_POINT_MAPPING_CHANGE_EVENT* point_mapping_change_event;
// const LEAP_HEAD_POSE_EVENT* head_pose_event;
// };
// } LEAP_CONNECTION_MESSAGE;
@FieldOrder({ "size", "type", "union" })
public static class LEAP_CONNECTION_MESSAGE extends Structure
{
public static class EventUnion extends Union
{
public Pointer pointer;
// Pointer is used for all event types I haven't mapped yet.
public Pointer connection_event;
public Pointer connection_lost_event;
public Pointer device_event;
public Pointer device_status_change_event;
public Pointer policy_event;
public Pointer device_failure_event;
public LEAP_TRACKING_EVENT.ByReference tracking_event;
public Pointer log_event;
public Pointer log_events;
public Pointer config_response_event;
public Pointer config_change_event;
public Pointer dropped_frame_event;
public Pointer image_event;
public Pointer point_mapping_change_event;
public Pointer head_pose_event;
}
public int size;
public short type;
public EventUnion union;
private eLeapEventType typeE;
@Override
public void read()
{
super.read();
// Convert the short in type to an enum constant.
typeE = eLeapEventType.None.getForValue(type);
if (typeE == null)
{
typeE = eLeapEventType.None;
}
switch (typeE)
{
case ConfigChange :
union.setType("config_change_event");
break;
case ConfigResponse :
union.setType("config_response_event");
break;
case Connection :
union.setType("connection_event");
break;
case ConnectionLost :
union.setType("connection_lost_event");
break;
case Device :
union.setType("device_event");
break;
case DeviceFailure :
union.setType("device_failure_event");
break;
case DeviceLost :
union.setType("device_event");
break;
case DeviceStatusChange :
union.setType("device_status_change_event");
break;
case DroppedFrame :
union.setType("dropped_frame_event");
break;
case HeadPose :
union.setType("head_pose_event");
break;
case Image :
union.setType("image_event");
break;
case ImageComplete :
break;
case ImageRequestError :
break;
case LogEvent :
union.setType("log_event");
break;
case LogEvents :
union.setType("log_events");
break;
case None :
union.setType("pointer");
break;
case PointMappingChange :
union.setType("point_mapping_change_event");
break;
case Policy :
union.setType("policy_event");
break;
case Tracking :
union.setType("tracking_event");
break;
default :
System.out.println("Unknown message type: " + typeE);
break;
}
union.read();
}
public static class ByReference extends LEAP_CONNECTION_MESSAGE
implements Structure.ByReference
{
}
}
LEAP_TRACKING_EVENT// Original C signature: See the top of this post.
@FieldOrder({ "info", "tracking_frame_id", "nHands", "pHands", "framerate" })
public static class LEAP_TRACKING_EVENT extends Structure
{
public LEAP_FRAME_HEADER info;
public long tracking_frame_id;
public int nHands;
public Pointer pHands;
public float framerate;
// Field to store all LEAP_HAND objects pointed to by pHands.
private LEAP_HAND[] hands;
@Override
public void read()
{
super.read();
// Print frame ID and hand count
System.out.println("======================");
System.out.println("ID: " + tracking_frame_id);
System.out.println("Hands: " + nHands);
if (nHands > 0)
{
// Attempt to read LEAP_HAND data and print info about hand #0.
System.out.println("===");
LEAP_HAND hand = new LEAP_HAND(pHands);
hands = (LEAP_HAND[]) hand.toArray(nHands);
String log = String.format(
"Hand 0| id: %d, type: %d, pos: [%.02f, %.02f, %.02f]%n",
hands[0].id, hands[0].type,
hands[0].palm.position.union.struct.x,
hands[0].palm.position.union.struct.y,
hands[0].palm.position.union.struct.z);
System.out.println(log);
}
System.out.println("======================");
}
public static class ByReference extends LEAP_TRACKING_EVENT
implements Structure.ByReference
{
}
}
new LEAP_HAND(pHands) simply passes the pointer to the super constructor and then calls read() on itself (the code for this is still available in the gist linked below, of course).
Exception in thread "main" java.lang.Error: Invalid memory access
at com.sun.jna.Native.getInt(Native Method)
at com.sun.jna.Pointer.getInt(Pointer.java:580)
at com.sun.jna.Pointer.getValue(Pointer.java:382)
at com.sun.jna.Structure.readField(Structure.java:732)
at com.sun.jna.Structure.read(Structure.java:591)
at komposten.leapmouse.LeapC$LEAP_HAND.read(LeapC.java:322)
at komposten.leapmouse.LeapC$LEAP_HAND.<init>(LeapC.java:316)
at komposten.leapmouse.LeapC$LEAP_TRACKING_EVENT.read(LeapC.java:236)
at com.sun.jna.Structure.autoRead(Structure.java:2203)
at com.sun.jna.Structure.conditionalAutoRead(Structure.java:561)
at com.sun.jna.Structure.updateStructureByReference(Structure.java:690)
at com.sun.jna.Pointer.getValue(Pointer.java:367)
at com.sun.jna.Structure.readField(Structure.java:732)
at com.sun.jna.Union.readField(Union.java:223)
at com.sun.jna.Structure.read(Structure.java:591)
at komposten.leapmouse.LeapC$LEAP_CONNECTION_MESSAGE.read(LeapC.java:196)
at com.sun.jna.Structure.autoRead(Structure.java:2203)
at com.sun.jna.Function.invoke(Function.java:381)
at com.sun.jna.Library$Handler.invoke(Library.java:265)
at com.sun.jna.Native$3.invoke(Native.java:1202)
at com.sun.proxy.$Proxy0.LeapPollConnection(Unknown Source)
at komposten.leapmouse.LeapTest.<init>(LeapTest.java:35)
at komposten.leapmouse.LeapTest.main(LeapTest.java:88)
I have created a gist with the full code for LeapTest and LeapC, after some changes based on the answer and comments here on SO: LeapJna Gist.
I can inline this code here if wanted, but a gist has line numbers and is easier to get an overview of (IMO).
If needed, here is my implementation of LEAP_HAND (as well as the full LEAP_TRACKING_EVENT code and other struct mappings used by those two).
After a lot of discussions with @DanielWiddis (thank you!) we finally found the issue. Using Pointer.dump(), as suggested by Daniel, I manually mapped the memory against the data that JNA loaded into the LEAP_TRACKING_EVENT class:
memory | mapped to
-----------|--------------
[bceba2ed] | info.reserved
[f5010000] | info.reserved
[d7990600] | info.frame_id
[00000000] | info.frame_id
[adbaca0b] | info.timestamp
[04000000] | info.timestamp
[9d990600] | tracking_frame_id
[00000000] | tracking_frame_id
[01000000] | nHands
[80e7a2ed] |
[f5010000] | pHands
[6c4edd42] | pHands
[00000000] | framerate
Because of JNA's default alignment settings for structs 4 bytes where skipped between nHands and pHands, causing pHands to point to 0x42dd4e6c000001f5 instead of 0x000001f5eda2e780 (in this example). Thus pHands pointed to the completely wrong location.
To fix this I simply changed LEAP_TRACKING_DATA to use ALIGN_NONE instead of ALIGN_DEFAULT.