I want to open an overlay window (pop up window) when a user selects a cell in my NSTableView similar to selecting an event in iCal. Selecting the event in iCal shows a Window to edit the event, but does so by smoothly animating the window open and adding an arrow pointing to the even in the underlying calendar. Does anyone know what is being used here? Is this a bunch of hidden/custom APIs or is this available for public use?
The editor pane appears to be a custom borderless, transparent window with a custom view (the view defines the shape and therefore the shadow it casts). Learn more here. You might even use MAAttachedWindow.
Regarding animation, it's as simple as asking the window's animator to animate the frame and the alpha value (grouping them together). You'll probably want to set everything up directly in its "start position" first (ie, while the window is off-screen, set its alpha to zero, and its frame to some smaller version so it "zooms in" a la iCal), then put it on screen and start the grouped animation:
[NSAnimationContext beginGrouping];
[[window animator] setFrame:someNewSlightlyLargerFrame];
[[window animator] setAlphaValue:1.0];
[NSAnimationContext endGrouping];
Once the grouping is ended, the animation will begin (asynchronously) and your code execution will continue. Something to think about is getting everything "staged" first (including making sure the subviews of your window are already updated beforehand, so they don't change in the middle of your animation ... unless you want them to).
So the two separate techniques you need to understand are a custom window and basic Cocoa animation.