Update - see my own answer, it was a retain cycle with very indirect results. I wrote an article about it and the struggle to diagnose.
I have a very weird bug apparently as some kind of side-effect of having shown an SKView in a different UIView. It seems a manifestation of the old iOS9 bug presentScene not working with transition.
Touchgram is an iMessage app extension which lets you create and play interactive messages. It's like a cross between PowerPoint and a game editor, on top of messages. The interface is built in UIKit and SpriteKit (for playback).
Touchgram's engine creates SKScene objects and can present them with or without transitions, in a multi-page message.
The editing interface starts with a tree view and shows detail editor screens to allow editing things like touch detection, page change actions, playing sounds etc.
Some of our editing screens for details also show an SKView. Specifically, the Touch editor has a mode where an SKView is shown to record either a bounding area or gesture to match.
The Text Element editor shows an SKView as a preview of how styling affects a small piece of text. It renders the text element on the main background of the page, which may be a solid colour or image.
So, what works is, launching the app (actually an app extension inside iMessage) and playing a saved message - it all runs fine, with scene transitions.
However, for one editing screen, if you just go into that screen, without it even drawing anything on its embedded SKView, it breaks playback. (This is the Text Element screen mentioned above).
When you then play any of the messages that have transitions, they get stuck with the nodes from the previous scene still showing.
I've narrowed it down to just the literal presence of that SKView on the errant screen. If the only change I make is to remove that SKView, the problem does not occur. There's (now) not even an outlet bound to the SKView and no code interacting with it.
A different screen (Touch editor) has an SKView on it and does not cause this side-effect.
I've spent days narrowing this down to realise it's this side-effect. It wasn't until I read the SO post linked above that I realised the apparent inconsistency of the bug was because some messages had transitions between scenes and others didn't.
This is blocking the first release of the product after over a year of core engine work.
Portion of the xib showing the nesting of views:
<objects>
<view opaque="NO" contentMode="scaleToFill" id="Mia-xp-RE9">
<rect key="frame" x="0.0" y="0.0" width="414" height="736"/>
<autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
<subviews>
...
<view hidden="YES" contentMode="scaleToFill" translatesAutoresizingMaskIntoConstraints="NO" id="rlf-kc-zvh" userLabel="RecordingOverlay">
<rect key="frame" x="0.0" y="0.0" width="414" height="736"/>
<subviews>
...
<skView contentMode="scaleToFill" translatesAutoresizingMaskIntoConstraints="NO" id="AnC-CG-Aqq" userLabel="RecordingArea">
<rect key="frame" x="57" y="191" width="300" height="543"/>
<constraints>
<constraint firstAttribute="width" secondItem="AnC-CG-Aqq" secondAttribute="height" multiplier="414:750" priority="950" id="zMg-Iv-5nu"/>
</constraints>
</skView>
</subviews>
...
</view>
</subviews>
...
</view>
Portion of the xib showing the nesting of views:
<objects>
<view opaque="NO" contentMode="scaleToFill" id="upw-ro-x1m">
<rect key="frame" x="0.0" y="0.0" width="375" height="812"/>
<autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
<subviews>
<view contentMode="scaleToFill" translatesAutoresizingMaskIntoConstraints="NO" id="KTl-Xs-TIo" userLabel="MainContentView">
<rect key="frame" x="0.0" y="44" width="375" height="734"/>
<subviews>
...
<skView contentMode="scaleToFill" translatesAutoresizingMaskIntoConstraints="NO" id="Wch-3x-CLz" userLabel="preview">
<rect key="frame" x="91" y="307.33333333333326" width="193" height="349.66666666666674"/>
<constraints>
<constraint firstAttribute="width" secondItem="Wch-3x-CLz" secondAttribute="height" multiplier="414:750" priority="950" id="jeK-un-SXq"/>
</constraints>
</skView>
...
</subviews>
...
</view>
</subviews>
...
</view>
Trying to work out differences between these screens and what may trigger this UIKit/SpriteKit behaviour. This is visible brainstorming - basically looking for side-effects of what's different between the Working vs Problem screens, without a theory in advance. These are mostly things which feel highly unlikely to have an effect but am just trying to identify differences.
viewDidLoad but noticed the way it is cleared as delegate is through an optional which may result in it not being cleared.deinit in the Working and Problem screens to confirm both are deleted after returning to the main tree view, prior to playing. FINALLY FOUND A DIFFERENCE! - the Problem screen never calls deinit.Things which probably have no effect.
So, as spelled out in the course of my meandering question.
How to diagnose
Adding a deinit to do logging let me zero in on the difference between the two screens - one gets cleaned up and the other doesn't.
The Cause
in viewDidLoad the problem screen had this
// this line, in a method called from viewWillAppear. This is calling a simple drawing kit created by PaintCode
self.thicknessSwatch.drawer = { self.hasThickness ? LinePickerKit.drawEdgeThickness() : LinePickerKit.drawNoEdge() }
The fix, which is horribly obvious in retrospect:
self.thicknessSwatch.drawer = {[weak self] in (self?.hasThickness ?? false) ? LinePickerKit.drawEdgeThickness() : LinePickerKit.drawNoEdge() }
ie: my closure had a retain cycle on the UIViewController. That retained the SKView, even though it was not used, not bound to an outlet, only loaded from the xib.
The confusing behaviour was that the presence of that floating, semi-live SKView had no effect on most SpriteKit behaviour, until there was a presentScene(transition)