When using fastlane to build and export an app all of our build machines have started failing to codesign during the export process. The only thing that has changed is the recent expiration of certificates related to TestFlight deployments that were regenerated by fastlane.
The builder runs as a launchd service that starts gitlab-runner.
I have cleared all expired certificates and invalid provisioning profiles from the keychains and validate the complete certificate chain exists in each.
I have written a stripped down example of the issue, a simple script runner.sh to illustrate the problem and pretend to be the CI runner:
runner.sh# Prepare to codesign
security unlock-keychain CodeSigning.keychain
# Codesign
/usr/bin/codesign -vvv --force --sign <HASH> ./MySDK.framework
When run from the shell EXPORT SUCCESS, but when run from launchd EXPORT FAILED. The clues:
Warning: unable to build chain to self-signed root for signer
and errSecInternalComponent
However, this works perfectly and worked right up until recently. Did Apple break keychains for launchd?
/Library/LaunchDaemons/com.example.runner.plist<key>Label</key>
<string>com.example.runner</string>
<key>SessionCreate</key><true/>
<key>RunAtLoad</key><true/>
<key>UserName</key>
<string>builder</string>
<key>GroupName</key>
<string>staff</string>
<key>StandardOutPath</key>
<string>/tmp/test.stdout</string>
<key>StandardErrorPath</key>
<string>/tmp/test.stderr</string>
<key>ProgramArguments</key>
<array>
<string>./runner.sh</string>
</array>
<key>WorkingDirectory</key>
<string>/Users/builder/path/to/runner</string>
After 3-weeks of messing around with keychains, rewriting our codesign scripts to directly call the Security.framework CoreFoundtion APIs, validating keychain dumps for correct trusted app and partition list settings and testing countless other theories, it is this simple, you can no longer have the WWDR Intermediates in any keychain other than the System.keychain. Once installed there, it magically works.
https://www.apple.com/certificateauthority/
This is a change made by Apple in newer versions of the keychain search mechanism and maybe related to the deprecation of the custom keychain APIs in the Security.framework.
Just look at all of the deprecated APIs. This effectively kills off custom keychains.
https://developer.apple.com/documentation/security/keychain_services/keychains?language=objc
As an aside to the solution, the next bit of pain is getting these certificates into the System.keychain. Apple has gone out of its way to make remote administration of dozens of machines as painful as possible.
sudo security import ../WWDRIntermediates.p7b -k /Library/Keychains/System.keychain
I downloaded all of the WWDR certificates into a .p7b and imported them.
Deploying certificates to the System keychain is now best handled using an MDM server like Jamf or Kandji. Increasingly normal administrative functions like macOS updates and importing certificates to the System keychain are either impossible using ansible/command-line.