Capistrano deploy failing due to SSH errors related Message too long and file descriptors

I'm having problems when deploying a project to 5 servers I have... the SSHD server version is OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017, as it is a CentOS server, and upgrading the SSHD service and compiling from source is not an option

The important SSHD service configuration:

TCPKeepAlive yes
UseDns no
Protocol 2

When deploying with Capistrano, some steps pass, until I get into the step restorecon, which executes /usr/sbin/restorecon -R /u/apps/my_app/releases/20230724100000

and I get the different errors:

mm_receive_fd: recvmsg: Message too long
mux_master_process_stdio_fwd: failed to receive fd 0 from client
mux_client_request_stdio_fwd: read from master failed: Broken pipe
channel_post_mux_listener: accept: Too many open files
muxclient: master hello exchange failed
mm_receive_fd: recvmsg: Message too long
mux_master_process_stdio_fwd: failed to receive fd 0 from client
mux_client_request_stdio_fwd: read from master failed: Broken pipe

Caused by:
Net::SSH::Proxy::ConnectError: command failed: ssh rorra@bastion.mydomain.com -o ConnectTimeout=60 -o ControlMaster=auto -o ControlPath=/tmp/sshrorra@bastion.mydomain.com -W server01.mydomain.com:\22

I tried to remove the ControlMaster, but didn't work. I also set the ControlMaster to none, but then it asks for credentials on every single step of Capistrano, and that could take forever and too many commands

I tried to restart the OpenSSH service:

sudo systemctl restart sshd.service

but got the same error

Solution

After checking everything on all servers, and the different limits having for the different users, file descriptor limits on kernel configuration and so on, I found that the problem was actually on my Desktop

By checking the limit on my desktop with ulimit -n, I saw that my Ventura OS Limit was on 256, a really low number on allowed number of file descriptors

By following the instructions on this link I was able to update the number of file descriptors on my system and deploy without issues

boot into recovery mode, select options, and then login with your user
open Terminal from menu and run: csrutil clear
shutdown. boot into recovery mode, select options, and then login with your user
select Terminal from menu and run: csrutil disable
boot normally, edit /Library/LaunchDaemons/limit.maxfiles.plist and put the content:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
    <key>Label</key>
    <string>limit.maxfiles</string>
    <key>ProgramArguments</key>
    <array>
        <string>launchctl</string>
        <string>limit</string>
        <string>maxfiles</string>
        <string>64000</string>
        <string>524288</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
    <key>ServiceIPC</key>
    <false/>
    </dict>
</plist>

boot into recovery mode, select options, and then login with your user
select Terminal from menu and run: csrutil enable (this is a very important step, as it will enable macosx system integrity protection)
reboot, execute: ulimit -n, you should now see 64.000 and any issue related message too long, fd console messages or anything during deploys or ssh into the servers should go away