ruby-on-railsrubybashubuntupuma

Puma workers not booting in Rails 5.2 on Ubuntu 20.04 VM


I was learning to deploy a demo rails app using Capistrano, with Puma as an app server and Nginx web server accordingly. I've set up the necessary puma configurations in a file, _stage.rb, and later set puma as a SysVinit service as /etc/init.d/puma_myarticles_staging. The executable file, puma_init.sh.erb was later written into the remote server as puma_init.sh which looked like,

#!/usr/bin/env bash

PATH=/usr/local/bin:/usr/local/sbin/:/sbin:/usr/sbin:/bin:/usr/bin
DESC="Puma rack web server"
NAME=puma_<%=fetch(:full_app_name)%>
SCRIPT_NAME=/etc/init.d/${NAME}
APP_ROOT=<%=current_path%>
PIDFILE=<%= fetch(:puma_pid) %>
STATE_FILE=<%= fetch(:puma_state) %>

log_daemon_msg() { echo "$@"; }
log_end_msg() { [ $1 -eq 0 ] && RES=OK; logger ${RES:=FAIL}; }

run_pumactl(){
  [ $# -lt 1 ] && echo "$# params were given, Expected 1" && exit 1
  cd ${APP_ROOT} && <%= fetch(:rbenv_prefix) %> bundle exec pumactl -F <%=fetch(:puma_conf)%> $1
}

# Function that starts the puma
#
start_task() {
  if [ -e ${PIDFILE} ]; then
    PID=`cat ${PIDFILE}`
    # If the puma isn't running, run it, otherwise restart it.
    if [ "`ps -A -o pid= | grep -c ${PID}`" -eq 0 ]; then
      do_start_task
    else
      restart_task
    fi
  else
    do_start_task
  fi
}
do_start_task() {
  log_daemon_msg "--> Woke up puma ${APP_ROOT}"
  run_pumactl start
}

# Function that stops the daemon/service
#
stop_task() {
  log_daemon_msg "--> Stopping puma in path: ${APP_ROOT} ..."
  if [ -e ${PIDFILE} ]; then
    PID=`cat ${PIDFILE}`
    if [ "`ps -A -o pid= | grep -c ${PID}`" -eq 0 ]; then
      log_daemon_msg "--> Puma isn't running in path: ${APP_ROOT}."
    else
      log_daemon_msg "--> About to kill puma with PID: `cat $PIDFILE` ..."
      if [ "`ps -A -o pid= | grep -c ${PID}`" -eq 0 ]; then
        log_daemon_msg "--> Puma isn't running in path: ${APP_ROOT}."
        return 0
      else
        run_pumactl stop
        log_daemon_msg "--> Waiting for status ..."
        sleep 5
        if [ "`ps -A -o pid= | grep -c ${PID}`" -eq 0 ]; then
          log_daemon_msg "--> Puma with pid ${PID} stopped successfully."
          rm -f ${PIDFILE} ${STATE_FILE}
        else
          log_daemon_msg "--> Unable to stop puma with pid ${PID}."
        fi
      fi
    fi
  else
    log_daemon_msg "--> Puma isn't running in path: ${APP_ROOT}."
  fi
  return 0
}

# Function that sends a SIGUSR2 to the daemon/service
#
restart_task() {
  if [ -e ${PIDFILE} ]; then
    log_daemon_msg "--> About to restart puma in path: ${APP_ROOT} ..."
    run_pumactl restart
  else
    log_daemon_msg "--> Your puma was never playing... Let's get it out there first ..."
    start_task
  fi
  return 0
}

# Function that sends a SIGUSR2 to the daemon/service
#
status_task() {
  if [ -e ${PIDFILE} ]; then
    log_daemon_msg "--> About to status puma ${APP_ROOT} ..."
    run_pumactl status
  else
    log_daemon_msg "---> Puma isn't running in path: ${APP_ROOT}."
  fi
  return 0
}

case "$1" in
  start)
    [ "$VERBOSE" != no ] && log_daemon_msg "Starting ${DESC}" "${NAME} ..."
    start_task
    case "$?" in
      0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
      2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  stop)
    [ "$VERBOSE" != no ] && log_daemon_msg "Stopping ${DESC}" "${NAME} ..."
    stop_task
    case "$?" in
      0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
      2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  status)
    log_daemon_msg "Status ${DESC}" "${NAME} ..."
    status_task
    case "$?" in
      0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
      2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  restart)
    log_daemon_msg "Restarting ${DESC}" "${NAME} ..."
    restart_task
    case "$?" in
      0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
      2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  *)
    echo "Usage:" >&2
    echo "  ${SCRIPT_NAME} {start|stop|status|restart}" >&2
    exit 3
    ;;
esac
:

The puma.rb was,

#!/usr/bin/env puma

directory '/app/myarticles_staging/current'
environment 'staging'

pidfile '/app/myarticles_staging/shared/tmp/pids/puma.pid'
state_path '/app/myarticles_staging/shared/tmp/states/puma.state'
stdout_redirect '/app/myarticles_staging/shared/log/puma_access.log', '/app/myarticles_staging/shared/log/puma_error.log', true

daemonize

threads 4, 8

bind 'unix:///app/myarticles_staging/shared/tmp/sockets/puma.myarticles_staging.sock'

activate_control_app 'unix:///app/myarticles_staging/shared/tmp/sockets/pumactl.myarticles_staging.sock'

workers '4'

preload_app!

on_worker_boot do
  require "active_record"
  ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
  ActiveRecord::Base.establish_connection(YAML.load_file('/app/myarticles_staging/shared/config/database.yml')['staging'])
end

# Allow puma to be restarted by the `rails restart` command.
plugin :tmp_restart

Where I'm fetching all the puma configurations from a file named _stage.rb that looks like,

set :stage, :staging
set :branch, :staging

set :server_port, 80

set :full_app_name, "#{fetch(:application)}_#{fetch(:stage)}"
set :rails_env, :staging

set :deploy_to, "/app/#{fetch(:full_app_name)}"

set :puma_user, fetch(:deploy_user)
set :puma_state, "#{shared_path}/tmp/states/puma.state"
set :puma_pid, "#{shared_path}/tmp/pids/puma.pid"
set :puma_rackup, -> { File.join(current_path, 'config.ru')}
set :puma_bind, "unix://#{shared_path}/tmp/sockets/puma.#{fetch(:full_app_name)}.sock"
set :puma_default_control_app, "unix://#{shared_path}/tmp/sockets/pumactl.#{fetch(:full_app_name)}.sock"
set :puma_conf, "#{shared_path}/config/puma.rb"
set :puma_workers, 4
set :puma_threads, [4, 8]
set :puma_role, :app
set :puma_env, :staging
set :puma_preload_app, true

set :puma_enable_socket_service, true
set :puma_access_log, "#{shared_path}/log/puma_access.log"
set :puma_error_log, "#{shared_path}/log/puma_error.log"

set :nginx_access_log, "#{shared_path}/log/nginx_access.log"
set :nginx_error_log, "#{shared_path}/log/nginx_error.log"

When I started the puma service as /etc/init.d/puma_myarticles_staging start, it outputs,

Starting Puma rack web server puma_myarticles_staging ...
--> Woke up puma /app/myarticles_staging/current
[3955] Puma starting in cluster mode...
[3955] * Version 4.3.12 (ruby 2.7.0-p0), codename: Mysterious Traveller
[3955] * Min threads: 4, max threads: 8
[3955] * Environment: staging
[3955] * Process workers: 1
[3955] * Preloading application
[3955] * Listening on unix:///app/myarticles_staging/shared/tmp/sockets/puma.myarticles_staging.sock
[3955] ! WARNING: Detected 1 Thread(s) started in app boot:
[3955] ! #<Thread:0x000055c692868bf8 /app/myarticles_staging/shared/bundle/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:323 sleep> - /app/myarticles_staging/shared/bundle/ruby/2.7.0/gems/activerecord-6.1.7.4/lib/active_record/connection_adapters/abstract/connection_pool.rb:329:in `sleep'
[3955] * Daemonizing...

Leaving no new puma pid or state files behind. Eventually, the puma service workers didn't boot as I checked the puma.pid and puma.state there are no files being written. When I ran rbenv exec bundle exec rails s to test manually on the current_path it worked.

I checked for the puma process that was demonized using ps ax | grep puma, but didn't find the actual puma workers,

1516 pts/0    S+     0:00 grep --color=auto puma

Any suggestions on what I might be doing wrong? Thanks in advance.


Solution

  • Perhaps the issue is with the way puma is being started. I'm pretty certain there is no Puma bug here. You haven't posted any detail about your systemctl setup but best practice is to use a user service file that way puma will always be started on boot and no passwords are needed

    A standard puma config that I use goes something like this

    # Puma can serve each request in a thread from an internal thread pool.
    # The `threads` method setting takes two numbers: a minimum and maximum.
    # Any libraries that use thread pools should be configured to match
    # the maximum value specified for Puma. Default is set to 5 threads for minimum
    # and maximum; this matches the default thread size of Active Record.
    #
    max_threads_count = ENV.fetch("RAILS_MAX_THREADS") { 5 }
    min_threads_count = ENV.fetch("RAILS_MIN_THREADS") { max_threads_count }
    threads min_threads_count, max_threads_count
    
    # Change to match your CPU core count
    workers 0
    
    # Specifies the `port` that Puma will listen on to receive requests; default is 3000.
    #
    port ENV.fetch("PORT") { 3000 }
    
    # Specifies the `environment` that Puma will run in.
    #
    
    rails_env = ENV['RAILS_ENV'] || "production"
    environment rails_env
    
    app_dir = File.expand_path("../..", __FILE__)
    #shared_dir = "#{app_dir}/shared"
    shared_dir = "/home/project/apps/comtech/shared" # Use your projects path
    
    # Specifies the `pidfile` that Puma will use.
    #pidfile ENV.fetch("PIDFILE") { "pids/server.pid" }
    
    pidfile "#{shared_dir}/pids/puma.pid"
    state_path "#{shared_dir}/pids/puma.state"
    activate_control_app
    
    # Set up socket location
    bind "unix://#{shared_dir}/sockets/comtech_puma.sock"
    
    on_worker_boot do
      require "active_record"
      ActiveRecord::Base.connection.disconnect! rescue ActiveRecord::ConnectionNotEstablished
      ActiveRecord::Base.establish_connection(YAML.load_file("#{app_dir}/config/database.yml")[rails_env])
    end
    
    # Specifies the number of `workers` to boot in clustered mode.
    # Workers are forked web server processes. If using threads and workers together
    # the concurrency of the application would be max `threads` * `workers`.
    # Workers do not work on JRuby or Windows (both of which do not support
    # processes).
    #
    # workers ENV.fetch("WEB_CONCURRENCY") { 2 }
    
    # Use the `preload_app!` method when specifying a `workers` number.
    # This directive tells Puma to first boot the application and load code
    # before forking the application. This takes advantage of Copy On Write
    # process behavior so workers use less memory.
    #
    # preload_app!
    
    # Allow puma to be restarted by `rails restart` command.
    plugin :tmp_restart
    

    Obviously you will need to set the ENVIRONMENT variable to staging on the server

    Adjust the pids folder accordingly to use tmp.

    To use a user service rather than a system service take a look at the following

    cd ~/.config/systemd/user creating the folders if they don’t exist

    nano name_of_your_puma.service and paste following contents adjusting paths accordingly.

    [Unit]
    Description=Puma Rack Server
    # Puma supports systemd's `Type=notify` and watchdog service
    # monitoring, if the [sd_notify](https://github.com/agis/ruby-sdnotify) gem is installed,
    # as of Puma 5.1 or later.
    # On earlier versions of Puma or JRuby, change this to `Type=simple` and remove
    # the `WatchdogSec` line.
    
    [Service]
    Type=simple
    
    # If your Puma process locks up, systemd's watchdog will restart it within seconds.
    #WatchdogSec=10
    RestartSec=10
    
    WorkingDirectory=/home/comtech/apps/comtech/current
    #PIDFile=/home/comtech/apps/comtech/shared/pids/puma.pid
    #User=comtech
    
    ExecStart=/home/comtech/.rvm/bin/rvm 3.1.3@comtech_cms_app do bundle exec puma -C /home/comtech/apps/comtech/current/config/puma_production.rb
    ExecStop=/home/comtech/.rvm/bin/rvm 3.1.3@comtech_cms_app do bundle exec pumactl -S /home/comtech/apps/comtech/shared/pids/puma.state stop
    ExecReload=/home/comtech/.rvm/bin/rvm 3.1.3@ruby-3.1.3@comtech_cms_app do bundle exec pumactl -S /home/comtech/apps/comtech/shared/pids/puma.state restart
    Restart=always
    
    [Install]
    WantedBy=multi-user.target
    

    Then run

    $ systemctl --user enable name_of_your_puma.service 
    

    To checkup on the service

    $ systemctl --user status
    

    To stop the service

    $ systemctl --user stop
    

    To start the service

    $ systemctl --user start
    

    The service will automatically start on boot and will restart puma if puma fails. Also note I' using RVM, so you need to adjust the start command accordingly