pythonfile-not-found

FileNotFoundError When file exists (when created in current script)

I am trying to create a secure (e.g., SSL/HTTPS) XML-RPC Client Server. The client-server part works perfectly when the required certificates are present on my system; however, when I try to create the certificates during execution, I receive a FileNotFoundError when opening the ssl-wrapped socket even though the certificates are clearly present (because the preceding function created them.)

Why is the FileNotFoundError given when the files are present? (If I simply close and restart the python script no error is produced when opening the socket and everything works with no issue whatsoever.)

I've searched elsewhere for solutions, but the best/closest answer I've found is, perhaps, "race conditions" between creating the certificates and opening them. However, I've tried adding "sleep" to alleviate the possibility of race conditions (as well as running each function individually via a user input menu) with the same error every time.

What I am missing?

Here is a snippet of my code:

import os
import threading
import ssl 
from xmlrpc.server import SimpleXMLRPCServer
import certs.gencert as gencert  # <---- My python module for generating certs

...

rootDomain = "mydomain"
CERTFILE = "certs/mydomain.cert"
KEYFILE = "certs/mydomain.key"

...

def listenNow(ipAdd, portNum, serverCert, serverKey):

    # Create XMLRPC Server, based on ipAdd/port received
    server = SimpleXMLRPCServer((ipAdd, portNum))

    # **THIS** is what causes the FileNotFoundError ONLY if
    # the certificates are created during THE SAME execution
    # of the program.
    server.socket = ssl.wrap_socket(server.socket,
                                    certfile=serverCert,
                                    keyfile=serverKey,
                                    do_handshake_on_connect=True,
                                    server_side=True)
    ...
    # Start server listening [forever]
    server.serve_forever()

...

# Verify Certificates are present; if not present,
# create new certificates
def verifyCerts():

    # If cert or key file not present, create new certs
    if not os.path.isfile(CERTFILE) or not os.path.isfile(KEYFILE):

        # NOTE: This [genert] will create certificates matching 
        # the file names listed in CERTFILE and KEYFILE at the top
        gencert.gencert(rootDomain)
        print("Certfile(s) NOT present; new certs created.")

    else:
        print("Certfiles Verified Present")


# Start a thread to run server connection as a daemon
def startServer(hostIP, serverPort):

    # Verify certificates present prior to starting server
    verifyCerts()

    # Now, start thread
    t = threading.Thread(name="ServerDaemon",
                         target=listenNow,
                         args=(hostIP,
                               serverPort,
                               CERTFILE,
                               KEYFILE
                               )
                         )
    t.daemon = True
    t.start()


if __name__ == '__main__':
    startServer("127.0.0.1", 12345)
    time.sleep(60)  # <--To allow me to connect w/client before closing

When I run the above, with NO certificates present, this is the error I receive:

$ python3 test.py
Certfile(s) NOT present; new certs created.
Exception in thread ServerDaemon:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "test.py", line 41, in listenNow
server_side=True)
File "/usr/lib/python3.5/ssl.py", line 1069, in wrap_socket
ciphers=ciphers)
File "/usr/lib/python3.5/ssl.py", line 691, in __init__
self._context.load_cert_chain(certfile, keyfile)
FileNotFoundError: [Errno 2] No such file or directory

When I simply re-run the script a second time (i.e., the cert files are already present when it starts, everything runs as expected with NO errors, and I can connect my client just fine. 

$ python3 test.py
Certfiles Verified Present

What is preventing the ssl.wrap_socket function from seeing/accessing the files that were just created (and thus producing the FileNotFoundError exception)?

EDIT 1: Thanks for the comments John Gordon. Here is a copy of gencert.py, courtesy of Atul Varm, found here https://gist.github.com/toolness/3073310

import os
import sys
import hashlib
import subprocess
import datetime

OPENSSL_CONFIG_TEMPLATE = """
prompt = no
distinguished_name = req_distinguished_name
req_extensions = v3_req
[ req_distinguished_name ]
C                      = US
ST                     = IL
L                      = Chicago
O                      = Toolness
OU                     = Experimental Software Authority
CN                     = %(domain)s
emailAddress           = varmaa@toolness.com
[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names
[ alt_names ]
DNS.1 = %(domain)s
DNS.2 = *.%(domain)s
"""

MYDIR = os.path.abspath(os.path.dirname(__file__))
OPENSSL = '/usr/bin/openssl'
KEY_SIZE = 1024
DAYS = 3650
CA_CERT = 'ca.cert'
CA_KEY = 'ca.key'

# Extra X509 args. Consider using e.g. ('-passin', 'pass:blah') if your
# CA password is 'blah'. For more information, see:
#
# http://www.openssl.org/docs/apps/openssl.html#PASS_PHRASE_ARGUMENTS
X509_EXTRA_ARGS = ()

def openssl(*args):
    cmdline = [OPENSSL] + list(args)
    subprocess.check_call(cmdline)

def gencert(domain, rootdir=MYDIR, keysize=KEY_SIZE, days=DAYS,
            ca_cert=CA_CERT, ca_key=CA_KEY):
    def dfile(ext):
        return os.path.join('domains', '%s.%s' % (domain, ext))

    os.chdir(rootdir)

    if not os.path.exists('domains'):
        os.mkdir('domains')

    if not os.path.exists(dfile('key')):
        openssl('genrsa', '-out', dfile('key'), str(keysize))

    # EDIT 3: mydomain.key gets output here during execution

    config = open(dfile('config'), 'w')
    config.write(OPENSSL_CONFIG_TEMPLATE % {'domain': domain})
    config.close()

    # EDIT 3: mydomain.config gets output here during execution

    openssl('req', '-new', '-key', dfile('key'), '-out', dfile('request'),
            '-config', dfile('config'))

    # EDIT 3: mydomain.request gets output here during execution    

    openssl('x509', '-req', '-days', str(days), '-in', dfile('request'),
            '-CA', ca_cert, '-CAkey', ca_key,
            '-set_serial',
            '0x%s' % hashlib.md5(domain + 
                                 str(datetime.datetime.now())).hexdigest(),
            '-out', dfile('cert'),
            '-extensions', 'v3_req', '-extfile', dfile('config'),
            *X509_EXTRA_ARGS)

    # EDIT 3: mydomain.cert gets output here during execution

    print "Done. The private key is at %s, the cert is at %s, and the " \
          "CA cert is at %s." % (dfile('key'), dfile('cert'), ca_cert)

if __name__ == "__main__":
    if len(sys.argv) < 2:
        print "usage: %s <domain-name>" % sys.argv[0]
        sys.exit(1)
    gencert(sys.argv[1])

EDIT 2: Regarding John's comment, "this might mean that those files are being created, but not in the directory [I] expect":

When I have the directory open in another window, I see the files pop up in the correct location during execution. In addition, when running the test.py script a second time with no changes, the files are identified as present in the correct (the same) location. This leads me to believe that file location is not the problem. Thanks for the suggestion. I'll keep looking.

EDIT 3: I stepped through the gencert.py program, and each of the files are correctly output at the right time during execution. I indicated when exactly they were output within the above file, labeled with "EDIT 3"

While gencert is paused awaiting my input (raw_input), I can open/view/edit the mentioned files in another program with no problem.

In addition, with the first test.py instance running (paused, waiting for user input, just after mydomain.cert appears), I can run a second instance of test.py in another terminal and it sees/uses the files just fine.

Within the first instance, however, if I continue the program it outputs "FileNotFoundError."

Solution

  • The problem contained in the above stems from the use of os.chdir(rootdir) as suggested by John; however, the specifics are slightly different than the created files being in the wrong location. The problem is the current working directory (cwd) of the running program being changed by gencert(). Here are the specifics:

    1. The program is started with test.py, which calls verifyCerts(). At this point the program is running in the current directory of whichever folder test.py is running inside of. Use os.getcwd() to find the current directory at this point. In this case (as an example), it is running in:

      /home/name/testfolder/

    2. Next, os.path.isfile() looks for the files "certs/mydomain.cert" and "certs/mydomain.key"; based on the file path above [e.g., the cwd], it is looking for the following files:

      /home/name/testfolder/certs/mydomain.cert
      /home/name/testfolder/certs/mydomain.key

    3. The above files are not present, so the program executes gencert.gencert(rootDomain) which correctly creates both files as expected in the exact locations mentioned above in number 2.

    4. The problem is indeed the os.chdir() call: When gencert() executes, it uses os.chdir() to change the cwd to "rootdir," which is os.path.abspath(os.path.dirname(__file__)), which is the directory of the current file (gencert.py). Since this file is located a folder deeper, the new cwd becomes:

      /home/name/testfolder/certs/

    5. When gencert() finishes execution and the control returns to test.py, the cwd never changes again; the cwd remains as /home/name/testfolder/certs/ even when execution returns to test.py.

    6. Later, when ssl.wrap_socket() tries to find the serverCert and serverKey, it looks for "certs/mydomain.cert" and "certs/mydomain.key" in the cwd, so here is the full path of what it is looking for:

      /home/name/testfolder/certs/certs/mydomain.cert /home/name/testfolder/certs/certs/mydomain.key

    7. These two files are NOT present, so the program correctly returns "FileNotFoundError".

    Solution

    A) Move the "gencert.py" file to the same directory as "test.py"

    B) At the beginning of "gencert.py" add cwd = os.getcwd() to record the original cwd of the program; then, at the very end, add os.chdir(cwd) to change back to the original cwd before ending and giving control back to the calling program.

    I went with option 'B', and my program now works flawlessly. I appreciate the assistance from John Gordon to point me toward finding the source of my problem.