javaazure-functionsazure-queues

Azure Queue Triggered Function Hangs Indefinitely on Call to ShareFileClient


I have a queue triggered function which processes messages containing the name of a file stored in a file share. The filename is parsed from the message and the file is downloaded from the file share using a ShareFileClient:

    public File downloadFileData(String fileName, File pstFile) throws Exception
    {
        // handle unset client name
        if(clientName == null)
        {
            throw new NullPointerException("Client name is unset");
        }
        InputStream fileInputStream = null;
        try
        {
            // connect to the client's file share
            FileShare clientFs = new FileShare(this.clientName);
            ShareFileClient fileClient = clientFs.getClientFileClient(fileName);
            log.info("("+fileName+") opening stream");
            // download the file
            log.info("file exists? "+fileClient.exists());
            log.info("file url: "+fileClient.getFileUrl());
            fileInputStream = fileClient.openInputStream();
            log.info("("+fileName+") saving to file");
            SubmissionUtils.copyInputStreamToFile(fileInputStream, pstFile);
            log.info("("+fileName+") saved to file");
            return pstFile;
        }
        catch (Exception e)
        {
            log.warning("Exception: " + e.getMessage());
            e.printStackTrace();
            throw new Exception("Could not download file "+fileName);
        }
        finally
        {
            // close stream
            if(fileInputStream!=null)
            {
                fileInputStream.close();
            }
        }
    }

the getClientFileClient method retrieves the file client.

public ShareFileClient getClientFileClient(String fileName) {
    return getFileClient(StorageAccount.saConnectionString, FileShare.sharePrefix + 
                         this.clientName,
                         fileName);
}

This has worked correctly for thousands of files, however strange behavior is now occurring with some files. Occasionally, the program executes until the ShareFileClient is first accessed log.info("file exists? "+fileClient.exists());

When the ShareFileClient is first accessed, the queue trigger processor hangs indefinitely. It does not throw an exception and the function does not return with 'failed' or 'succeeded'. The last message shown in the log is the output of log.info("("+fileName+") opening stream");. The function also does not honor its timeout when it is in this state and hangs until the function app is restarted. Other functions still execute correctly. This only occurs for specific files in the file share.

When I run the function app locally using AZFunctions Core Tools the issue does not occur for these specific files.

I have tried:

I would really appreciate some help with this. Please let me know if additional info is needed.


Solution

  • I have not found a solution to this problem, however a workaround is to call existsWithResponse using a timeout before attempting to interact with the file. This will throw an exception once the timeout is reached allowing the program to fail gracefully.

    This issue seems to occur at random. It does not depend on the file that is being accessed, and only occurs when the function app is deployed to Azure (does not occur when running locally).

    This is continuing to cause many issues for us and we will need to find an alternative to Azure File Shares.

    public File downloadFileData(String fileName, File pstFile) throws Exception
        {
            // handle unset client name
            if(clientName == null)
            {
                throw new NullPointerException("Client name is unset");
            }
            InputStream fileInputStream = null;
            try
            {
                // connect to the client's file share
                FileShare clientFs = new FileShare(this.clientName);
                ShareFileClient fileClient = clientFs.getClientFileClient(fileName);
                // verify that fileClient is accessible
                Context context = new Context("","");
                fileClient.existsWithResponse(Duration.ofSeconds((long)FileShare.shareFileClientTimeout), context);
                // download the file
                fileInputStream = fileClient.openInputStream();
                SubmissionUtils.copyInputStreamToFile(fileInputStream, pstFile);
                return pstFile;
            }
            catch (Exception e)
            {
                System.out.println("Exception: " + e.getMessage());
                e.printStackTrace();
                throw new Exception("Could not download file "+fileName+": "+e.getMessage());
            }
            finally
            {
                // close stream
                if(fileInputStream!=null)
                {
                    fileInputStream.close();
                }
            }
        }