amazon-s3google-apps-scriptgmail-addons

How to work around maximum execution time when uploading to S3 Bucket?

I am using the S3-for-Google-Apps-Script library to export full attachments from Gmail to an S3 bucket. I changed the S3 code to upload the actual content of the attachment rather than an encoded string, as detailed in this post.

However, when attempting to upload an attachment approximately > 5 MB, apps script throws the following error: "Maximum Execution Time Exceeded". I used timestamps to measure the difference in time to ensure that the time issue occurred in the s3.putObject(bucket,objectKey,file) function.

It might be also helpful to note that for a file barely over the limit, it still gets uploaded to my s3 bucket, but apps script returns that the execution time has been exceeded (30 seconds) to the user, disrupting user flow.

Reproducible Example

This is basically a simple button that scrapes a current email for all attachments, if they are pdf's then it calls the export function. and it exports those attachments to our s3 instance. the problem is that when the file > 5mb, it throws the error:

"exportHandler exceeded execution time"

If you're trying to reproduce this be aware that you need to copy an instance of s3 for gas and initialize that as a separate library in apps script with the changes made here.

In order to link the libraries, go to file>libraries, and add the respective library id, version, and development mode in the google apps script console. You'll also need to save your AWS access key and secret key in your property service cache, as detailed in the library documentation.

  1. An initial button that triggers an export of a single attachment on the current Gmail thread:
export default function testButton() {
  const Card = CardService.newCardBuilder();
  const exportButtonSection = CardService.newCardSection();
  const exportWidget = CardService.newTextButton()
    .setText('Export File')
    .setOnClickAction(CardService.newAction().setFunctionName('exportHandler'));
  exportButtonSection.addWidget(exportWidget);
  Card.addSection(exportButtonSection);
  return Card.build();
}

  1. Export an attachment to a specified s3 bucket. Note that S3Modified is an instance of the s3 for google apps script that is modified in accordance to the post outlined above, it's a separate Apps Script file, s3.putObject is where it takes a long time to process an attachment (this is where the error occurs I think).

  2. credentials initialize your s3 awsAccessKey and awsBucket, and can be stored in PropertiesService.

function exportAttachment(attachment) {
  const fileName = attachment.getName();
  const timestamp = Date.now();
  const credentials = PropertiesService.getScriptProperties().getProperties();
  const s3 = S3Modified.getInstance(credentials.awsAccessKeyId, credentials.awsSecretAccessKey);
  s3.putObject(credentials.awsBucket, fileName, attachment, { logRequests: true });
  const timestamp2 = Date.now();
  Logger.log('difference: ', timestamp2 - timestamp);
}
  1. This gets all the attachments that are PDFs in the current email message, this function is pretty much the same as the one on the apps script site for handling Gmail attachments, this specifically looks for pdf's though (not a requirement for the code):
function getAttachments(event) {
  const gmailAccessToken = event.gmail.accessToken;
  const messageIdVal = event.gmail.messageId;

  GmailApp.setCurrentMessageAccessToken(gmailAccessToken);
  const mailMessage = GmailApp.getMessageById(messageIdVal);

  const thread = mailMessage.getThread();
  const messages = thread.getMessages();
  const filteredAttachments = [];
  for (let i = 0; i < messages.length; i += 1) {
    const allAttachments = messages[i].getAttachments();
    for (let j = 0; j < allAttachments.length; j += 1) {
      if (allAttachments[j].getContentType() === 'application/pdf') {
        filteredAttachments.push(allAttachments[j]);
      }
    }
  }
  return filteredAttachments;
}
  1. the global handler that gets attachments and exports them to the s3 bucket when the button is clicked:
function exportHandler(event) {
  const currAttachment = getAttachments(event).flat()[0];
  exportAttachment(currAttachment);
}
global.export = exportHandler;

To be absolutely clear, the bulk of the time is being processed in the second code sample (exportAttachment), since that is where the object is being put into the s3 application.

The timestamps help log how much time that function takes, test it with a 300kb file, you'll get 2 seconds, 4mb 20 seconds, >5mb approx 30 seconds. This part contributes the most to the max execution time.

So this is what leads me to my question, why do I get the maximum execution time exceeded error and how can I fix it? Here are my two thoughts on potential solutions:

  1. Why does the execution limit occur? The quotas say that the runtime limit for a custom function is 30 seconds, and the runtime limit for the script is 6 minutes.

    After some research, I only found custom function mentions in the context of AddOns in Google Sheets, but the function where I'm getting the error is a global function (so that it can be recognized by a callback) in my script. Is there a way to change it to not be recognized as a custom function so that I'm not limited to the 30-second execution limit?

  2. Now, how can I work around this execution limit? Is this an issue with the recommendation to modify the S3 library in this post? Essentially, the modification suggests that we export the actual bytes of the attachment rather than the encoded string.

    This definitely increases the load that Apps Script has to handle which is why it increases the execution time required. How can I work around this issue? Is there a way to change the S3 library to improve processing speed?

Solution

  • Regarding the first question

    From https://developers.google.com/gsuite/add-ons/concepts/actions#callback_functions

    Warning: The Apps Script Card service limits callback functions to a maximum of 30 seconds of execution time. If the execution takes longer than that, your add-on UI may not update its card display properly in response to the Action.

    Regarding the second question

    On the answer to Google Apps Script Async function execution on Server side it's suggested a "hack": Use an "open link" action to call something that can run asynchronously the task that will requiere a long time to run.

    Related

    Answer to rev 1.

    Regarding the first question

    In Google Apps Script, a custom function is a function to be used in a Google Sheets formula. There is no way not extend this limit. Reference https://developers.google.com/app-script/guides/sheets/functions

    onOpen and onEdit simple triggers has also a 30 seconds execution time limit. Reference https://developers.google.com/apps-script/guides/triggers

    Functions being executed from the Google Apps Script editor, a custom menu, an image that has assigned the function, installable triggers, client side code, Google Apps Script API has an execution time limit of 6 minutes for regular Google accounts (like those that have a @gmail.com email address) by the other hand G Suite accounts have a 30 minutes limit.