Process external HTTP request with CFTHREAD and return asynchronous response

I have build an API where an external server can post JSON data to our ColdFusion Server. The ColdFusion Server then processes this data (loops over arrays and inserts data into a database etc), and then responds to the external server that everything went fine or not.

But I would rather run the process in a CFTHREAD due the load will become more heavy in the future. How can I do this kind of asynchronous action, and still respond to the server that made the request?

I've looked into Event Gateways, but that does not seem to be the way to go. I've also thought about letting the external server do recurring calls, like "are you done yet", "are you done yet", but that does not feel like an ultimate solution.

I can get the guys managing the external server, to call our ColdFusion server the way we want, so at least I'm not bound by that.

Here is a simplified version of my index.cfm that receives the call:

<cftry>
  <cfinclude template="udf.cfm" />
  <cfset _run()>
  <cfcatch type="any">
    <cfcontent type="application/json; charset=utf-8" reset="yes">
    <cfoutput>{"success":0,"message":"#cfcatch.Message# - #cfcatch.Detail#"}</cfoutput>
  </cfcatch>
</cftry>

And here is my udf.cfm, that processes the call (again a simplified version of it):

<cffunction name="_run" output="yes" returntype="void">
    <cfset local.success=true>
    <cfset local.msg="">
    <cftry>
        <!---get request data--->
        <cfset local.requestData=GetHttpRequestData()>
        <!---validate post--->
        <cfif NOT StructKeyExists(local.requestData,"headers")>
            <!---headers is missing--->
            <cfset _returnJson(msg="Headers is missing")>
        </cfif>
        <!---and a bunch of other validations...--->
        <!---get body--->
        <cfset local.isBinaryBody=IsBinary(local.requestData.content)>
        <cfset local.bodyAsString=local.isBinaryBody ? ToString(local.requestData.content) : local.requestData.content>
        <cfset local.body=DeserializeJson(local.bodyAsString)>
        <cftransaction action="begin">
            <cftry>
                <!---prepare for database inserts by cleansing tables etc--->
                <!---loop data array, can be tens of thousands of rows--->
                <cfloop array="#local.body.items#" index="local.item">
                    <!---do some inserts/updates into MySQL Database--->
                </cfloop>
                <cfcatch type="any">
                    <!---error--->
                    <cfset local.msg=_parseCatchError(catch=cfcatch)>
                    <cfset local.success=false>
                </cfcatch>
            </cftry>
            <!---rollback or commit transaction depending on success flag--->
            <cftransaction action="#local.success ? 'commit' : 'rollback'#" />
        </cftransaction>
        <cfcatch type="any">
            <!---error--->
            <cfset local.msg=_parseCatchError(catch=cfcatch)>
            <cfset local.success=false>
        </cfcatch>
    </cftry>
    <!---return JSON--->
    <cfset _returnJson(msg=local.msg,success=local.success)>
</cffunction>
<cffunction name="_returnJson" output="yes" returntype="void">
    <cfargument name="msg" type="string" required="yes">
    <cfargument name="success" type="boolean" default="false">
    <cfscript>getPageContext().getOut().clearBuffer();</cfscript>
    <cfcontent type="application/json; charset=utf-8" reset="yes">
    <cfoutput>#SerializeJson(arguments)#</cfoutput>
    <cfabort>
</cffunction>
<cffunction name="_parseCatchError" output="no" returntype="string">
    <cfargument name="catch" type="any" required="yes">
    <cfset local.error="#Trim(arguments.catch.message)# - #Trim(arguments.catch.detail)#">
    <cfif StructKeyExists(arguments.catch,"tagContext")>
        <!---grab line info--->
        <cfset local.tagContext=arguments.catch.tagContext[1]>
        <cfset local.error=local.error&" - #Trim(local.tagContext.template)# line #Trim(local.tagContext.line)#">
    </cfif>
    <cfreturn local.error>
</cffunction>

Sounds like you just need a callback supplied by the caller.

1. accept request, including an external url for your code to hit once complete
2. start thread, do not join
3. return info to caller that request is being processed, ending the primary request
4. thread keeps running and at end hit the callback url, supplying any additional info about success or failure

A few other options are available here as well. First of all you could just accept and log the data needed for the request in a queue and then hit them at a later time. Also, as you stated, you can provide a status endpoint for the caller to check on progress.