c websocket embedded-linux libwebsockets

libwebsockets single thread http server multiple requests handling

I built a single thread http server using lws on embedded linux. When a request is received, while previous one is being processed, it starts processing the second one BEFORE the first is finished.

Here are the logs while handling simple GET: Test client (5 requests/threads at one time):

Thread Thread Thread 140539624781568140539641566976 started 140539633174272 started  started 


Thread 140539517204224 started 
Thread 140539616388864 started 
Thread 140539624781568 finished
Thread 140539641566976 finished
Thread 140539633174272 finished
Thread 140539616388864 finished
Thread 140539517204224 finished
1. CURL code: 0 HTTP code: -1 str: 
2. CURL code: 0 HTTP code: -1 str: 
3. CURL code: 0 HTTP code: 200 str: {"status": true}
4. CURL code: 0 HTTP code: -1 str: 
5. CURL code: 0 HTTP code: 200 str: {"status": true}

Server

my-http[5681]: lws: lws_http_action: (null), checking access rights for mask 0x0 - l.160. lwsLogger()
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: lws: lws_http_action: (null), checking access rights for mask 0x0 - l.160. lwsLogger()
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: lws: lws_http_action: (null), checking access rights for mask 0x0 - l.160. lwsLogger()
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: lws: lws_http_action: (null), checking access rights for mask 0x0 - l.160. lwsLogger()
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT:  - l.93. callback_dynamic_http_HTTP_WRITEABLE()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93. callback_dynamic_http_HTTP_WRITEABLE()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93. callback_dynamic_http_HTTP_WRITEABLE()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93. callback_dynamic_http_HTTP_WRITEABLE()
my-http[5681]: lws: lws_http_action: (null), checking access rights for mask 0x0 - l.160. lwsLogger()
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93. callback_dynamic_http_HTTP_WRITEABLE()

As you see, the order is pretty random, while the demanded one would be:

my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93. 
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93. 
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93. 
my-http[5681]: REST LWS_CALLBACK_HTTP - l.89. callback()
my-http[5681]: LWS_CALLBACK SEND - l.33. callback_dynamic_http_action()
my-http[5681]: REST LWS_CALLBACK_HTTP_WRITEABLE - l.94. callback()
my-http[5681]: TO BE SENT: {"status": true} - l.93.

So I am looking for some kind of solution implementing a queue of requests to be handled in a sequence, and a mechanism that will prevent from handling a request before the previous one is finished.

Is there a solution provided by lws?

Thanks!

Solution

Ok, after some examining I found a solution.

First I confirmed that server is single-threaded state machine, blocking it during request processing (std::this_thread_sleep()) and called next request - it was processed after sleep was finished.

Inside our int callback(lws *wsi, lws_callback_reasons reason, void *user, void *in, size_t len) we have to

dynamically alloc new connection/request processing object on state LWS_CALLBACK_HTTP and...
free it on LWS_CALLBACK_HTTP_WRITEABLE

My problem was that, I assumed that requests are queued in the layer above the callback func and they are processed synchrously - ONE AT TIME, so I was setting pointer to statically allocated global variable...

So it should look like this (just an idea example, without any error checks)

int callback(lws *wsi, lws_callback_reasons reason, void *user, void *in, size_t len)
{
    switch (reason)
    {
        case LWS_CALLBACK_HTTP:
        user = malloc(sizeof(struct HTTPResponse));
        /*Init HTTP processing*/
       
        break;

        case LWS_CALLBACK_HTTP_BODY:
        /*Receive body*/
        break;

        case LWS_CALLBACK_HTTP_BODY_COMPLETION:
        /*Complete body*/
        break;

        case LWS_CALLBACK_HTTP_WRITEABLE:
        /*Send piece of responce data*/
        if(allDataWasSent(user))
        {
            free(user);
        }
        break;

        default:
        break;
    }


    return lws_callback_http_dummy(wsi, reason, user, in, len);
}

Such code will deliver proper connection/request object to demanded context.

Instead of manually mallocing, u can use some more complex data structure like circular buffer or even the std::list<T> (it mustnt be continous dynamic memory container like std::vector<T>!) which is what I did. Smart pointers are not welcomed though.

Cheers!