jmeterload-testingstress-testinggraphql-subscriptionsk6

Load/stress test in a SPA with Hasura Cloud Graphql as a backend and subscriptions

I'm trying to do a performance test on a

The problem is that i'm loss about how to do it or if i'm doing it correctly, seeing how most of our stuff are "subscriptions/mutations" that I had to transform into queries. I tried doing those test with k6 and Jmeter but i'm not sure if i'm doing them properly.

k6 test

At first, i did a quick search and collected around 10 subscriptions that are commonly used. Then i tried to create a performance test with k6 https://k6.io/docs/using-k6/http-requests/ but i wasn't able to create a working subscription test so i just transform each subscription into a query and perform a http.post with this setup:

export const options = {
  stages: [
    { duration: '30s', target: 75 },
    { duration: '120s', target: 75 },
    { duration: '60s', target: 50 },
    { duration: '30s', target: 30 },
    { duration: '10s', target: 0 }
  ]
};

export default function () {
  var res = http.post(prod,  
    JSON.stringify({
    query: listaQueries.GetDesafiosCursosByKey(
      keys.desafioCursoKey
    )}), params);
  sleep(1)
}

I did this for every query and ran each test individually. Unfortunately, the numbers i got were bad, and somehow our test environment was getting better times than production. (The only difference afaik is that we're using Hasura Cloud for production).

I tried to implement websocket, but i couldn't getthem work and configure them to do a stress/load test.

K6 result

Jmeter test

After that, i tried something similar with Jmeter, but again i couldn't figure how to set up a subscription test (after i while, i read in a blog that jmeter doesn't support it https://qainsights.com/deep-dive-into-graphql-in-jmeter/ ) so i simply transformed all subscriptions into a query and tried to do the same, but the numbers I was getting were different and much higher than k6.

Jmeter query Config 1

Jmeter query config 2

Jmeter thread config

Questions

I'm not sure if i'm doing it correctly, if transforming every subscription into a query and perform a http request is a correct approach for it. (At least I know that those queries return the data correctly).

Should i just increase the number of VUS/threads until i get a constant timeout to simulate a stress test? There were some test that are causing a graphql error on the website Graphql error, and others were having a

""WARN[0059] Request Failed error="Post \"https://xxxxxxx-xxxxx.herokuapp.com/v1/graphql\": EOF""

in the k6 console. Or should i just give up with k6/jmeter and try to search for another tool to perfom those test?

Thanks you in advance, and sorry for my English and explanation, but i'm a complete newbie at this.

Solution

  • I'm not sure if i'm doing it correctly, if transforming every subscription into a query and perform a http request is a correct approach for it. (At least I know that those queries return the data correctly).

    Ideally you would be using WebSocket as that is what actual clients will most likely be using.

    For code samples, check out the answer here.

    Here's a more complete example utilizing a main.js entry script with modularized Subscription code in subscriptions\bikes.brands.js. It also uses the Httpx library to set a global request header:

    // main.js
import { Httpx } from 'https://jslib.k6.io/httpx/0.0.5/index.js';

import { getBikeBrandsByIdSub } from './subscriptions/bikes-brands.js';

const session = new Httpx({
  baseURL: `http://54.227.75.222:8080`
});

const wsUri = 'wss://54.227.75.222:8080/v1/graphql';

const pauseMin = 2;
const pauseMax = 6;

export const options = {};

export default function () {
  session.addHeader('Content-Type', 'application/json');

  getBikeBrandsByIdSub(1);
}

    // subscriptions/bikes-brands.js
import ws from 'k6/ws';

/* using string concatenation */
export function getBikeBrandsByIdSub(id) {
  const query = `
    subscription getBikeBrandsByIdSub {
      bikes_brands(where: {id: {_eq: ${id}}}) {
        id
        brand
        notes
        updated_at
        created_at
      }
    }
  `;

  const subscribePayload = {
    id: "1",
    payload: {
      extensions: {},
      operationName: "query",
      query: query,
      variables: {},
    },
    type: "start",
  }

  const initPayload = {
    payload: {
      headers: {
        "content-type": "application/json",
      },
      lazy: true,
  
    },
    type: "connection_init",
  };

  console.debug(JSON.stringify(subscribePayload));

  // start a WS connection
  const res = ws.connect(wsUri, initPayload, function(socket) {
    socket.on('open', function() {
      console.debug('WS connection established!');

      // send the connection_init:
      socket.send(JSON.stringify(initPayload));

      // send the chat subscription:
      socket.send(JSON.stringify(subscribePayload));
    });

    socket.on('message', function(message) {
      let messageObj;
      try {
        messageObj = JSON.parse(message);
      }
      catch (err) {
        console.warn('Unable to parse WS message as JSON: ' + message);
      }

      if (messageObj.type === 'data') {
        console.log(`${messageObj.type} message received by VU ${__VU}: ${Object.keys(messageObj.payload.data)[0]}`);
      }

      console.log(`WS message received by VU ${__VU}:\n` + message);
    });
  });
}

    Should i just increase the number of VUS/threads until i get a constant timeout to simulate a stress test?

    Timeouts and errors that only happen under load are signals that you may be hitting a bottleneck somewhere. Do you only see the EOFs under load? These are basically the server sending back incomplete responses/closing connections early which shouldn't happen under normal circumstances.