cassandraruntime-errordatastaxbulkdsbulk

Issue with dsbulk unload

I am getting the below messages while unloading using dsbulk. I am not able to figure out what this means

[s0|347101951|0] Error sending cancel request. This is not critical (the request will eventually time out server-side). (HeartbeatException: null)

Not sending heartbeat because a previous one is still in progress. Check that advanced.heartbeat.interval is not lower than advanced.heartbeat.timeout.

Thanks

Solution

  • "Error sending cancel request" is typical of continuous paging queries. It seems the coordinator is in trouble for some reason, which is why you are also seeing heartbeat failures. Dsbulk may be putting too much load on the cluster.

    You didn't mention which version of dsbulk exactly, but assuming 1.4+ I would recommend trying the following actions (individually or combined):

    1. Disable continuous paging with dsbulk.executor.continuousPaging.enabled = false (this is likely to slow down dsbulk).

    2. Use smaller page sizes, e.g. 1000 rows:

      1. If not using continuous paging: datastax-java-driver.basic.request.page-size = 1000 .
      2. If using continuous paging: datastax-java-driver.advanced.continuous-paging.page-size = 1000.

    3. Throttle dsbulk to reduce the load on the cluster

      1. Either "soft" throttle by limiting the number of concurrent requests, e.g. 128:
        1. DSBulk < 1.6: dsbulk.executor.maxInFlight = 128.
        2. DSBulk >= 1.6: dsbulk.engine.maxConcurrentQueries = 128.
      2. Or "hard" throttle by limiting the number of requests per second, e.g. 500: dsbulk.executor.maxPerSecond = 500.