apache-flinkoceanbase

Flink CDC fails to read OceanBase (MySQL mode) binlog: Timeout to receive log messages in LogProxyClient.RecordListener


I am using Flink CDC to capture binlog changes from OceanBase (Mysql mode), but I encountered the following error after running for a while:

com.oceanbase.clogproxy.client.exception.LogProxyClientException: unsupported protocol version: 19968

ENV details:

Flink related logs:

15:50:41,667 ERROR com.oceanbase.clogproxy.client.connection.ClientHandler      [log-proxy-client-worker-1-thread-1]  - Unsupported protocol version: 19968
15:50:41,667 ERROR com.oceanbase.clogproxy.client.connection.ClientHandler      [log-proxy-client-worker-1-thread-1]  - Exception occurred ClientId: 192.168.2.32_11436_1720511441_151_ocp_meta: rootserver_list=192.168.2.181:2881;192.168.2.182:2881;192.168.2.183:2881, cluster_id=, cluster_user=root@ocp_meta, cluster_password=******, , sys_user=, sys_password=******, tb_white_list=ocp_meta.db88.student, tb_black_list=|, start_timestamp=1720511314, start_timestamp_us=0, timezone=+08:00, working_mode=storage, with LogProxy: 192.168.2.182:2983
com.oceanbase.clogproxy.client.exception.LogProxyClientException: Unsupported protocol version: 19968
    at com.oceanbase.clogproxy.client.connection.ClientHandler.checkHeader(ClientHandler.java:276)
    at com.oceanbase.clogproxy.client.connection.ClientHandler.handleHeader(ClientHandler.java:193)
    at com.oceanbase.clogproxy.client.connection.ClientHandler.channelRead(ClientHandler.java:160)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
    at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
    at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
    at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
    at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
    at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
    at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
    at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995)
    at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
    at java.lang.Thread.run(Thread.java:748)

ObLogproxy prompts:

Unexpected seq num, expected value is 1, actual value is 51

Is it because the version of oceanbase-cdc does not match that of ObLogproxy? Do I need to lower the version of oceanbase-cdc? But similar problems occur with oceanbase-cdc-2.2.0


Now SHOW MASTER STATUS and SHOW BINLOG STATUS look fine.

obclient [(none)]> show binlog status\G
*************************** 1. row ***************************
cluster: sgoceanbase
 tenant: ocp_meta
 status: {
        "binlog_files" :
        [
                {
                        "binlog_name" : "mysql-bin.000001",
                        "binlog_size" : 524288251
                },
                {
                        "binlog_name" : "mysql-bin.000002",
                        "binlog_size" : 524288096
                },
                {
                        "binlog_name" : "mysql-bin.000003",
                        "binlog_size" : 524288078
                },
                {
                        "binlog_name" : "mysql-bin.000004",
                        "binlog_size" : 64105871
                }
        ],
        "client_id" : "/usr/local/oblogproxy/run/sgoceanbase/ocp_meta",
        "cpu_status" :
        {
                "cpu_count" : 8,
                "cpu_used_ratio" : 1.5647226572036743
        },
        "disk_status" :
        {
                "disk_total_size_mb" : 51175,
                "disk_usage_size_process_mb" : 1561,
                "disk_used_ratio" : 0.7589448094367981,
                "disk_used_size_mb" : 38839
        },
        "memory_status" :
        {
                "mem_total_size_mb" : 11834,
                "mem_used_ratio" : 0.0,
                "mem_used_size_mb" : 1000
        },
        "network_status" :
        {
                "network_rx_bytes" : 0,
                "network_wx_bytes" : 0
        },
        "pid" : 25270
}
1 row in set (0.011 sec)

obclient [oceanbase]> show master status;
+------------------+----------+--------------+------------------+------------------------------------------------+
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set                              |
+------------------+----------+--------------+------------------+------------------------------------------------+
| mysql-bin.000004 | 63912272 |              |                  | 64db67ca-3eb4-11ef-8570-0050562ef501:1-3737768 |
+------------------+----------+--------------+------------------+------------------------------------------------+
1 row in set (0.682 sec)

Solution

  • The binlog mode of logproxy should be used with flink-mysql-cdc, which is equivalent to treating observer + logproxy + obproxy as a mysql instance. In this way, the connection information uses jdbc (that is, connecting to obproxy).
    Reference to https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1/docs/connectors/flink-sources/mysql-cdc/

    It is recommended to use the latest version 3.1.1.