[SOLVED] Enabling metrics snapshotter in YugabyteDB fails with `system.metrics` table not found error

Enabling metrics snapshotter in YugabyteDB fails with `system.metrics` table not found error

We recently enabled the metrics snapshotter on both masters and tservers using the following GFlags:

master_enable_metrics_snapshotter
tserver_enable_metrics_snapshotter

After enabling these, we observed the following warning in the logs regarding a missing metrics table in the system keyspace (YCQL):

W0505 07:00:22.391158 3996485 client-internal.cc:1583] GetTableSchemaRpc(table_identifier: table_name: "metrics" namespace { name: "system" database_type: YQL_DATABASE_CQL }, num_attempts: 1) failed: Not found (yb/master/catalog_manager.cc:5552): Table system.metrics not found: OBJECT_NOT_FOUND (master error 3)
W0505 07:00:22.391261 3996486 metrics_snapshotter.cc:665] P ef86ce7868424b22bcd85d454913371f: Failed to snapshot metrics, code=Not found (yb/master/catalog_manager.cc:5552): Table system.metrics not found: OBJECT_NOT_FOUND (master error 3)

Could you please confirm:

Is this a known issue or expected behaviour in the 2024.2.2.1 version?
Is there any additional configuration required to ensure the system.metrics table is created?

Solution

Once the flag is enabled on the yb-master process and a new YCQL table is created, the metrics table should get created automatically (confirmed by looking at source code).

Please create a test table in YCQL and the metrics table will be created automatically.