apache-flinkdatabricksdelta-lakedatabricks-connect

Can Flink/Delta Connector create multiple databricks table


I'm using a new Flink/Databricks connector and I'm trying to understand if a single Flink sink can create multiple delta tables?

Looking at the documentation the reference to the path/s3/hdfs is single location

   /**
 * Convenience method for creating a {@link RowDataDeltaSinkBuilder} for {@link DeltaSink} to a
 * Delta table.
 *
 * @param basePath root path of the Delta table
 * @param conf     Hadoop's conf object that will be used for creating instances of
 *                 {@link io.delta.standalone.DeltaLog} and will be also passed to the
 *                 {@link ParquetRowDataBuilder} to create {@link ParquetWriterFactory}
 * @param rowType  Flink's logical type to indicate the structure of the events in the stream
 * @return builder for the DeltaSink
 */
public static RowDataDeltaSinkBuilder forRowData(
    final Path basePath,
    final Configuration conf,
    final RowType rowType
) {
    return new RowDataDeltaSinkBuilder(
        basePath,
        conf,
        rowType,
        false // mergeSchema
    );

Is it possible to dynamically change the basePath based on event stream?


Solution

  • So generally speaking this is not supported. Both Delta/Flink Source and Sink can read from/write to one location (one Table) at the moment.