I'm using a new Flink/Databricks connector and I'm trying to understand if a single Flink sink can create multiple delta tables?
Looking at the documentation the reference to the path/s3/hdfs is single location
/**
* Convenience method for creating a {@link RowDataDeltaSinkBuilder} for {@link DeltaSink} to a
* Delta table.
*
* @param basePath root path of the Delta table
* @param conf Hadoop's conf object that will be used for creating instances of
* {@link io.delta.standalone.DeltaLog} and will be also passed to the
* {@link ParquetRowDataBuilder} to create {@link ParquetWriterFactory}
* @param rowType Flink's logical type to indicate the structure of the events in the stream
* @return builder for the DeltaSink
*/
public static RowDataDeltaSinkBuilder forRowData(
final Path basePath,
final Configuration conf,
final RowType rowType
) {
return new RowDataDeltaSinkBuilder(
basePath,
conf,
rowType,
false // mergeSchema
);
Is it possible to dynamically change the basePath based on event stream?
So generally speaking this is not supported. Both Delta/Flink Source and Sink can read from/write to one location (one Table) at the moment.