aws-lake-formation

Trigger AWS Lake Formation Workflow programmatically


In the documentation I did not find a mention of programmatically triggering workflows (e.g. to load data from a RDBMS into S3). I also did not find anything in the API specification of AWS Lake Formation. We are using Airflow for scheduling our ETL pipeline and it would be nice if we could trigger the Workflow via an API call.


Solution

  • Lake Formation is used to specify permissions for tables and columns in the Glue Catalog. It does not manage ETL. You can use Glue to setup crawlers to download data from a RDBMS. A crawler can be part of a Glue Workflow.

    Reference Links:

    Documentation on adding a crawler for RDBMS

    Glue API to start a crawler

    Glue API to start a workflow