I have notebooks that performs transformation in tables stored in dbfs(databricks file system).I want to capture and display the data lineage. Additionally i want to know how to do the same in hdinsight.
Spline is derived from the words Spark and Lineage. It is a tool which is used to visualize and track how the data changes over time. Spline provides a GUI where the user can view and analyze how the data transforms to give rise to the insights.
You may checkout article which explains Spark Data Lineage on Databricks Notebook using Spline and Data Lineage Tracking And Visualization Solution.