palantir-foundryfoundry-code-repositories

Saving CSV files in Foundry Transforms


Is it possible to save a CSV file in Foundry Code Repositories transforms-python language rather than saving them in the Parquet format?


Solution

  • Yes, this can be done by specifying the output dataset format when calling the write_dataset function. You can include compression options as well when making the call. For example:

    @transform(
        my_input = Input('/path/to/input/dataset'),
        my_output = Output('/path/to/output/dataset')
    )
    def compute_function(my_input, my_output):
        my_output.write_dataframe(
            my_input.dataframe(),
            output_format = "csv",
            options = {
                "compression": "gzip"
            }
        )