Spark details are greyed out - How can I reduce the building time of my pipeline in Foundry Code Repo?
We built a pipeline which includes several transforms. The whole pipeline build is completing in more than 30 minutes today while we need to ensure that the data is available in less than 15 minutes.
How is it possible to reduce the total building time?
While the transforms are running, we have noticed that the spark details are greyed out. Taking an example that is representative of several transforms: during more than 10 minutes the Spark details are greyed out, and then the Spark job is actually getting executed:
The Spark job is running in only 3 minutes. After taking that into consideration, it turned out that during 80% of the build duration, the Spark details are greyed out.
What is happening on the build when Spark details are greyed out? How could we reduce this duration?
Spark details are available once the Spark environment initialization has complete. In this step, the most important stage is the download of all packages. Could you look at the list of packages that you are installed in your Library panel and remove any package that is not used by your transforms?
Some packages, due to their size, can take more time than others to be downloaded. Therefore, removing the ones that you are not using is the most efficient way to save time and accelerate the Spark environment initialization.
- Connecting Palantir to Salesforce - how do you access the data tables (the SObjects)?
- Is that possible to create a custom input widget in Foundry Slate?
- formating text to a new field in foundry contour
- Automatically updating a dataset on Palantir Foundry when any of the immediate upstream resources updates
- Saving CSV files in Foundry Transforms
- Can I access directories in Palantir and use FOR to get names of all tables inside folder?
- How to make a working cycle for files in Palantir?
- In Foundry, how to read "added" rows in output since the last built?
- In Foundry, how to read entire current output dataset which has snaphot transaction type?
- Foundry Slate text box layout
- How to read entire data of output_df in Foundry Code repository?
- Rest API: Is there any endpoint that provides dataset lineage info?
- How to publish a Spark ML pyspark.ml.PipelineModel object in code repositories?
- Can I export .csv (in Gzip) from Palantir but only the first part has headers?
- How can I save filter and variable statuses in Workshop?
- How do you avoid sorting when writing partitioned data in Spark on Palantir Foundry?
- How to delete a file from dataset view in Foundry?
- Refresh platform data in tables after action
- Is it Possibility to change font type in the Workshop module?
- Optimal Data Sync Strategy for Palantir Foundry: Handling Incremental Updates Based on Timestamps
- Importing RIDs from a dataset with column RIDs with Palantir Foundry Code Repository
- Palantir Foundry - How to load PDF files from Compass folder into code repository transform
- Hide tabs in workshop module
- What object types are avaliable for export on <hostname>/phonograph2-export/api/?
- Using a Code Repo to call a Webhook created in Palantir Foundry
- Delete all rows with duplicate values in column in Palantir Foundry Countor
- Palantir Foundry - Making http request and capturing JSON response
- Pass a whole dataset contains multiple files to HuggingFace function in Palantir
- Is there any way to create a generic typescript function that works in different places with two different objects?
- Reference Workshop App Input Widget Value Externally (in a backend process e.g. Python Transform)