Delete all rows with duplicate values in column in Palantir Foundry Countor
In a contour analysis with over 2 million rows I need to remove all rows where the 1st column WORKORDER_ID is a duplicated STRING value. I have already sorted by modified time so I want to keep the 1st unique instance.
I have tried writing expressions but have not been able to create anything that would run
Solution
This expression will give you row_numbers starting from 1 for each WORKORDER_ID. After that you just need to add a filter to only keep the rows where row_number is 1.
row_number() OVER (
PARTITION BY "WORKORDER_ID"
ORDER BY "modified_time" ASC
)
- In Foundry, how to read "added" rows in output since the last built?
- In Foundry, how to read entire current output dataset which has snaphot transaction type?
- Foundry Slate text box layout
- How to read entire data of output_df in Foundry Code repository?
- Rest API: Is there any endpoint that provides dataset lineage info?
- How to publish a Spark ML pyspark.ml.PipelineModel object in code repositories?
- Can I export .csv (in Gzip) from Palantir but only the first part has headers?
- How can I save filter and variable statuses in Workshop?
- How do you avoid sorting when writing partitioned data in Spark on Palantir Foundry?
- How to delete a file from dataset view in Foundry?
- Refresh platform data in tables after action
- Is it Possibility to change font type in the Workshop module?
- Optimal Data Sync Strategy for Palantir Foundry: Handling Incremental Updates Based on Timestamps
- Importing RIDs from a dataset with column RIDs with Palantir Foundry Code Repository
- Palantir Foundry - How to load PDF files from Compass folder into code repository transform
- Hide tabs in workshop module
- What object types are avaliable for export on <hostname>/phonograph2-export/api/?
- Using a Code Repo to call a Webhook created in Palantir Foundry
- Delete all rows with duplicate values in column in Palantir Foundry Countor
- Palantir Foundry - Making http request and capturing JSON response
- Pass a whole dataset contains multiple files to HuggingFace function in Palantir
- Is there any way to create a generic typescript function that works in different places with two different objects?
- Reference Workshop App Input Widget Value Externally (in a backend process e.g. Python Transform)
- How to filter out specific rows of dataframe in pyspark?
- How to replace output's column values with input's values in Foundry?
- Palantir Foundry - can we add object attachments in python transform?
- How to convert a Dataset of PDFs to a Media Set?
- Palantir Foundry: Composite-Key for DataSet-Rows / Object-Instance Primary Key
- How do you merge raw data feeds and connections into one in Palantir Foundry?
- Palantir Foundry - run Spark-NLP library offline