When I deleted content of NoSqlTarget (key-value storage) in MLRun/v3io via standard command line utility such as:
rm -rf /v3io/project/test-featureset/
It took approx. 1 hour for 1 million of files and in case of 20 millions of files it was near to one day.
Do you have tips/tricks, how to use faster delete?
I got very useful tip for speedup delete process, use this command line:
hdfs dfs -rm -f -r v3io://projects/test-featureset/
The new performance was excellent, 1 million of files per 3 minutes.
BTW: the command hdfs dfs is possible to use for speedup other operations such as -ls -cp -find, more information see the link