rubyfograckspacecloudfilesrackspace-cloudfiles

Delete a huge amount of files in Rackspace using fog


I have millions of files in my Rackspace Files. I would like to delete a part of them, passing lists of file names instead of deleting one by one, which is very slow. Is there any way to do this with fog? Right now, I have a script to delete each file, but would be nice to have something with better performance.

connection = Fog::Storage.new({
  :provider           => 'Rackspace',
  :rackspace_username => "xxxx",
  :rackspace_api_key  => "xxxx",
  :rackspace_region   => :iad  
})

dir = connection.directories.select {|d| d.key == "my_directory"}.first

CloudFileModel.where(duplicated: 1).each do |record| 
    f = record.file.gsub("/","")
    dir.files.destroy(f) rescue nil
    puts "deleted #{record.id}"
end

Solution

  • Yes, you can with delete_multiple_objects.

    Deletes multiple objects or containers with a single request.

    To delete objects from a single container, container may be provided and object_names should be an Array of object names within the container.

    To delete objects from multiple containers or delete containers, container should be nil and all object_names should be prefixed with a container name.

    Containers must be empty when deleted. object_names are processed in the order given, so objects within a container should be listed first to empty the container.

    Up to 10,000 objects may be deleted in a single request. The server will respond with 200 OK for all requests. response.body must be inspected for actual results.

    Examples: Delete objects from a container

    object_names = ['object', 'another/object']
    conn.delete_multiple_objects('my_container', object_names)
    

    Delete objects from multiple containers

    object_names = ['container_a/object', 'container_b/object']
    conn.delete_multiple_objects(nil, object_names)
    

    Delete a container and all it's objects

    object_names = ['my_container/object_a', 'my_container/object_b', 'my_container']
    conn.delete_multiple_objects(nil, object_names)