pythonshutil

Lost files while tried to move files using python shutil.move

I had 120 files in my source folder which I need to move to a new directory (destination). The destination is made in the function I wrote, based on the string in the filename. For example, here is the function I used.

path   ='/path/to/source'
dropbox='/path/to/dropbox'
files = = [os.path.join(path,i).split('/')[-1] for i in os.listdir(path) if i.startswith("SSE")]
sam_lis      =list()
for sam in files:
    sam_list =sam.split('_')[5]
    sam_lis.append(sam_list)
    sam_lis  =pd.unique(sam_lis).tolist()

# Using the above list
ID  = sam_lis

def filemover(ID,files,dropbox):
    """
    Function to move files from the common place to the destination folder
    """
    for samples in ID:
        for fs in files:
            if samples in fs:
                desination = dropbox  + "/"+ samples + "/raw/"
                if not os.path.isdir(desination):
                    os.makedirs(desination)
        for rawfiles in fnmatch.filter(files, pat="*"):
            if samples in rawfiles:
                shutil.move(os.path.join(path,rawfiles), 
                            os.path.join(desination,rawfiles))

In the function, I am creating the destination folders, based on the ID's derived from the files list. When I tried to run this for the first time it threw me FILE NOT exists error. However, later when I checked the source all files starting with SSE were missing. In the beginning, the files were there. I want some insights here;

  1. Whether or not os.shutil.move moves the files to somewhere like a temp folder instead of destination folder?
  2. whether or not the os.shutil.move deletes the files from the source in any circumstance?
  3. Is there any way I can test my script to find the potential reasons for missing files?

Any help or suggestions are much appreciated?

Solution

  • It is late but people don't understand the op's question. If you move a file into a non-existing folder, the file seems to become a compressed binary and get lost forever. It has happened to me twice, once in git bash and the other time using shutil.move in Python. I remember the python happens when your shutil.move destination points to a folder instead of to a copy of the full file path.

    For example, if you run the code below, a similar situation to what the op described will happen:

    src_folder = r'C:/Users/name'
dst_folder = r'C:/Users/name/data_images'
file_names = glob.glob(r'C:/Users/name/*.jpg')

for file in file_names:
    file_name = os.path.basename(file)
    shutil.move(os.path.join(src_folder, file_name), dst_folder)

    Note that dst_folder is just a folder. It should be dst_folder + file_name. This will cause what the Op described in his question. I find something similar on the link here with a more detailed explanation of what went wrong: File moving mistake with Python