downloading a file using python without knowing the file type (file name + file type)

I have been trying to download a file from the American Gut Project in qiita (study number 10317). I can download directly from the web, however I want to create a python code to download the files automatically. This is the url from the site I need to download from: https://qiita.ucsd.edu/study/description/10317 and this is the data I need to download: https://qiita.ucsd.edu/download_raw_data/10317. As you might see, the ending of the file I want to download is not a <path/to/website/file_name.file_type>.

because the download url is not as specified above, I couldn't use the requests library to download it `req = requests.get(download_url) filename = req.url[downloadurl.rfind('/')+1:]

with open(filename, 'wb') as f: for chunk in req.iter_content(chunk): if chunk: f.write(chunk)`

correction and clarification: The problem here is that when I try to download the file I get an html file describing the content of the download link. One of the issues that came to light thanks to the user chepner, is that when trying to use the link it redirects the user to the https://qiita.ucsd website. Is there a way to bypass that? Is there a solution in order to download the file in python?

Thanks in advance!

Solution

I understand the issue you're facing. To automate the download using Python despite the redirection to the authentication page, you can use Selenium, a tool for automating web browsers. Below is a Python code snippet that demonstrates how you can achieve this:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

# Set up Chrome options to specify the download directory and enable automatic download
download_dir = "/path/to/download/folder"  # Replace with your desired download directory

options = webdriver.ChromeOptions()
prefs = {
    "download.default_directory": download_dir,
    "download.prompt_for_download": False,  # Disable download prompt
    "download.directory_upgrade": True,     # Ensure directory is updated if changed
    "safebrowsing.enabled": True            # Enable safe browsing to avoid blocking
}
options.add_experimental_option("prefs", prefs)

# Initialize the WebDriver with the specified options
driver = webdriver.Chrome(executable_path='/path/to/chromedriver', options=options)

try:
    # Step 1: Go to the login page
    driver.get('https://qiita.ucsd.edu/')

    # Step 2: Locate and fill in the login form
    username = driver.find_element(By.ID, 'login_field')  # Replace with actual ID of the username field
    password = driver.find_element(By.ID, 'password')     # Replace with actual ID of the password field

    username.send_keys('your_username')  # Replace with your username
    password.send_keys('your_password')  # Replace with your password
    password.send_keys(Keys.RETURN)      # Submit the form

    # Step 3: Wait for the welcome phrase to appear in the navigation bar
    wait = WebDriverWait(driver, 10)
    welcome_phrase = wait.until(EC.presence_of_element_located((By.XPATH, '//*[@id="navigation-bar"]/div/div[2]/ul[2]/li[1]/a')))

    # Step 4: Go to the download page
    driver.get('https://qiita.ucsd.edu/download_raw_data/10317')

    # The download should start automatically.
    time.sleep(10)  # Wait for the download to complete

finally:
    # Close the driver
    driver.quit()

You can change the default directory where you want to save this file automatically :

download_dir = "/path/to/download/folder"

Make sure to replace 'path_to_chromedriver_executable' with the actual path to your ChromeDriver executable. And if you want to use headless browser, you can add it as option on the driver options we have defined.