Extracting multiple abstracts from pubmed from external pubmed IDs list with biopython

I am trying to extract the abstracts for 60K articles from Pubmed using PubmedIDs. I am trying to export the abstracts into a dictionary. I guess there is some issue with the code I am using, especially while parsing the pubmed IDs. Please help in correcting the code and let me know where it is going wrong.

from Bio import Entrez
import sys

Entrez.email = 'anonymous@gmail.com'

abstract_dict = {}
without_abstract = []

pub_ids = sys.argv[1]
f = open(pub_ids, "r")
for i in f:
    handle = Entrez.efetch(db="pubmed", id=','.join(map(str, i)),
                        rettype="xml", retmode="text")
    records = Entrez.read(handle)
    abstracts = [pubmed_article['MedlineCitation']['Article']['Abstract']['AbstractText'][0]
        if 'Abstract' in pubmed_article['MedlineCitation']['Article'].keys()
        else pubmed_article['MedlineCitation']['Article']['ArticleTitle']
            for pubmed_article in records['PubmedArticle']]
    abstract_dict = dict(zip(i, abstracts))
print(abstract_dict)

Some example Pubmed Ids are:

I am getting result but few lines of abstract or empty dictionary. Is it possible to export the results into tab separated text file from dictionary?

Any suggestion will be appreciated

Thank you

Solution

Note that Entrez.efetch will only return 1000 records. Since you indicate that you want to download 60K abstracts, I have modified your code to download the abstracts in batches.

from Bio import Entrez
import sys
import csv

Entrez.email = 'anonymous@gmail.com'
 
def fetch_abstracts(pub_ids, retmax=1000, output_file='abstracts.csv'):    
    # Make sure requests to NCBI are not too big
    for i in range(0, len(pub_ids), retmax):
        j = i + retmax
        if j >= len(pub_ids):
            j = len(pub_ids)

        print(f"Fetching abstracts from {i} to {j}.")
        handle = Entrez.efetch(db="pubmed", id=','.join(pub_ids[i:j]),
                        rettype="xml", retmode="text", retmax=retmax)
        
        records = Entrez.read(handle)

        abstracts = [pubmed_article['MedlineCitation']['Article']['Abstract']['AbstractText'][0]
                      if 'Abstract' in pubmed_article['MedlineCitation']['Article'].keys()
                      else pubmed_article['MedlineCitation']['Article']['ArticleTitle']
                          for pubmed_article in records['PubmedArticle']]

        abstract_dict = dict(zip(pub_ids[i:j], abstracts))

        with open(output_file, 'a', newline='') as csvfile:
            fieldnames = ['pub_id', 'abstract']
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames, delimiter='\t')
            if i == 0:
              writer.writeheader()
            for pub_id, abstract in abstract_dict.items():
              writer.writerow({'pub_id': pub_id, 'abstract': abstract})

if __name__ == '__main__':
  filename = sys.argv[1]
  pub_ids = open(filename, "r").read().splitlines()
  fetch_abstracts(pub_ids)

If you run this like:

stack73000220.py pubids.txt

where pubids.txt looks like:

then you will get the following output in abstracts.csv:

pub_id  abstract
17284678    Eimeria tenella is an intracellular protozoan parasite that infects the intestinal tracts of domestic fowl and causes coccidiosis, a serious and sometimes lethal enteritis. Eimeria falls in the same phylum (Apicomplexa) as several human and animal parasites such as Cryptosporidium, Toxoplasma, and the malaria parasite, Plasmodium. Here we report the sequencing and analysis of the first chromosome of E. tenella, a chromosome believed to carry loci associated with drug resistance and known to differ between virulent and attenuated strains of the parasite. The chromosome--which appears to be representative of the genome--is gene-dense and rich in simple-sequence repeats, many of which appear to give rise to repetitive amino acid tracts in the predicted proteins. Most striking is the segmentation of the chromosome into repeat-rich regions peppered with transposon-like elements and telomere-like repeats, alternating with repeat-free regions. Predicted genes differ in character between the two types of segment, and the repeat-rich regions appear to be associated with strain-to-strain variation.
15531828    To study the occurrence of nosocomial diarrhea in pediatric wards and the role of infections in its causation.
11791095    Based on single case reports, parvovirus B19 (B19) has repeatedly been proposed as an etiologic agent in patients with Henoch-Schönlein purpura (HSP), perhaps causing vasculitis by direct invasion of vascular endothelial cells because of the tissue distribution of the cellular B19 receptor. A cohort of children with HSP and other vasculitic diseases was investigated and compared with healthy control children to assess the role of B19 as well as parvovirus V9 (a putative emerging B19-like virus).
10708056    The effects of chemokine and chemokine receptor genetic polymorphisms such as stromal derived factor 1 (SDF1-3'A), CCR2-64I, and CCR5-delta32 associated with HIV-1 transmission and/or rate of disease progression in infected study subjects remain highly controversial and have been analyzed primarily only in adults. We have investigated whether these polymorphisms may provide similar beneficial effects in children exposed to HIV-1 perinatally. The prevalence of CCR2-64I allele was significantly increased (p = .03) and the CCR2-64I genotype distribution was not in Hardy-Weinberg equilibrium, among HIV-1-exposed uninfected infants. Moreover, in the HIV-1-infected group, a delay to AIDS progression was observed among carriers of CCR2-64I allele. This is the first report that suggests a protective role of CCR2-64I allele in mother-to-infant HIV-1 transmission and documents a delay in disease progression, after the child has been infected with HIV-1. However, SDFI-3'A and CCR5-delta32 alleles did not modify the rate of HIV-1 transmission or disease progression in HIV-1-infected children.