I have a text file consisting of URL per line as follows:
https://www.google.com
https://www.facebook.com
https://www.gmail.com
I use the following script:
import requests
add = open("manual_list.txt","r")
for a in add:
response = requests.get(a, timeout=(2, 5), verify=False)
fout = open("mylist.txt","a")
fout.write(response.url+"\n")
fout.close()
The problem is, when I write the resulting URL to a file, I get additional %0A
at the end of each line. Can you please explain to me why is this happening?
The problem could be solved by adding strip
function to the input:
response = requests.get(add.strip(), timeout=(2, 5), verify=False)
My questions:
1) I can not understand why this is needed?
2) Searching about %0A
, it turns a line feeding character. This is different from new line character. Can you explain how is it added? Is it my list's fault or the library?
I used the same list with other programs and I don't seem to have similar problem. Why is it problematic here? is it the library's fault? or the list's fault?
EDIT: I use Ubuntu 18.04 and python 3.6.5
requests.get(add, timeout=(2, 5), verify=False)
should probably be
requests.get(a, timeout=(2, 5), verify=False)
Can you try again with that change?
EDIT:
with open("url_list.txt","r") as f:
content = f.readlines()
print(content)
will print out
['https://www.google.com\n', 'https://www.facebook.com\n', 'https://www.gmail.com\n']
Here you can see that your lines in your file do have a '\n', this is normal It just tells the program where a new line should begin. That's why you need an .strip()