I have many FILE type files (the files that are saved on your system without any extension). These files contain HTML parsed content of news - websites. I need to find the canonical link (URL) hidden there. I am using this code to test for one of the files first -
with open(file, 'r') as f:
html_text = f.read()
soup = BeautifulSoup(html_text, 'html.parser')
link = soup.find('link', rel = 'canonical')
But I am getting a NoneType object error. I also tried these variations
# Variation 1
link = soup.find('link', {'rel':'canonical'})
# Variation 2
link = soup.find('link', rel = 'canonical')['href']
# Variation 3
link = soup.find('link', {'rel':'canonical'}).get['href']
# Variation 4
link = soup.find('link', {'rel':'canonical'})['href']
I also tried a soup.find_all variation, but these failed too. (Errors: NoneType object is not subscriptable/NoneType object doesn't have attribute href)
I manually checked my file by opening it in Notepad and I find that there is a snip in there <link rel=\"canonical\" href=\"https://theintercept.com/2020/10/27/senator-perdue-ossoff-china/feed/\"/>
proving that the canonical is indeed NOT a NoneType object.
This seems like such a simple problem but there seems to be something wrong that I am not able to catch. I went through a lot of questions on StackOverflow that deal with similar issues and tried out their solutions (hence the variations). Any help is appreciated.
Edit - Adding the file content upon request
"<!DOCTYPE html>\n<!--\n ______ __ ______ __ __\n/\\__ _\\/\\ \\ /\\__ _\\ /\\ \\__ /\\ \\__\n\\/_/\\ \\/\\ \\ \\___ __\\/_/\\ \\/ ___\\ \\ ,_\\ __ _ __ ___ __ _____\\ \\ ,_\\\n \\ \\ \\ \\ \\ _ `\\ /'__`\\ \\ \\ \\ /' _ `\\ \\ \\/ /'__`\\/\\`'__\\/'___\\ /'__`\\/\\ '__`\\ \\ \\/\n \\ \\ \\ \\ \\ \\ \\ \\/\\ __/ \\_\\ \\__/\\ \\/\\ \\ \\ \\_/\\ __/\\ \\ \\//\\ \\__//\\ __/\\ \\ \\L\\ \\ \\ \\_\n \\ \\_\\ \\ \\_\\ \\_\\ \\____\\ /\\_____\\ \\_\\ \\_\\ \\__\\ \\____\\\\ \\_\\\\ \\____\\ \\____\\\\ \\ ,__/\\ \\__\\\n \\/_/ \\/_/\\/_/\\/____/ \\/_____/\\/_/\\/_/\\/__/\\/____/ \\/_/ \\/____/\\/____/ \\ \\ \\/ \\/__/\n \\ \\_\\\n \\/_/\n-->\n<html lang=\"en\">\n <head>\n <title>The Intercept</title>\n <meta charset=\"utf-8\">\n <meta http-equiv=\"X-UA-Compatible\" content=\"IE=edge\">\n <meta name=\"viewport\" content=\"width=device-width, initial-scale=1, maximum-scale=1, user-scalable=no\">\n <meta name=\"msapplication-TileColor\" content=\"#000000\">\n <meta name=\"msapplication-TileImage\" content=\"/static/mstile-144x144.png\">\n <meta name=\"msapplication-config\" content=\"/static/browserconfig.xml\">\n <meta name=\"theme-color\" content=\"#ffffff\">\n <meta property=\"og:url\" content=\"https://theintercept.com/2020/10/27/senator-perdue-ossoff-china/feed/\">\n <link rel=\"apple-touch-icon\" sizes=\"57x57\" href=\"/static/apple-touch-icon-57x57.png\">\n <link rel=\"apple-touch-icon\" sizes=\"60x60\" href=\"/static/apple-touch-icon-60x60.png\">\n <link rel=\"apple-touch-icon\" sizes=\"72x72\" href=\"/static/apple-touch-icon-72x72.png\">\n <link rel=\"apple-touch-icon\" sizes=\"76x76\" href=\"/static/apple-touch-icon-76x76.png\">\n <link rel=\"apple-touch-icon\" sizes=\"114x114\" href=\"/static/apple-touch-icon-114x114.png\">\n <link rel=\"apple-touch-icon\" sizes=\"120x120\" href=\"/static/apple-touch-icon-120x120.png\">\n <link rel=\"apple-touch-icon\" sizes=\"144x144\" href=\"/static/apple-touch-icon-144x144.png\">\n <link rel=\"apple-touch-icon\" sizes=\"152x152\" href=\"/static/apple-touch-icon-152x152.png\">\n <link rel=\"apple-touch-icon\" sizes=\"180x180\" href=\"/static/apple-touch-icon-180x180.png\">\n <link rel=\"icon\" type=\"image/png\" href=\"/static/favicon-32x32.png\" sizes=\"32x32\">\n <link rel=\"icon\" type=\"image/png\" href=\"/static/android-chrome-192x192.png\" sizes=\"192x192\">\n <link rel=\"icon\" type=\"image/png\" href=\"/static/favicon-96x96.png\" sizes=\"96x96\">\n <link rel=\"icon\" type=\"image/png\" href=\"/static/favicon-16x16.png\" sizes=\"16x16\">\n <link rel=\"manifest\" href=\"/static/manifest.json\">\n <link rel=\"shortcut icon\" href=\"/static/favicon.ico\">\n <link rel=\"canonical\" href=\"https://theintercept.com/2020/10/27/senator-perdue-ossoff-china/feed/\"/>\n \n \n \n \n <!--[if !IE]><!--><link rel=\"stylesheet\" type=\"text/css\" href=\"/assets/app42e762a729b53f810f04.css\"><!--<![endif]-->\n <!--[if gte IE 9]><link rel=\"stylesheet\" type=\"text/css\" href=\"/assets/app42e762a729b53f810f04.css\"><![endif]-->\n <!--[if lte IE 8]><link rel=\"stylesheet\" type=\"text/css\" href=\"/assets/ie842e762a729b53f810f04.css\"><![endif]-->\n \n <!--[if lte IE 8]>\n <script>\n document.createElement('header');\n document.createElement('nav');\n document.createElement('section');\n document.createElement('article');\n document.createElement('aside');\n document.createElement('footer');\n document.createElement('hgroup');\n document.createElement('picture');\n </script>\n <![endif]-->\n <script id=\"ad-block-test\" src=\"/ads.js\" data-blocked=\"true\"></script>\n </head>\n <body>\n <script src=\"/assets/sniffer42e762a729b53f810f04.js\"></script>\n <div id=\"Root\"><div class=\"InterceptWrapper\" data-reactroot=\"\" data-reactid=\"1\" data-react-checksum=\"1442884202\"><div data-reactid=\"2\"><!-- react-empty: 3 --><!-- react-empty: 4 --></div><div class=\"Header Header--en Header--route-theintercept\" data-reactid=\"5\"><span data-reactid=\"6\"><div class=\"Header-hamburger\" data-reactid=\"7\"><a class=\"Header-hamburger-link\" style=\"color:;\" href=\"/2020/10/27/senator-perdue-ossoff-china/feed/?menu=1\" data-reactid=\"8\"><span class=\"Icon Icon--Menu icon-TI_Menu\" data-reactid=\"9\"></span></a></div><nav class=\"Header-menu\" data-reactid=\"10\"><div class=\"Logo\" data-reactid=\"11\"><div class=\"Logo-bg-block\" data-reactid=\"12\"><div class=\"GridContainer\" data-reactid=\"13\"><div class=\"GridRow\" data-reactid=\"14\"><div class=\"Logo-bg\" data-reactid=\"15\"></div></div></div></div><div class=\"Logo-block\" data-reactid=\"16\"><a href=\"/\" data-reactid=\"17\"><span class=\"Logo-fallback\" style=\"color:#111;\" data-reactid=\"18\"><!-- react-text: 19 -->The<!-- /react-text --><br data-reactid=\"20\"/><!-- react-text: 21 -->Intercept_<!-- /react-text --><span data-reactid=\"22\"><br data-reactid=\"23\"/><!-- react-text: 24 --><!-- /react-text --></span></span><svg class=\"Logo-svg\" height=\"50px\" version=\"1.1\" viewBox=\"0 0 140 50\" width=\"140px\" data-reactid=\"25\"><g data-reactid=\"26\"><path class=\"Logo-path\" d=\"M51.731,30.458c1.246,0,2.264,1.425,2.264,3.206l-4.605,0.56C49.517,31.781,50.28,30.458,51.731,30.458 M40.789,8.601 c1.247,0,2.265,1.424,2.265,3.206l-4.606,0.559C38.575,9.924,39.339,8.601,40.789,8.601 M92.774,30.458 c1.247,0,2.264,1.425,2.264,3.206l-4.605,0.56C90.56,31.781,91.323,30.458,92.774,30.458 M128.295,46.463H140v-2.188h-11.705 V46.463z M106.642,31.679c0.279-0.101,0.61-0.178,1.272-0.178c2.544,0,4.275,1.68,4.275,5.42c0,3.104-1.705,5.216-4.173,5.216 c-0.408,0-0.891-0.076-1.374-0.229V31.679z M68.652,33.206h3.18v-4.097c-0.61-0.254-1.017-0.356-1.603-0.356 c-0.992,0-2.188,0.662-3.435,1.654l-0.916,0.713v-2.367h-0.661l-6.285,1.272v0.967l2.112,0.637v10l-2.112,0.763v0.865h9.313v-0.865 l-2.367-0.763v-9.39c0.611-0.229,1.68-0.407,1.934-0.407L68.652,33.206z M80.484,28.753c-3.995,0-8.372,2.494-8.372,8.066 c0,4.631,3.079,6.82,6.412,6.82c1.146,0,2.469-0.153,4.402-0.967l2.341-0.993l-0.33-0.916c-0.586,0.229-1.4,0.382-2.545,0.382 c-3.155,0-5.547-1.883-5.547-6.132c0-2.952,1.705-4.555,3.257-4.555c0.153,0,0.331,0.025,0.484,0.102l1.399,2.646h2.926v-3.613 C83.741,29.16,82.138,28.753,80.484,28.753 M123.792,26.489h-1.171l-5.521,3.18v1.069l1.857,0.178v10.076 c0,1.807,1.425,2.647,3.435,2.647c0.942,0,2.697-0.306,3.868-0.662l2.035-0.636l-0.178-0.941c-0.33,0.076-0.916,0.101-1.246,0.101 c-1.858,0-3.079-0.687-3.079-2.977v-7.455l4.351,0.229v-2.163h-4.351V26.489z M40.128,26.489h-1.171l-5.522,3.18v1.069l1.858,0.178 v10.076c0,1.807,1.425,2.647,3.435,2.647c0.942,0,2.697-0.306,3.868-0.662l2.035-0.636L44.453,41.4 c-0.331,0.076-0.916,0.101-1.247,0.101c-1.857,0-3.078-0.687-3.078-2.977v-7.455l4.351,0.229v-2.163h-4.351V26.489z M58.626,35.293 v-0.407c0-2.189-1.476-6.133-6.437-6.133c-3.766,0-7.558,2.494-7.558,8.066c0,4.555,3.334,6.82,6.972,6.82 c1.069,0,2.621-0.178,4.428-0.967l2.264-0.993l-0.33-0.916c-0.586,0.229-1.4,0.382-2.596,0.382c-2.799,0-5.878-1.628-6.005-5.852 H58.626z M99.67,35.293v-0.407c0-2.189-1.476-6.133-6.438-6.133c-3.766,0-7.557,2.494-7.557,8.066c0,4.555,3.333,6.82,6.972,6.82 c1.068,0,2.62-0.178,4.427-0.967l2.265-0.993l-0.331-0.916c-0.585,0.229-1.4,0.382-2.596,0.382c-2.798,0-5.877-1.628-6.005-5.852 H99.67z M47.685,13.435v-0.407c0-2.188-1.476-6.132-6.438-6.132c-3.766,0-7.557,2.493-7.557,8.066c0,4.555,3.333,6.819,6.972,6.819 c1.069,0,2.621-0.178,4.427-0.967l2.265-0.992l-0.331-0.916c-0.585,0.229-1.399,0.382-2.595,0.382 c-2.799,0-5.878-1.629-6.005-5.853H47.685z M6.438,25.598v15.725L3.97,42.265v0.992h10.865v-0.992l-2.468-0.942V25.598l2.468-0.942 v-0.992H3.97v0.992L6.438,25.598z M31.781,41.629V33.74c0-2.926-1.094-4.987-4.045-4.987c-1.222,0-2.138,0.382-3.334,0.916 l-2.163,0.942v-1.858h-0.661l-6.285,1.272v0.967l2.112,0.637v10l-2.112,0.763v0.865h8.804v-0.865l-1.858-0.763v-9.924 c0.458-0.127,1.247-0.204,1.833-0.204c1.577,0,2.875,0.865,2.875,3.461v6.667l-1.858,0.763v0.865h8.804v-0.865L31.781,41.629z M106.642,28.753h-0.662l-6.285,1.272v0.967l2.112,0.637v15.598l-2.112,0.763v0.865h9.567V47.99l-2.62-0.763v-3.588h0.992 c5.954,0,9.287-3.614,9.287-8.626c0-4.301-3.002-6.26-5.241-6.26c-0.891,0-2.443,0.535-3.868,1.247l-0.967,0.483h-0.203V28.753z M31.527,19.771v-7.888c0-2.926-1.094-4.987-4.046-4.987c-1.221,0-2.137,0.381-3.333,0.916l-2.163,0.941V0h-0.662l-4.911,1.807H0 v4.911h2.316L3.257,3.69h3.181v15.776L3.97,20.407V21.4h10.61v-0.993l-2.213-0.941V3.69h4.783v16.081l-1.857,0.763V21.4h8.55 v-0.866l-1.858-0.763V9.873c0.458-0.127,1.247-0.229,1.832-0.229c1.578,0,2.875,0.865,2.875,3.46v6.667l-1.857,0.763V21.4h8.804 v-0.866L31.527,19.771z\" fill=\"#111\" data-reactid=\"27\"></path></g></svg></a></div></div><ul class=\"Header-language-list\" data-reactid=\"28\"><li class=\"Header-language-list-item Header-language-list-item--active\" data-reactid=\"29\"><a class=\"Header-language-link\" href=\"/\" data-reactid=\"30\">English</a></li><li class=\"Header-language-list-item\" data-reactid=\"31\"><a class=\"Header-language-link\" href=\"/brasil/\" data-reactid=\"32\">Portugu\u00eas</a></li></ul><div class=\"Header-search-block\" data-reactid=\"33\"><form action=\"/search\" type=\"get\" data-reactid=\"34\"><label class=\"Header-search-label\" for=\"search\" data-reactid=\"35\"><span class=\"Icon Icon--Search icon-TI_Search\" data-reactid=\"36\"></span></label><input id=\"search\" class=\"Header-search-input\" name=\"s\" data-reactid=\"37\"/></form></div><div class=\"Header-menu-mission-block\" data-reactid=\"38\"><ul class=\"Header-menu-list Header-menu-list--collection-items\" data-reactid=\"39\"><li class=\"Header-menu-list-item\" data-reactid=\"40\"><a class=\"Header-menu-link\" href=\"/politics/\" data-reactid=\"41\">Politics</a></li><li class=\"Header-menu-list-item\" data-reactid=\"42\"><a class=\"Header-menu-link\" href=\"/justice/\" data-reactid=\"43\">Justice</a></li><li class=\"Header-menu-list-item\" data-reactid=\"44\"><a class=\"Header-menu-link\" href=\"/national-security/\" data-reactid=\"45\">National Security</a></li><li class=\"Header-menu-list-item\" data-reactid=\"46\"><a class=\"Header-menu-link\" href=\"/world/\" data-reactid=\"47\">World</a></li><li class=\"Header-menu-list-item\" data-reactid=\"48\"><a class=\"Header-menu-link\" href=\"/technology/\" data-reactid=\"49\">Technology</a></li><li class=\"Header-menu-list-item\" data-reactid=\"50\"><a class=\"Header-menu-link\" href=\"/environment/\" data-reactid=\"51\">Environment</a></li></ul></div><div class=\"Header-menu-mission-block\" data-reactid=\"52\"><ul class=\"Header-menu-list Header-menu-list--mission-items\" data-reactid=\"53\"><li class=\"Header-menu-list-item\" data-reactid=\"54\"><a class=\"Header-menu-link\" href=\"/special-investigations/\" data-reactid=\"55\">Special Investigations</a></li><li class=\"Header-menu-list-item\" data-reactid=\"56\"><a class=\"Header-menu-link\" href=\"/voices/\" data-reactid=\"57\">Voices</a></li><li class=\"Header-menu-list-item\" data-reactid=\"58\"><a class=\"Header-menu-link\" href=\"/podcasts/\" data-reactid=\"59\">Podcasts</a></li><li class=\"Header-menu-list-item\" data-reactid=\"60\"><a class=\"Header-menu-link\" href=\"/videos/\" data-reactid=\"61\">Videos</a></li><li class=\"Header-menu-list-item\" data-reactid=\"62\"><a class=\"Header-menu-link\" href=\"/documents/\" data-reactid=\"63\">Documents</a></li><li class=\"Header-menu-list-item\" data-reactid=\"64\"><a class=\"Header-menu-link\" href=\"https://join.theintercept.com/donate/now?source=web_intercept_20200601_hamburger\" data-reactid=\"65\"><div class=\"Header-menu-list-item-button\" data-reactid=\"66\"><!-- react-text: 67 -->Become A Member<!-- /react-text --><span class=\"Icon Icon--Arrow_02_Right icon-TI_Arrow_02_Right\" data-reactid=\"68\"></span></div></a></li></ul></div><ul class=\"Header-menu-list Header-menu-list--content-items\" data-reactid=\"69\"><li class=\"Header-menu-list-item\" data-reactid=\"70\"><a class=\"Header-menu-link\" href=\"/about/\" data-reactid=\"71\">About</a></li><li class=\"Header-menu-list-item\" data-reactid=\"72\"><a class=\"Header-menu-link\" href=\"/policies/\" data-reactid=\"73\">Editorial Policies</a></li><li class=\"Header-menu-list-item\" data-reactid=\"74\"><a class=\"Header-menu-link\" href=\"/source/\" data-reactid=\"75\">Become a Source</a></li><li class=\"Header-menu-list-item\" data-reactid=\"76\"><a class=\"Header-menu-link\" href=\"/newsletter/?source=web_hamburger\" data-reactid=\"77\">Join Newsletter</a></li></ul><div class=\"Header-footer\" data-reactid=\"78\"><div class=\"Header-social-links\" data-reactid=\"79\"><a class=\"Header-social-link\" target=\"_blank\" data-label=\"facebook\" href=\"https://www.facebook.com/theinterceptflm\" data-reactid=\"80\"><span class=\"Icon Icon--Facebook icon-TI_Facebook\" data-reactid=\"81\"></span></a><a class=\"Header-social-link\" target=\"_blank\" data-label=\"twitter\" href=\"https://twitter.com/theintercept\" data-reactid=\"82\"><span class=\"Icon Icon--Twitter icon-TI_Twitter\" data-reactid=\"83\"></span></a><a class=\"Header-social-link\" target=\"_blank\" data-label=\"instagram\" href=\"https://www.instagram.com/theintercept/\" data-reactid=\"84\"><span class=\"Icon Icon--Instagram icon-TI_Instagram\" data-reactid=\"85\"></span></a><a class=\"Header-social-link\" target=\"_blank\" data-label=\"tumblr\" href=\"https://the-intercept.tumblr.com\" data-reactid=\"86\"><span class=\"Icon Icon--Tumblr icon-TI_Tumblr\" data-reactid=\"87\"></span></a><a class=\"Header-social-link\" target=\"_blank\" data-label=\"snapchat\" href=\"https://www.snapchat.com/add/theintercept\" data-reactid=\"88\"><span class=\"Icon Icon--Snapchat icon-TI_Snapchat\" data-reactid=\"89\"></span></a><a class=\"Header-social-link\" target=\"_blank\" data-label=\"flipboard\" href=\"https://flipboard.com/@TheIntercept\" data-reactid=\"90\"><span class=\"Icon Icon--Flipboard icon-TI_Flipboard\" data-reactid=\"91\"></span></a><a class=\"Header-social-link\" data-label=\"rss\" href=\"/feeds/\" data-reactid=\"92\"><span class=\"Icon Icon--RSS icon-TI_RSS\" data-reactid=\"93\"></span></a></div><img class=\"Header-FLM-svg\" src=\"/static/FLM.svg\" alt=\"First Look Media logo\" data-reactid=\"94\"/><p class=\"Header-TM\" data-reactid=\"95\">The Intercept is a First Look Media Company.</p><cite class=\"Header-copyright\" data-reactid=\"96\"><!-- react-text: 97 -->\u00a9 First Look Media. <!-- /react-text --><span data-reactid=\"98\">All rights reserved</span></cite><ul class=\"Header-footer-list\" data-reactid=\"99\"><li class=\"Header-footer-list-item\" data-reactid=\"100\"><a class=\"Header-footer-link\" data-label=\"terms of use\" href=\"/terms-use/\" data-reactid=\"101\"><span data-reactid=\"102\">Terms of use</span></a></li><li class=\"Header-footer-list-item\" data-reactid=\"103\"><a class=\"Header-footer-link\" data-label=\"privacy policy\" href=\"/privacy-policy/\" data-reactid=\"104\"><span data-reactid=\"105\">Privacy</span></a></li></ul></div></nav></span></div><div class=\"ErrorPage\" data-reactid=\"106\"><div class=\"Logo\" data-reactid=\"107\"><div class=\"Logo-block\" data-reactid=\"108\"><a href=\"/\" data-reactid=\"109\"><span class=\"Logo-fallback\" style=\"color:#fff;\" data-reactid=\"110\"><!-- react-text: 111 -->The<!-- /react-text --><br data-reactid=\"112\"/><!-- react-text: 113 -->Intercept_<!-- /react-text --><span data-reactid=\"114\"><br data-reactid=\"115\"/><!-- react-text: 116 --><!-- /react-text --></span></span><svg class=\"Logo-svg\" height=\"50px\" version=\"1.1\" viewBox=\"0 0 140 50\" width=\"140px\" data-reactid=\"117\"><g data-reactid=\"118\"><path class=\"Logo-path\" d=\"M51.731,30.458c1.246,0,2.264,1.425,2.264,3.206l-4.605,0.56C49.517,31.781,50.28,30.458,51.731,30.458 M40.789,8.601 c1.247,0,2.265,1.424,2.265,3.206l-4.606,0.559C38.575,9.924,39.339,8.601,40.789,8.601 M92.774,30.458 c1.247,0,2.264,1.425,2.264,3.206l-4.605,0.56C90.56,31.781,91.323,30.458,92.774,30.458 M128.295,46.463H140v-2.188h-11.705 V46.463z M106.642,31.679c0.279-0.101,0.61-0.178,1.272-0.178c2.544,0,4.275,1.68,4.275,5.42c0,3.104-1.705,5.216-4.173,5.216 c-0.408,0-0.891-0.076-1.374-0.229V31.679z M68.652,33.206h3.18v-4.097c-0.61-0.254-1.017-0.356-1.603-0.356 c-0.992,0-2.188,0.662-3.435,1.654l-0.916,0.713v-2.367h-0.661l-6.285,1.272v0.967l2.112,0.637v10l-2.112,0.763v0.865h9.313v-0.865 l-2.367-0.763v-9.39c0.611-0.229,1.68-0.407,1.934-0.407L68.652,33.206z M80.484,28.753c-3.995,0-8.372,2.494-8.372,8.066 c0,4.631,3.079,6.82,6.412,6.82c1.146,0,2.469-0.153,4.402-0.967l2.341-0.993l-0.33-0.916c-0.586,0.229-1.4,0.382-2.545,0.382 c-3.155,0-5.547-1.883-5.547-6.132c0-2.952,1.705-4.555,3.257-4.555c0.153,0,0.331,0.025,0.484,0.102l1.399,2.646h2.926v-3.613 C83.741,29.16,82.138,28.753,80.484,28.753 M123.792,26.489h-1.171l-5.521,3.18v1.069l1.857,0.178v10.076 c0,1.807,1.425,2.647,3.435,2.647c0.942,0,2.697-0.306,3.868-0.662l2.035-0.636l-0.178-0.941c-0.33,0.076-0.916,0.101-1.246,0.101 c-1.858,0-3.079-0.687-3.079-2.977v-7.455l4.351,0.229v-2.163h-4.351V26.489z M40.128,26.489h-1.171l-5.522,3.18v1.069l1.858,0.178 v10.076c0,1.807,1.425,2.647,3.435,2.647c0.942,0,2.697-0.306,3.868-0.662l2.035-0.636L44.453,41.4 c-0.331,0.076-0.916,0.101-1.247,0.101c-1.857,0-3.078-0.687-3.078-2.977v-7.455l4.351,0.229v-2.163h-4.351V26.489z M58.626,35.293 v-0.407c0-2.189-1.476-6.133-6.437-6.133c-3.766,0-7.558,2.494-7.558,8.066c0,4.555,3.334,6.82,6.972,6.82 c1.069,0,2.621-0.178,4.428-0.967l2.264-0.993l-0.33-0.916c-0.586,0.229-1.4,0.382-2.596,0.382c-2.799,0-5.878-1.628-6.005-5.852 H58.626z M99.67,35.293v-0.407c0-2.189-1.476-6.133-6.438-6.133c-3.766,0-7.557,2.494-7.557,8.066c0,4.555,3.333,6.82,6.972,6.82 c1.068,0,2.62-0.178,4.427-0.967l2.265-0.993l-0.331-0.916c-0.585,0.229-1.4,0.382-2.596,0.382c-2.798,0-5.877-1.628-6.005-5.852 H99.67z M47.685,13.435v-0.407c0-2.188-1.476-6.132-6.438-6.132c-3.766,0-7.557,2.493-7.557,8.066c0,4.555,3.333,6.819,6.972,6.819 c1.069,0,2.621-0.178,4.427-0.967l2.265-0.992l-0.331-0.916c-0.585,0.229-1.399,0.382-2.595,0.382 c-2.799,0-5.878-1.629-6.005-5.853H47.685z M6.438,25.598v15.725L3.97,42.265v0.992h10.865v-0.992l-2.468-0.942V25.598l2.468-0.942 v-0.992H3.97v0.992L6.438,25.598z M31.781,41.629V33.74c0-2.926-1.094-4.987-4.045-4.987c-1.222,0-2.138,0.382-3.334,0.916 l-2.163,0.942v-1.858h-0.661l-6.285,1.272v0.967l2.112,0.637v10l-2.112,0.763v0.865h8.804v-0.865l-1.858-0.763v-9.924 c0.458-0.127,1.247-0.204,1.833-0.204c1.577,0,2.875,0.865,2.875,3.461v6.667l-1.858,0.763v0.865h8.804v-0.865L31.781,41.629z M106.642,28.753h-0.662l-6.285,1.272v0.967l2.112,0.637v15.598l-2.112,0.763v0.865h9.567V47.99l-2.62-0.763v-3.588h0.992 c5.954,0,9.287-3.614,9.287-8.626c0-4.301-3.002-6.26-5.241-6.26c-0.891,0-2.443,0.535-3.868,1.247l-0.967,0.483h-0.203V28.753z M31.527,19.771v-7.888c0-2.926-1.094-4.987-4.046-4.987c-1.221,0-2.137,0.381-3.333,0.916l-2.163,0.941V0h-0.662l-4.911,1.807H0 v4.911h2.316L3.257,3.69h3.181v15.776L3.97,20.407V21.4h10.61v-0.993l-2.213-0.941V3.69h4.783v16.081l-1.857,0.763V21.4h8.55 v-0.866l-1.858-0.763V9.873c0.458-0.127,1.247-0.229,1.832-0.229c1.578,0,2.875,0.865,2.875,3.46v6.667l-1.857,0.763V21.4h8.804 v-0.866L31.527,19.771z\" fill=\"#fff\" data-reactid=\"119\"></path></g></svg></a></div></div><div class=\"GridContainer\" data-reactid=\"120\"><div class=\"GridRow\" data-reactid=\"121\"><div class=\"ErrorPage-container\" data-reactid=\"122\"><h2 class=\"ErrorPage-pagetitle\" data-reactid=\"123\">Error 404</h2><h1 class=\"ErrorPage-title\" data-reactid=\"124\">Page not found</h1><p class=\"ErrorPage-text\" data-reactid=\"125\"><!-- react-text: 126 -->We couldn\u2019t find anything at this address. Please check the URL or go to the <!-- /react-text --><a href=\"/\" data-reactid=\"127\">homepage</a><!-- react-text: 128 -->.<!-- /react-text --></p></div></div></div></div><!-- react-empty: 129 --><div style=\"display:none;\" data-reactid=\"130\"><svg\n xmlns=\"http://www.w3.org/2000/svg\"\n xmlns:xlink=\"http://www.w3.org/1999/xlink\"\n height=\"500\"\n width=\"500\"\n viewBox=\"0 0 500 500\"\n aria-labelledby=\"title desc\"\n>\n <title id=\"title\">Filters SVG</title>\n <defs>\n <filter id=\"bleed\" filterUnits=\"objectBoundingBox\">\n <feColorMatrix\n type=\"matrix\"\n values=\"1 0 0 0 0, 0 0.15 0 0 0, 0 0 .20 0 0, 0 0 0 1 0\"\n />\n </filter>\n </defs>\n</svg></div><div class=\"ThirdPartySlot\" id=\"third-party--viewport-top\" data-reactid=\"131\"></div><div class=\"ThirdPartySlot\" id=\"third-party--viewport-takeover\" data-reactid=\"132\"></div><div class=\"ThirdPartySlot\" id=\"third-party--viewport-bottom\" data-reactid=\"133\"></div></div></div>\n <script>\n window.initialStoreTree = {\"bodyClasses\":[],\"categoryPostIDs\":{},\"commentsExpanded\":{},\"contentLanguage\":null,\"dispatcher\":{\"backend\":null,\"node\":null,\"type\":null},\"documentCloud\":{\"document\":{},\"embedUrl\":\"\",\"text\":{}},\"documentIDs\":[],\"error\":{\"message\":\"Page not found\",\"status\":404},\"featureIDs\":[],\"featuresLanguage\":{},\"googleAMPUrl\":\"\",\"hamburgerColor\":null,\"host\":\"theintercept.com\",\"inInitialRender\":false,\"languageLanding\":{},\"liveBlogsUpdatesIDs\":{},\"loading\":false,\"mediaPlayer\":null,\"newsletter\":{\"form\":{\"description\":\"\",\"status\":\"\"}},\"podcastPage\":{},\"podcastsHomepage\":{\"speakingIDs\":[]},\"postLanding\":{\"redirect\":null},\"postsMetaIDs\":{},\"resources\":{\"alerts\":{},\"annotationSets\":{},\"annotations\":{},\"categories\":{},\"comments\":{},\"documents\":{},\"liveBlogs\":{},\"platform\":{\"theintercept\":{\"Article\":{},\"Author\":{},\"Document\":{},\"GeoLocation\":{},\"HttpReturn\":{},\"Podcast\":{},\"PodcastEpisode\":{},\"Section\":{},\"bySpeakingID\":{},\"documentArchives\":{},\"documentReleases\":{},\"nodePromos\":{\"default\":{}}},\"theintercept-brasil\":{\"HttpReturn\":{},\"bySpeakingID\":{},\"latestPromos\":[],\"nodePromos\":{\"default\":{}}}},\"postCommentMeta\":{},\"posts\":{},\"promoBanners\":{},\"series\":{},\"seriesDocuments\":{},\"staff\":{},\"taxonomies\":{},\"timeline\":{}},\"reverseChronIDs\":{},\"route\":{\"names\":[],\"params\":{},\"path\":\"\",\"pathname\":\"\",\"query\":{}},\"routed\":false,\"scrollToPostComments\":null,\"searchResultIDs\":[],\"seriesHomepage\":{\"curatedItems\":{\"postIds\":[],\"seriesSlugs\":[]},\"recentItems\":{\"postIds\":[],\"seriesSlugs\":[]}},\"seriesPostIDs\":{},\"sidToday\":{\"search\":{\"lastCursor\":null,\"loading\":false,\"speakingIDs\":[],\"totalCount\":null}},\"sidTodayFilesUpdateReports\":{\"category\":{}},\"specialSeriesItems\":[],\"squirrelDocumentIDs\":[],\"squirrelIDs\":[],\"staffIDs\":[],\"surveillanceCatalogData\":null,\"surveillanceCatalogVendors\":null,\"tocChapters\":{},\"tracking\":{\"currentUrl\":\"https://theintercept.com/2020/10/27/senator-perdue-ossoff-china/feed/\",\"previousUrl\":null}};\n window.config = {\"assets\":{\"host\":\"\",\"webpack\":false},\"aws_static\":\"https://static.theintercept.com\",\"coral_talk_api_origin\":\"https://talk.theintercept.com\",\"coral_talk_origin\":\"https://talk.theintercept.com\",\"coral_talk_permalink_cutover_date\":\"2019-06-24T14:00:00.000Z\",\"donation_base_url\":\"https://join.theintercept.com/donate/\",\"donation_base_url_brasil\":\"http://catarse.me/intercept/\",\"env\":{\"NODE_ENV\":\"production\"},\"facebook\":{\"tracking_pixel_id\":\"2151258874911575\"},\"google\":{\"id\":\"UA-79475609-15\"},\"graphql_realm_id\":\"UmVhbG06NTJiOWMwOGEtMjQwYS00NzMxLThlYTAtMjMyY2RiYTYwNzBh\",\"graphql_realm_id__brasil\":\"UmVhbG1Db250ZW50OjE3NTU4MzMyLWUwZTQtNDIyYy1iNDcyLWZkMDQzMmRiOGRhYw==\",\"graphql_url\":\"http://read.usq.flmcloud.local:3002/graphql\",\"hash\":\"42e762a729b53f810f04\",\"host\":\"theintercept.com\",\"imgix\":{\"additional_origins\":[\"https://firstlook.org\"],\"domain\":\"theintercept.imgix.net\"},\"logs\":{\"level\":\"info\"},\"onsite_origins\":[\"https://theintercept.com\"],\"origin\":\"https://theintercept.com\",\"override_private_wp_host\":\"theintercept.com\",\"parsely\":{\"endpoint\":\"https://c.prod.theintercept.com/a\",\"site_id\":\"theintercept.com\"},\"piano\":{\"application_id\":\"hsZyoAWmIE\",\"origin\":\"https://o.prod.theintercept.com\"},\"port\":8080,\"private_wp_origin\":\"https://wp.theintercept.com\",\"public_api_origin\":\"https://theintercept.com\",\"public_wp_origin\":\"https://theintercept.com\",\"request_timeout\":30000,\"set_headers\":{\"Access-Control-Allow-Origin\":\"*\",\"Referrer-Policy\":\"strict-origin-when-cross-origin\",\"Strict-Transport-Security\":\"max-age=63072000; includeSubDomains; preload\",\"Vary\":\"Accept-Encoding\",\"X-Content-Type-Options\":\"nosniff\",\"X-Frame-Options\":\"SAMEORIGIN\",\"X-Xss-Protection\":\"1; mode=block\"},\"site_prefix\":\"\"};\n window.__COUNTRY_CODE__ = \"US\";\n </script>\n \n \n <script type=\"application/ld+json\">\n {\"url\":\"https://theintercept.com/2020/10/27/senator-perdue-ossoff-china/feed/\"}\n </script>\n \n <div id=\"parsely-root\" style=\"display: none\">\n <div id=\"parsely-cfg\" data-parsely-site=\"theintercept.com\"></div>\n </div>\n \n <script src=\"/assets/app42e762a729b53f810f04.js\"></script>\n </body>\n</html>\n"
Actually you were loading the file outside the loop which is actually closed! so you just load an empty soup!
Also since you are dealing with broken HTML
, the rel is equal to rel=\"canonical\"
so you've to take care about that. or to explicit specify it or to use *
within selectors.
from bs4 import BeautifulSoup
with open('a.html') as f:
soup = BeautifulSoup(f.read(), 'lxml')
for i in soup.select('link[rel*=canonical]'):
print(i['href'])
Output:
\"https://theintercept.com/2020/10/27/senator-perdue-ossoff-china/feed/\"/