pythonxpathscrapy

Scrapy Xpath select specific element from file-extension.net


I'm having some trouble to understand xpath.

I'm trying to scrape all the magic number from http://file-extension.net

Let's take that link as an example: http://file-extension.net/seeker/file_extension_c10

Part of its source code:

<table border=4 RULES=ROWS FRAME=HSIDES width=728>
            <tr class="tabhead">
              <td></td>
              <td><b>Website</b></td>
              <td><b>&nbsp;EXT&nbsp;</b></td>
              <td><b>&nbsp;Filetype description</b>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;</td>
            </tr>
            
<tr class="rre"><td>&nbsp;<img src="images/icon-filext.png" width="16" height="16"> &nbsp;</td><td><a href="http://filext.com/file-extension/C10">FILExt</a></td><td>&nbsp;<a class='fesl' href='file_extension_c10'>C10</a>&nbsp;</td><td>&nbsp;<a class='fesl' href='program_extension_irig'>IRIG</a> 106 <a class='fesl' href='program_extension_original'>Original</a> <a class='fesl' href='program_extension_recording'>Recording</a> <a class='fesl' href='program_extension_file'>File</a> (<a class='fesl' href='program_extension_range'>Range</a> <a class='fesl' href='program_extension_commanders'>Commanders</a> <a class='fesl' href='program_extension_council'>Council</a>)</td></tr>

<tr class="rro"><td>&nbsp;<img src="images/icon-fsorg.png" width="16" height="16"> &nbsp;</td><td><a href="http://www.file-extensions.org/c10-file-extension">File Extensions</a></td><td>&nbsp;<a class='fesl' href='file_extension_c10'>C10</a>&nbsp;</td><td>&nbsp;<a class='fesl' href='program_extension_irig'>IRIG</a> 106 <a class='fesl' href='program_extension_original'>original</a> <a class='fesl' href='program_extension_recording'>recording</a> <a class='fesl' href='program_extension_file'>file</a></td></tr>

<tr class="rre"><td>&nbsp;<img src="images/icon-dotwhat.png" width="16" height="16"> &nbsp;</td><td><a href="http://dotwhat.net/c10/9166/">DotWhat</a></td><td>&nbsp;<a class='fesl' href='file_extension_c10'>C10</a>&nbsp;</td><td>&nbsp;<a class='fesl' href='program_extension_split'>Split</a> <a class='fesl' href='program_extension_compressed'>Compressed</a> <a class='fesl' href='program_extension_archive'>Archive</a> <a class='fesl' href='program_extension_file'>File</a> <a class='fesl' href='program_extension_part'>Part</a> 10</td></tr>

<tr class="rro"><td>&nbsp;<img src="images/icon-fsorg.png" width="16" height="16"> &nbsp;</td><td><a href="http://www.file-extensions.org/c10-file-extension">File Extensions</a></td><td>&nbsp;<a class='fesl' href='file_extension_c10'>C10</a>&nbsp;</td><td>&nbsp;<a class='fesl' href='program_extension_split'>Split</a> <a class='fesl' href='program_extension_multi'>Multi</a>-<a class='fesl' href='program_extension_volume'>volume</a> ACE <a class='fesl' href='program_extension_compressed'>compressed</a> <a class='fesl' href='program_extension_file'>file</a> <a class='fesl' href='program_extension_archive'>archive</a></td></tr>

<tr class="rre"><td>&nbsp;<img src="images/icon-trid.png" width="16" height="16"> &nbsp;</td><td><a href="http://mark0.net/soft-trid-e.html">TrID</a></td><td>&nbsp;<a class='fesl' href='file_extension_c10'>C10</a>&nbsp;</td><td>&nbsp;<a class='fesl' href='program_extension_virtual'>Virtual</a> MC-10 <a class='fesl' href='program_extension_tape'>tape</a> <a class='fesl' href='program_extension_image'>image</a><br>&nbsp;<b><small>Header Hexdump</b>: <span class='hexdump'>&nbsp;55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 55 &nbsp;</span></small></td></tr>

<tr class="rro"><td>&nbsp;<img src="images/icon-filext.png" width="16" height="16"> &nbsp;</td><td><a href="http://filext.com/file-extension/C10">FILExt</a></td><td>&nbsp;<a class='fesl' href='file_extension_c10'>C10</a>&nbsp;</td><td>&nbsp;<a class='fesl' href='program_extension_winace'>WinAce</a> <a class='fesl' href='program_extension_compressed'>Compressed</a> <a class='fesl' href='program_extension_file'>File</a> <a class='fesl' href='program_extension_split'>Split</a> <a class='fesl' href='program_extension_portion'>Portion</a> of <a class='fesl' href='program_extension_compressed'>Compressed</a> <a class='fesl' href='program_extension_file'>File</a> (e-<a class='fesl' href='program_extension_merge'>merge</a> <a class='fesl' href='program_extension_gmbh'>GmbH</a>)</td></tr>

<tr class="rre"><td>&nbsp;<img src="images/icon-fileinfo.png" width="16" height="16"> &nbsp;</td><td><a href="http://www.fileinfo.com/extension/c10">FileInfo</a></td><td>&nbsp;<a class='fesl' href='file_extension_c10'>C10</a>&nbsp;</td><td>&nbsp;<a class='fesl' href='program_extension_winace'>WinAce</a> <a class='fesl' href='program_extension_split'>Split</a> <a class='fesl' href='program_extension_archive'>Archive</a> <a class='fesl' href='program_extension_part'>Part</a> 10</td></tr>

          </table>

I only want to get the Filetype Description from Trid (the one with the hex value)

Problem is I don't know how cause each word from Filtype Description are links.

here is my code:

for sel in response.xpath('//table[@border=4]'):
    hex = sel.xpath('//span[@class="hexdump"]/text()').extract_first(default='Rien t nul')
    if len(hex) > 7:
        ext = sel.xpath('//a[text()="TrID"]/@href.a[@class="fesl"]/text()').extract()
        print "Nom : %s Hex %s " % (ext,hex)

Of course //a[text()="TrID"]/@href.a[@class="fesl" doesn't work But this is what I want:

If you find a link name which contains "Trid" give me it's file description

Any idea?


Solution

  • '//td[./a[contains(text(), "TrID")]]/following-sibling::td[2]//text()'
    

    Just change TrID for another text in the row you want.