bashawkgrepxargsdd

Using grep to search for hex strings in a file


Does anyone know how to get grep, or similar tool, to retrieve offsets of hex strings in a file?

I have a bunch of hexdumps (from GDB) that I need to check for strings and then run again and check if the value has changed.

I have tried hexdump and dd, but the problem is because it's a stream, I lose my offset for the files.

Someone must have had this problem and a workaround. What can I do?

To clarify:

Perl COULD be a option, but at this point, I would assume my lack of knowledge with bash and its tools is the main culprit.

Desired output format

It's a little hard to explain the output I am getting since I really am not getting any output.

I am anticipating (and expecting) something along the lines of:

<offset>:<searched value>

Which is the pretty well standard output I would normally get with grep -URbFo <searchterm> . > <output>

What I tried:

A. Problem is, when I try to search for hex values, I get the problem of if just not searching for the hex values, so if I search for 00 I should get like a million hits, because thats always the blankspace, but instead its searching for 00 as text, so in hex, 3030. Any idea's?

B. I CAN force it through hexdump or something of the link but because its a stream it will not give me the offsets and filename that it found a match in.

C. Using grep -b option doesnt seem to work either, I did try all the flags that seemed useful to my situation, and nothing worked.

D. Using xxd -u /usr/bin/xxd as an example I get a output that would be useful, but I cannot use that for searching..

0004760: 73CC 6446 161E 266A 3140 5E79 4D37 FDC6  s.dF..&j1@^yM7..
0004770: BF04 0E34 A44E 5BE7 229F 9EEF 5F4F DFFA  ...4.N[."..._O..
0004780: FADE 0C01 0000 000C 0000 0000 0000 0000  ................

Nice output, just what I want to see, but it just doesn't work for me in this situation..

E. Here are some of the things I've tried since posting this:

xxd -u /usr/bin/xxd | grep 'DF'
00017b0: 4010 8D05 0DFF FF0A 0300 53E3 0610 A003  @.........S.....

root# grep -ibH "df" /usr/bin/xxd
Binary file /usr/bin/xxd matches
xxd -u /usr/bin/xxd | grep -H 'DF'
(standard input):00017b0: 4010 8D05 0DFF FF0A 0300 53E3 0610 A003  @.........S.....

Solution

  • We tried several things before arriving at an acceptable solution:

    xxd -u /usr/bin/xxd | grep 'DF'
    00017b0: 4010 8D05 0DFF FF0A 0300 53E3 0610 A003  @.........S.....
    
    
    root# grep -ibH "df" /usr/bin/xxd
    Binary file /usr/bin/xxd matches
    xxd -u /usr/bin/xxd | grep -H 'DF'
    (standard input):00017b0: 4010 8D05 0DFF FF0A 0300 53E3 0610 A003  @.........S.....
    

    Then found we could get usable results with

    xxd -u /usr/bin/xxd > /tmp/xxd.hex ; grep -H 'DF' /tmp/xxd
    

    Note that using a simple search target like 'DF' will incorrectly match characters that span across byte boundaries, i.e.

    xxd -u /usr/bin/xxd | grep 'DF'
    00017b0: 4010 8D05 0DFF FF0A 0300 53E3 0610 A003  @.........S.....
    --------------------^^
    

    So we use an ORed regexp to search for ' DF' OR 'DF ' (the searchTarget preceded or followed by a space char).

    The final result seems to be

    xxd -u -ps -c 10000000000 DumpFile > DumpFile.hex
    egrep ' DF|DF ' Dumpfile.hex
    
    0001020: 0089 0424 8D95 D8F5 FFFF 89F0 E8DF F6FF  ...$............
    -----------------------------------------^^
    0001220: 0C24 E871 0B00 0083 F8FF 89C3 0F84 DF03  .$.q............
    --------------------------------------------^^