vb.netpowershellbatch-filecertutil

Replicating Certutil.exe's -decodehex output exactly

I have done a ton more research since I first posted this question and I think I had a few terms goofed up as well.

Dilemma: My company's Information Security team has flagged certutil.exe as a potentially dangerous application to be used after a recent phishing attack. This really stinks because we LOVE certutil.exe for the lightning fast and dead-nuts accurate conversion of hex files into ascii files. These ascii files have to be converted exactly as certutil.exe -decodehex performs in order to be parsed by another program to read and interpret various pieces of data produced by a separate in-house program.

I have a PowerShell script which seems to convert Hex to ASCII very accurately, BUT, it never seems to "finish" as though maybe the while loop is dorked up. Additionally, because of how the stream is is broken down, there are far too many line-breaks in the resulting file. The files are typically 7 MB a piece, roughly 7-8 billion characters long, and roughly 11 line breaks.

The script I mention is below and is an adaptation of the work presented in this link. Instead of converting the data stream into a Hex representation of the Hex data, I convert it to ASCII.

$bufferSize = 65536
$ASCIIFile = "C:\FooBar.dat"
$stream = [System.IO.File]::OpenRead(
  "C:\FixedOutput.dat")
while ( $stream.Position -lt $stream.Length ) {
#BEGIN CALLOUT A
  $buffer = new-object Byte[] $bufferSize
  $bytesRead = $stream.Read($buffer, 0, $bufferSize)
#END CALLOUT A
  for ( $line = 0; $line -lt [Math]::Floor($bytesRead /
  16); $line++ ) {
  
$slice = $buffer[($line * 16)..(($line * 16) + 15)]

$bytes=[System.Text.Encoding]::ASCII.GetString($slice)
$asc = -join($bytes-split"(..)"|?{$_}|%{[char][convert]::ToByte($_,16)})
$asc | Write-Host >> $ASCIIFile -NoNewline

  }
#BEGIN CALLOUT B
  if ( $bytesRead % 16 -ne 0 ) {
    $slice = $buffer[($line * 16)..($bytesRead - 1)]
    $output = ""
    foreach ( $byte in $slice ) {
      $output=[System.Text.Encoding]::ASCII.GetString($byte)
    }
    $output | ADD-Content $Asciifile
#END CALLOUT B
  }
}


$stream.Close()

Additionally, I had adapted the PowerShell code from this S.O. article in addition to the presumably previous duplicate question and answer. The problem with this set of code, is the output still took 15 minutes or so, but the output is not identical to certutil.exe -decodehex so the information cannot be parsed by our in-house program!

Additionally, I can literally copy and paste the hex data from the original file, paste it into a hex editor and then save the output as a new file to get what I need.

Problem is, we often have 30 - 40 of these files at once and we need a lightning fast solution. . .

I've looked for VB.net (my 2nd most familiar language) solutions, but they are comparable in methods as the PowerShell methods I have found, and nothing adequately takes an entire file and puts it to ASCII with relative ease or Accuracy.

UPDATE:

In addition to re-formatting the question, I have also put to test the very well-detailed answer below from TheMadTechnician and this brought magnificently glorious tears to my eyes. If I could reach through the silicon and kiss you, I probably would. TEXTBOOK MATCH. LIGHTNING FAST. BEAUTIFUL.

Now. . . . let us hope my I.S. Dept. doesn't flag this methodology as well and make noise about it. . .

I modified the -join statement since I'm concatenating the files prior to calling PowerShell from within a Batch script, but this would work beautifully within PowerShell as well.

Lastly, since our I.S. Dept. restricts the usage of .ps1 scripts, a while back, I found an awesome option to embed complex commands as Base64 strings instead and then calling this using Start /MIN powershell -encodedcommand _insertEncodedCommandHere_

Again - I cannot thank you enough! If I ever get a working method utilizing this same crypt32.dll library via VB.Net, I'll come back and post it as an answer as well, but YOU have won the prize!

Solution

  • If you don't mind using the Crypt32.dll library you can add it's CryptStringToBinary method to convert your hex to binary, and then just dump the array to a file.

    Credit where credit is due! I didn't come up with the conversion bit of this, the crypto genius Vadims Podāns over at Sysadmins LV did. Please check out his post about it here: https://www.sysadmins.lv/blog-en/convert-data-between-binary-hex-and-base64-in-powershell.aspx

    So, first we have to add the method from the DLL, we can do that by defining the signature, and using Add-Type like this:

    $signature = @"
[DllImport("Crypt32.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern bool CryptStringToBinary(
    string pszString,
    int cchString,
    int dwFlags,
    byte[] pbBinary,
    ref int pcbBinary,
    int pdwSkip,
    ref int pdwFlags
);
"@
Add-Type -MemberDefinition $signature -Namespace PKI -Name Crypt32 -UsingNamespace "System.Text"

    Mind you, he adds two methods on his page, but for only converting hex to binary we just need the one. He also shows how to go binary to hex, but you can read about that on his page.

    Now, you say you have several files you need to merge right? We can just stitch them together in memory and convert them all at once. You don't really say how they're named or how they're ordered, but let's assume they have sequential names (HexFile.01, HexFile.02 and so on). You can figure out how to get them in the right order I'm sure. Let's load them up:

    $RawHex = (Get-content HexFile.* -raw) -join ''

    That was easy! So now we have the type added, and we have the original hex formatted file reconstructed in memory, now we just need to convert it to binary and dump it to a file. To do the conversion I'll go ahead and use Vadims' function since it works so well, modified ever so slightly so that it actually outputs the byte array:

    function Convert-HexToBinary ([string]$hex) {
    # decoding hashtable contains universal flags: Base64Any and HexAny. The function attempts to
    # get the correct input string format and then decode it
    $decoding = @{'Base64Header' = 0; 'Base64' = 1; 'HexRaw' = 12; 'Hex' = 4; 'HexAddr' = 10;
        'HexAscii' = 5; 'HexAddrAscii' = 11; 'Base64Any' = 6; 'HexAny' = 8}
    # initialize variables to receive resulting byte array size and actual input string format
    $pcbBinary = 0
    $pdwFlags = 0
    # call CryptStringToBinary to get resulting byte array size and actual input string format
    if ([PKI.Crypt32]::CryptStringToBinary($hex,$hex.Length,$decoding['HexAny'],$null,[ref]$pcbBinary,0,[ref]$pdwFlags)) {
        # create enough large byte array
        $array = New-Object byte[] -ArgumentList $pcbBinary
        # call the function again to write converted bytes to a byte array
        [void][PKI.Crypt32]::CryptStringToBinary($hex,$hex.Length,$decoding['HexAny'],$array,[ref]$pcbBinary,0,[ref]$pdwFlags)
        $array
    } else {
        Write-Warning $((New-Object ComponentModel.Win32Exception ([Runtime.InteropServices.Marshal]::GetLastWin32Error())).Message)
    }
}

    Now that function dumps the byte array, and we'll capture that in a variable. Once we have it we can use the [io.file] class to write it to a file directly.

    $BinArr = Convert-HexToBinary $RawHex
[io.file]::WriteAllBytes("C:\Path\To\OutFile.bin",$BinArr)

    In my own testing this comes out identical to using certutil. I get the same filehash either way I do it.