pdffontsitextpdfsharppdf-extraction

Extracting Text from a PDF with CID fonts


I'm writing a web app that extracts a line at the top of each page in a PDF. The PDFs come from different versions of a product and could go through a number of PDF printers, also in different versions and also different settings.

So far using PDFSharp and iTextSharp I have managed to get it to work for all versions of PDFs. My hang-up is with documents that have CID fonts (Identity-H).

I have written a partial parser to find the font table reference and the text blocks, but converting these to readable text is beating me.

Does anyone have either: - a parser (like this one https://stackoverflow.com/a/1732265/5169050) that copes with CID fonts; or - some example code for how to parse a pages resource dictionary to find the pages fonts and get its ToUnicode stream to help finish off this example (https://stackoverflow.com/a/4048328/5169050)

We have to use iTextSharp 4.1 to retain the free-to-use license.

Here's my partial parser.

public string ExtractTextFromCIDPDFBytes(byte[] input)
{
    if (input == null || input.Length == 0) return "";

    try
    {
        // Holds the final result to be returned
        string resultString = "";
        // Are we in a block of text or not
        bool blnInText = false;
        // Holds each line of text before written to resultString
        string phrase = "";
        // Holds the 4-character hex codes as they are built
        string hexCode = "";
        // Are we in a font reference or not (much like a code block)
        bool blnInFontRef = false;
        // Holds the last font reference and therefore the CMAP table
        // to be used for any text found after it
        string currentFontRef = "";

        for (int i = 0; i < input.Length; i++)
        {
            char c = (char)input[i];

            switch (c)
            {
                case '<':
                    {
                        blnInText = true;
                        break;
                    }
                case '>':
                    {
                        resultString = resultString + Environment.NewLine + phrase;
                        phrase = "";
                        blnInText = false;
                        break;
                    }
                case 'T':
                    {
                        switch (((char)input[i + 1]).ToString().ToLower())
                        {
                            case "f":
                                {
                                    // Tf represents the start of a font table reference
                                    blnInFontRef = true;
                                    currentFontRef = "";
                                    break;
                                }
                            case "d":
                                {
                                    // Td represents the end of a font table reference or
                                    // the start of a text block
                                    blnInFontRef = false;
                                    break;
                                }
                        }
                        break;
                    }
                default:
                    {
                        if (blnInText)
                        {
                            // We are looking for 4-character blocks of hex characters
                            // These will build up a number which refers to the index
                            // of the glyph in the CMAP table, which will give us the
                            // character
                            hexCode = hexCode + c;
                            if (hexCode.Length == 4)
                            {
                                // TODO - translate code to character
                                char translatedHexCode = c;



                                phrase = phrase + translatedHexCode;
                                // Blank it out ready for the next 4
                                hexCode = "";
                            }
                        }
                        else
                        {
                            if (blnInFontRef)
                            {
                                currentFontRef = currentFontRef + c;
                            }
                        }
                        break;
                    }
            }
        }

        return resultString;
    }
    catch
    {
        return "";
    }
}

Solution

  • It took a while but I finally have some code to read plain text from a Identity-H encoded PDF. I post it here to help others, and I know there will be ways to improve upon it. For instance, I haven't touched character mappings (beginbfchar) and my ranges are not actually ranges. I've spent over a week on this already and can't justify the time unless we hit files that work differently. Sorry.

    Usage:

    PdfDocument inputDocument = PDFHelpers.Open(physcialFilePath, PdfDocumentOpenMode.Import)
    foreach (PdfPage page in inputDocument.Pages)
    {
        for (Int32 index = 0; index < page.Contents.Elements.Count; index++)
        {
            PdfDictionary.PdfStream stream = page.Contents.Elements.GetDictionary(index).Stream;
            String outputText = new PDFParser().ExtractTextFromPDFBytes(stream.Value).Replace(" ", String.Empty);
    
            if (outputText == "" || outputText.Replace("\n\r", "") == "")
            {
                // Identity-H encoded file
                string[] hierarchy = new string[] { "/Resources", "/Font", "/F*" };
                List<PdfItem> fonts = PDFHelpers.FindObjects(hierarchy, page, true);
                outputText = PDFHelpers.FromUnicode(stream, fonts);
            }
        }
    }
    

    And the actual helper class, which I'll post in its entirety, because they are all used in the example, and be because I've found so few complete examples myself when I was trying to solve this issue. The helper uses both PDFSharp and iTextSharp to be able to able to open PDFs pre- and post-1.5, ExtractTextFromPDFBytes to read in a standard PDF, and my FindObjects (to search the document tree and return objects) and FromUnicode that takes encrypted texts and a fonts collection to translate it.

    using PdfSharp.Pdf;
    using PdfSharp.Pdf.Content;
    using PdfSharp.Pdf.Content.Objects;
    using System;
    using System.Collections.Generic;
    using System.IO;
    
    namespace PdfSharp.Pdf.IO
    {
        /// <summary>
        /// uses itextsharp 4.1.6 to convert any pdf to 1.4 compatible pdf, called instead of PdfReader.open
        /// </summary>
        static public class PDFHelpers
        {
            /// <summary>
            /// uses itextsharp 4.1.6 to convert any pdf to 1.4 compatible pdf, called instead of PdfReader.open
            /// </summary>
            static public PdfDocument Open(string PdfPath, PdfDocumentOpenMode openmode)
            {
                return Open(PdfPath, null, openmode);
            }
            /// <summary>
            /// uses itextsharp 4.1.6 to convert any pdf to 1.4 compatible pdf, called instead of PdfReader.open
            /// </summary>
            static public PdfDocument Open(string PdfPath, string password, PdfDocumentOpenMode openmode)
            {
                using (FileStream fileStream = new FileStream(PdfPath, FileMode.Open, FileAccess.Read))
                {
                    int len = (int)fileStream.Length;
                    // TODO: Setting this byteArray causes the out of memory exception which is why we
                    // have the 70mb limit. Solve this and we can increase the file size limit
                    System.Diagnostics.Process proc = System.Diagnostics.Process.GetCurrentProcess();
                    long availableMemory = proc.PrivateMemorySize64 / 1024 / 1024; //Mb of RAM allocated to this process that cannot be shared with other processes
                    if (availableMemory < (fileStream.Length / 1024 / 1024))
                    {
                        throw new Exception("The available memory " + availableMemory + "Mb is not enough to open, split and save a file of " + fileStream.Length / 1024 / 1024);
                    }
    
                    try
                    {
                        Byte[] fileArray = new Byte[len];
                        fileStream.Read(fileArray, 0, len);
                        fileStream.Close();
                        fileStream.Dispose();
    
    
                        PdfDocument result = Open(fileArray, openmode);
                        if (result.FullPath == "")
                        {
                            // The file was converted to a v1.4 document and only exists as a document in memory
                            // Save over the original file so other references to the file get the compatible version
                            // TODO: It would be good if we could do this conversion without opening every document another 2 times
                            PdfDocument tempResult = Open(fileArray, PdfDocumentOpenMode.Modify);
    
                            iTextSharp.text.pdf.BaseFont bfR = iTextSharp.text.pdf.BaseFont.CreateFont(Environment.GetEnvironmentVariable("SystemRoot") + "\\fonts\\arial.ttf", iTextSharp.text.pdf.BaseFont.IDENTITY_H, iTextSharp.text.pdf.BaseFont.EMBEDDED);
                            bfR.Subset = false;
    
                            tempResult.Save(PdfPath);
                            tempResult.Close();
                            tempResult.Dispose();
                            result = Open(fileArray, openmode);
                        }
    
                        return result;
                    }
                    catch (OutOfMemoryException)
                    {
                        fileStream.Close();
                        fileStream.Dispose();
    
                        throw;
                    }
                }
            }
    
            /// <summary>
            /// uses itextsharp 4.1.6 to convert any pdf to 1.4 compatible pdf, called instead of PdfReader.open
            /// </summary>
            static public PdfDocument Open(byte[] fileArray, PdfDocumentOpenMode openmode)
            {
                return Open(new MemoryStream(fileArray), null, openmode);
            }
            /// <summary>
            /// uses itextsharp 4.1.6 to convert any pdf to 1.4 compatible pdf, called instead of PdfReader.open
            /// </summary>
            static public PdfDocument Open(byte[] fileArray, string password, PdfDocumentOpenMode openmode)
            {
                return Open(new MemoryStream(fileArray), password, openmode);
            }
    
            /// <summary>
            /// uses itextsharp 4.1.6 to convert any pdf to 1.4 compatible pdf, called instead of PdfReader.open
            /// </summary>
            static public PdfDocument Open(MemoryStream sourceStream, PdfDocumentOpenMode openmode)
            {
                return Open(sourceStream, null, openmode);
            }
            /// <summary>
            /// uses itextsharp 4.1.6 to convert any pdf to 1.4 compatible pdf, called instead of PdfReader.open
            /// </summary>
            static public PdfDocument Open(MemoryStream sourceStream, string password, PdfDocumentOpenMode openmode)
            {
                PdfDocument outDoc = null;
                sourceStream.Position = 0;
    
                try
                {
                    outDoc = (password == null) ?
                          PdfReader.Open(sourceStream, openmode) :
                          PdfReader.Open(sourceStream, password, openmode);
    
                    sourceStream.Position = 0;
                    MemoryStream outputStream = new MemoryStream();
                    iTextSharp.text.pdf.PdfReader reader = (password == null) ?
                          new iTextSharp.text.pdf.PdfReader(sourceStream) :
                          new iTextSharp.text.pdf.PdfReader(sourceStream, System.Text.ASCIIEncoding.ASCII.GetBytes(password));
                    System.Collections.ArrayList fontList = iTextSharp.text.pdf.BaseFont.GetDocumentFonts(reader, 1);
                }
                catch (PdfSharp.Pdf.IO.PdfReaderException)
                {
                    //workaround if pdfsharp doesn't support this pdf
                    sourceStream.Position = 0;
                    MemoryStream outputStream = new MemoryStream();
                    iTextSharp.text.pdf.PdfReader reader = (password == null) ?
                          new iTextSharp.text.pdf.PdfReader(sourceStream) :
                          new iTextSharp.text.pdf.PdfReader(sourceStream, System.Text.ASCIIEncoding.ASCII.GetBytes(password)); 
                    iTextSharp.text.pdf.PdfStamper pdfStamper = new iTextSharp.text.pdf.PdfStamper(reader, outputStream);
                    pdfStamper.FormFlattening = true;
                    pdfStamper.Writer.SetPdfVersion(iTextSharp.text.pdf.PdfWriter.PDF_VERSION_1_4);
                    pdfStamper.Writer.CloseStream = false;
                    pdfStamper.Close();
    
                    outDoc = PdfReader.Open(outputStream, openmode);
                }
    
                return outDoc;
            }
    
            /// <summary>
            /// Uses a recurrsive function to step through the PDF document tree to find the specified objects.
            /// </summary>
            /// <param name="objectHierarchy">An array of the names of objects to look for in the tree. Wildcards can be used in element names, e.g., /F*. The order represents 
            /// a top-down hierarchy if followHierarchy is true. 
            /// If a single object is passed in array it should be in the level below startingObject, or followHierarchy set to false to find it anywhere in the tree</param>
            /// <param name="startingObject">A PDF object to parse. This will likely be a document or a page, but could be any lower-level item</param>
            /// <param name="followHierarchy">If true the order of names in the objectHierarchy will be used to search only that branch. If false the whole tree will be parsed for 
            /// any items matching those in objectHierarchy regardless of position</param>
            static public List<PdfItem> FindObjects(string[] objectHierarchy, PdfItem startingObject, bool followHierarchy)
            {
                List<PdfItem> results = new List<PdfItem>();
                FindObjects(objectHierarchy, startingObject, followHierarchy, ref results, 0);
                return results;
            }
    
            static private void FindObjects(string[] objectHierarchy, PdfItem startingObject, bool followHierarchy, ref List<PdfItem> results, int Level)
            {
                PdfName[] keyNames = ((PdfDictionary)startingObject).Elements.KeyNames;
                foreach (PdfName keyName in keyNames)
                {
                    bool matchFound = false;
                    if (!followHierarchy)
                    {
                        // We need to check all items for a match, not just the top one
                        for (int i = 0; i < objectHierarchy.Length; i++)
                        {
                            if (keyName.Value == objectHierarchy[i] ||
                                (objectHierarchy[i].Contains("*") &&
                                    (keyName.Value.StartsWith(objectHierarchy[i].Substring(0, objectHierarchy[i].IndexOf("*") - 1)) &&
                                    keyName.Value.EndsWith(objectHierarchy[i].Substring(objectHierarchy[i].IndexOf("*") + 1)))))
                            {
                                matchFound = true;
                            }
                        }
                    }
                    else
                    {
                        // Check the item in the hierarchy at this level for a match
                        if (Level < objectHierarchy.Length && (keyName.Value == objectHierarchy[Level] || 
                            (objectHierarchy[Level].Contains("*") &&
                                    (keyName.Value.StartsWith(objectHierarchy[Level].Substring(0, objectHierarchy[Level].IndexOf("*") - 1)) &&
                                    keyName.Value.EndsWith(objectHierarchy[Level].Substring(objectHierarchy[Level].IndexOf("*") + 1))))))
                        {
                            matchFound = true;
                        }
                    }
    
                    if (matchFound)
                    {
                        PdfItem item = ((PdfDictionary)startingObject).Elements[keyName];
                        if (item != null && item is PdfSharp.Pdf.Advanced.PdfReference)
                        {
                            item = ((PdfSharp.Pdf.Advanced.PdfReference)item).Value;
                        }
    
                        System.Diagnostics.Debug.WriteLine("Level " + Level.ToString() + " - " + keyName.ToString() + " matched");
    
                        if (Level == objectHierarchy.Length - 1)
                        {
                            // We are at the end of the hierarchy, so this is the target
                            results.Add(item);
                        }
                        else if (!followHierarchy)
                        {
                            // We are returning every matching object so add it
                            results.Add(item);
                        }
    
                        // Call back to this function to search lower levels
                        Level++;
                        FindObjects(objectHierarchy, item, followHierarchy, ref results, Level);
                        Level--;
                    }
                    else
                    {
                        System.Diagnostics.Debug.WriteLine("Level " + Level.ToString() + " - " + keyName.ToString() + " unmatched");
                    }
                }
                Level--;
                System.Diagnostics.Debug.WriteLine("Level " + Level.ToString());
            }
    
            /// <summary>
            /// Uses the Font object to translate CID encoded text to readable text
            /// </summary>
            /// <param name="unreadableText">The text stream that needs to be decoded</param>
            /// <param name="font">A List of PDFItems containing the /Font object containing a /ToUnicode with a CMap</param>
            static public string FromUnicode(PdfDictionary.PdfStream unreadableText, List<PdfItem> PDFFonts)
            {
                Dictionary<string, string[]> fonts = new Dictionary<string, string[]>();
    
                // Get the CMap from each font in the passed array and store them by font name
                for (int font = 0; font < PDFFonts.Count; font++)
                {
                    PdfName[] keyNames = ((PdfDictionary)PDFFonts[font]).Elements.KeyNames;
                    foreach (PdfName keyName in keyNames)
                    {
                        if (keyName.Value == "/ToUnicode") {
                            PdfItem item = ((PdfDictionary)PDFFonts[font]).Elements[keyName];
                            if (item != null && item is PdfSharp.Pdf.Advanced.PdfReference)
                            {
                                item = ((PdfSharp.Pdf.Advanced.PdfReference)item).Value;
                            }
                            string FontName = "/F" + font.ToString();
                            string CMap = ((PdfDictionary)item).Stream.ToString();
    
                            if (CMap.IndexOf("beginbfrange") > 0)
                            {
                                CMap = CMap.Substring(CMap.IndexOf("beginbfrange") + "beginbfrange".Length);
    
                                if (CMap.IndexOf("endbfrange") > 0)
                                {
                                    CMap = CMap.Substring(0, CMap.IndexOf("endbfrange") - 1);
    
                                    string[] CMapArray = CMap.Split(new string[] { "\r\n" }, StringSplitOptions.RemoveEmptyEntries);
                                    fonts.Add(FontName, CMapArray);
                                }
                            }
                            break;
                        }
                    }
                }
    
                // Holds the final result to be returned
                string resultString = "";
    
                // Break the input text into lines
                string[] lines = unreadableText.ToString().Split(new string[] {"\n"} , StringSplitOptions.RemoveEmptyEntries);
    
                // Holds the last font reference and therefore the CMAP table
                // to be used for any text found after it
                string[] currentFontRef = fonts["/F0"];
    
                // Are we in a block of text or not? They can break across lines so we need an identifier
                bool blnInText = false;
    
                for (int line = 0; line < lines.Length; line++)
                {
                    string thisLine = lines[line].Trim();
    
                    if (thisLine == "q")
                    {
                        // I think this denotes the start of a text block, and where we need to reset to the default font
                        currentFontRef = fonts["/F0"];
                    }
                    else if (thisLine.IndexOf(" Td <") != -1)
                    {
                        thisLine = thisLine.Substring(thisLine.IndexOf(" Td <") + 5);
                        blnInText = true;
                    }
    
                    if (thisLine.EndsWith("Tf"))
                    {
                        // This is a font assignment. Take note of this and use this fonts ToUnicode map when we find text
                        if (fonts.ContainsKey(thisLine.Substring(0, thisLine.IndexOf(" "))))
                        {
                            currentFontRef = fonts[thisLine.Substring(0, thisLine.IndexOf(" "))];
                        }
                    } 
                    else if (thisLine.EndsWith("> Tj"))
                    {
                        thisLine = thisLine.Substring(0, thisLine.IndexOf("> Tj"));
                    }
    
                    if(blnInText)
                    {
                        // This is a text block
                        try
                        {
                            // Get the section of codes that exist between angled brackets
                            string unicodeStr = thisLine;
                            // Wrap every group of 4 characters in angle brackets
                            // This will directly match the items in the CMap but also allows the next for to avoid double-translating items
                            unicodeStr = "<" + String.Join("><", unicodeStr.SplitInParts(4)) + ">";
    
                            for (int transform = 0; transform < currentFontRef.Length; transform++)
                            {
                                // Get the last item in the line, which is the unicode value of the glyph
                                string glyph = currentFontRef[transform].Substring(currentFontRef[transform].IndexOf("<"));
                                glyph = glyph.Substring(0, glyph.IndexOf(">") + 1);
    
                                string counterpart = currentFontRef[transform].Substring(currentFontRef[transform].LastIndexOf("<") + 1);
                                counterpart = counterpart.Substring(0, counterpart.LastIndexOf(">"));
    
                                // Replace each item that matches with the translated counterpart
                                // Insert a \\u before every 4th character so it's a C# unicode compatible string
                                unicodeStr = unicodeStr.Replace(glyph, "\\u" + counterpart);
                                if (unicodeStr.IndexOf(">") == 0)
                                {
                                    // All items have been replaced, so lets get outta here
                                    break;
                                }
                            }
                            resultString = resultString + System.Text.RegularExpressions.Regex.Unescape(unicodeStr);
                        }
                        catch
                        {
                            return "";
                        }
                    }
    
                    if (lines[line].Trim().EndsWith("> Tj"))
                    {
                        blnInText = false;
                        if (lines[line].Trim().IndexOf(" 0 Td <") == -1)
                        {
                            // The vertical coords have changed, so add a new line
                            resultString = resultString + Environment.NewLine;
                        }
                        else
                        {
                            resultString = resultString + " ";
                        }
                    }
                }
                return resultString;
            }
    
            // Credit to http://stackoverflow.com/questions/4133377/
            private static IEnumerable<String> SplitInParts(this String s, Int32 partLength)
            {
                if (s == null)
                    throw new ArgumentNullException("s");
                if (partLength <= 0)
                    throw new ArgumentException("Part length has to be positive.", "partLength");
    
                for (var i = 0; i < s.Length; i += partLength)
                    yield return s.Substring(i, Math.Min(partLength, s.Length - i));
            }
        }
    }
    
    public class PDFParser
    {
        /// BT = Beginning of a text object operator
        /// ET = End of a text object operator
        /// Td move to the start of next line
        ///  5 Ts = superscript
        /// -5 Ts = subscript
    
        #region Fields
    
        #region _numberOfCharsToKeep
        /// <summary>
        /// The number of characters to keep, when extracting text.
        /// </summary>
        private static int _numberOfCharsToKeep = 15;
        #endregion
    
        #endregion
    
    
    
        #region ExtractTextFromPDFBytes
        /// <summary>
        /// This method processes an uncompressed Adobe (text) object
        /// and extracts text.
        /// </summary>
        /// <param name="input">uncompressed</param>
        /// <returns></returns>
        public string ExtractTextFromPDFBytes(byte[] input)
        {
            if (input == null || input.Length == 0) return "";
    
            try
            {
                string resultString = "";
    
                // Flag showing if we are we currently inside a text object
                bool inTextObject = false;
    
                // Flag showing if the next character is literal
                // e.g. '\\' to get a '\' character or '\(' to get '('
                bool nextLiteral = false;
    
                // () Bracket nesting level. Text appears inside ()
                int bracketDepth = 0;
    
                // Keep previous chars to get extract numbers etc.:
                char[] previousCharacters = new char[_numberOfCharsToKeep];
                for (int j = 0; j < _numberOfCharsToKeep; j++) previousCharacters[j] = ' ';
    
    
                for (int i = 0; i < input.Length; i++)
                {
                    char c = (char)input[i];
    
                    if (inTextObject)
                    {
                        // Position the text
                        if (bracketDepth == 0)
                        {
                            if (CheckToken(new string[] { "TD", "Td" }, previousCharacters))
                            {
                                resultString += "\n\r";
                            }
                            else
                            {
                                if (CheckToken(new string[] { "'", "T*", "\"" }, previousCharacters))
                                {
                                    resultString += "\n";
                                }
                                else
                                {
                                    if (CheckToken(new string[] { "Tj" }, previousCharacters))
                                    {
                                        resultString += " ";
                                    }
                                }
                            }
                        }
    
                        // End of a text object, also go to a new line.
                        if (bracketDepth == 0 &&
                            CheckToken(new string[] { "ET" }, previousCharacters))
                        {
    
                            inTextObject = false;
                            resultString += " ";
                        }
                        else
                        {
                            // Start outputting text
                            if ((c == '(') && (bracketDepth == 0) && (!nextLiteral))
                            {
                                bracketDepth = 1;
                            }
                            else
                            {
                                // Stop outputting text
                                if ((c == ')') && (bracketDepth == 1) && (!nextLiteral))
                                {
                                    bracketDepth = 0;
                                }
                                else
                                {
                                    // Just a normal text character:
                                    if (bracketDepth == 1)
                                    {
                                        // Only print out next character no matter what.
                                        // Do not interpret.
                                        if (c == '\\' && !nextLiteral)
                                        {
                                            nextLiteral = true;
                                        }
                                        else
                                        {
                                            if (((c >= ' ') && (c <= '~')) ||
                                                ((c >= 128) && (c < 255)))
                                            {
                                                resultString += c.ToString();
                                            }
    
                                            nextLiteral = false;
                                        }
                                    }
                                }
                            }
                        }
                    }
    
                    // Store the recent characters for
                    // when we have to go back for a checking
                    for (int j = 0; j < _numberOfCharsToKeep - 1; j++)
                    {
                        previousCharacters[j] = previousCharacters[j + 1];
                    }
                    previousCharacters[_numberOfCharsToKeep - 1] = c;
    
                    // Start of a text object
                    if (!inTextObject && CheckToken(new string[] { "BT" }, previousCharacters))
                    {
                        inTextObject = true;
                    }
                }
                return resultString;
            }
            catch
            {
                return "";
            }
        }
        #endregion
    
        #region CheckToken
        /// <summary>
        /// Check if a certain 2 character token just came along (e.g. BT)
        /// </summary>
        /// <param name="search">the searched token</param>
        /// <param name="recent">the recent character array</param>
        /// <returns></returns>
        private bool CheckToken(string[] tokens, char[] recent)
        {
            foreach (string token in tokens)
            {
                if (token.Length > 1)
                {
                    if ((recent[_numberOfCharsToKeep - 3] == token[0]) &&
                        (recent[_numberOfCharsToKeep - 2] == token[1]) &&
                        ((recent[_numberOfCharsToKeep - 1] == ' ') ||
                        (recent[_numberOfCharsToKeep - 1] == 0x0d) ||
                        (recent[_numberOfCharsToKeep - 1] == 0x0a)) &&
                        ((recent[_numberOfCharsToKeep - 4] == ' ') ||
                        (recent[_numberOfCharsToKeep - 4] == 0x0d) ||
                        (recent[_numberOfCharsToKeep - 4] == 0x0a))
                        )
                    {
                        return true;
                    }
                }
                else
                {
                    return false;
                }
    
            }
            return false;
        }
        #endregion
    }
    

    Thank you to all those who provided help and snippets that allowed me to finally pull a working solution together