androidxamarinimageviewtext-recognitionvision-api

How to get position of text in an image using Mobile Vision API?


How to get the position of text on the screen in an image using Mobile Vision API, and how to draw a rectangle around them?

Example:

enter image description here


Solution

  • How to do it

    Put an ImageView in the layout

    <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:orientation="vertical">
    <ImageView
        android:layout_width="match_parent"
        android:layout_height="250.0dp"
        android:minWidth="25px"
        android:minHeight="25px"
        android:id="@+id/imageView1" />
    </LinearLayout>
    

    Instantiate ImageView in onCreate Method

    ImageView imgView;
        protected override void OnCreate(Bundle savedInstanceState)
        {
            base.OnCreate(savedInstanceState);
            SetContentView(Resource.Layout.activity_main);
    
            imgView = FindViewById<ImageView>(Resource.Id.imageView1);
            OCR();
        }
    

    THIS IS IMPORTANT CODE FOR GETTING TEXT AND DRAWING RECTANGLE ON IT

    Please read Comments on top of code

    public void OCR()
        {
            //Convert The Image To Bitmap
            Bitmap bitmap = BitmapFactory.DecodeResource(ApplicationContext.Resources, Resource.Mipmap.lineindent);
    
            TextRecognizer textRecognizer = new TextRecognizer.Builder(ApplicationContext).Build();
    
            if (!textRecognizer.IsOperational)
            {
    
                Log.Error("Main Activity", "Dependencies not available");
    
                // Check android for low storage so dependencies can be loaded, DEPRICATED CHANGE LATER
                IntentFilter intentLowStorage = new IntentFilter(Intent.ActionDeviceStorageLow);
    
                bool hasLowStorage = RegisterReceiver(null, intentLowStorage) != null;
    
                if (hasLowStorage)
                {
    
                    Toast.MakeText(this, "Low Memory On Disk", ToastLength.Long);
                    Log.Error("Main Activity", "Low Memory On Disk");
                }
    
            }
            else
            {
                Frame frame = new Frame.Builder().SetBitmap(bitmap).Build();
    
    
                SparseArray items = textRecognizer.Detect(frame);
                List<TextBlock> blocks = new List<TextBlock>();
    
                TextBlock myItem = null;
                for (int i = 0; i < items.Size(); ++i)
                {
                    myItem = (TextBlock)items.ValueAt(i);
    
                    //Add All TextBlocks to the `blocks` List
                    blocks.Add(myItem);
    
                }
                //END OF DETECTING TEXT
    
                //The Color of the Rectangle to Draw on top of Text
                Paint rectPaint = new Paint();
                rectPaint.Color = Color.White;
                rectPaint.SetStyle(Paint.Style.Stroke);
                rectPaint.StrokeWidth = (4.0f);
    
                //Create the Canvas object,
                //Which ever way you do image that is ScreenShot for example, you 
                //need the views Height and Width to draw recatngles 
                //because the API detects the position of Text on the View
                //So Dimesnions are important for Draw method to draw at that Text 
                //Location
                Bitmap tempBitmap = Bitmap.CreateBitmap(bitmap.Width, bitmap.Height, Bitmap.Config.Rgb565);
                Canvas canvas = new Canvas(tempBitmap);
                canvas.DrawBitmap(bitmap, 0, 0, null);
    
                //Loop through each `Block`
                foreach (TextBlock textBlock in blocks)
                {
                    IList<IText> textLines = textBlock.Components; 
    
                    //loop Through each `Line`
                    foreach (IText currentLine in textLines)
                    {
                        IList<IText>  words = currentLine.Components;
    
                        //Loop through each `Word`
                        foreach (IText currentword in words)
                        {
                            //Get the Rectangle/boundingBox of the word
                            RectF rect = new RectF(currentword.BoundingBox);
                            rectPaint.Color = Color.Black;
    
                            //Finally Draw Rectangle/boundingBox around word
                            canvas.DrawRect(rect, rectPaint);
    
                            //Set image to the `View`
                            imgView.SetImageDrawable(new BitmapDrawable(Resources, tempBitmap));
    
    
                        }
    
                    }
                }
    
            }
        }
    

    RESULT

    enter image description here

    If you want the Rectangle on Lines then remove the code from words loops and put it in the Lines loop, same goes to the blocks

    enter image description here