javaswiftvideo-streamingclient-serveruiimagejpegrepresentation

How to send an image(UIImage) from a SWIFT ios client program to a JAVA server program?

My Question:

Expected Result from my program - The ios swift app should efficiently connect to my java server and send images of video frames live to it. The images should then be converted to bufferedImage on the Java server machine and played as video!

Regarding previously asked questions - I found only one similar question but the answer was not very informative.

Details

Entire Swift Program's viewController.swift for video capture is derived from this github project ( https://github.com/FlexMonkey/LiveCameraFiltering)

Edit - I have figured out the problem and posted it as an answer but this is still just a workaround because the server video feed still hangs a lot and I had to reduce the quality of the image byte data being sent by the swift client. There definitely is a better way to do things and I request people to share their knowledge.

Solution

  • So, I was not able to find a complete and absolutely perfect solution to the above mentioned problem but for the benefit of any other swift beginner who might stumble across a similar cross-language client-server program based problem , here are my two cents:

    The first and foremost error in the above mentioned code is this line:

     let data = UIImageJPEGRepresentation(image, 1.0)

    1) Here, I was encoding the UIImage to the highest possible quality by supplying the compression factor as 1. This , as I later examined was leading to the creation of byte array with counts exceeding 100000 and thus it was pretty difficult to easily and quickly send such a large data through the TCPClient socket.

    2) Even if such a large array was efficiently sent by the TCPClient socket.It would be difficult for the Java DataInputStream at the server side to read the complete data at once. It was probably reading only small chunks of data at a time and therefore the image generated at the java server end was partial and fuzzy.

    3) This line was another problem:

       count = in.available();
   if(count>0) System.out.println("LENGTH="+count);
   byte[] arr=new byte[count];
   System.out.println("byte="+arr);
   in.read(arr);

    The in.available() method does probably not return the complete length of data which was sent by the client side. This lead to reading incomplete byte data and thus incomplete images.

    The solution/workaround(kind of)

    Modified Swift Code(ios client)

        func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!)
{


    var cameraImage: CIImage

    var image: UIImage ;
    let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
    cameraImage = CIImage(cvPixelBuffer: pixelBuffer!)
    let context:CIContext = CIContext.init(options: nil)
    let cgImage:CGImage = context.createCGImage(cameraImage, from: cameraImage.extent)!
    image = UIImage(cgImage: cgImage)
    DispatchQueue.main.async
        {



            self.imageView.image = image //live video captured from camera streamed to the device's own UIImageView


    }





    let thumbnail = resizeImage(image: image, targetSize: CGSize.init(width: 400, height: 400)) // snapshot image from camera resized 

    let data = UIImageJPEGRepresentation(thumbnail,0.000005) //the snapshot image converted into byte data
    let base64String = data!.base64EncodedString(options: Data.Base64EncodingOptions(rawValue: 0))  
    // byte image data is encoded to a base64String

    var encodeImg=base64String.addingPercentEncoding(withAllowedCharacters:  .urlQueryAllowed  )


    encodeImg = encodeImg! + String("%") // termination char % is added at the end
    var sendData = String("0%")
    if(live)
     {
    sendData = encodeImg!
     }
    client?.send(string: sendData!) //sent as a String using TCPClient socket



}

    Modified Java Server Side run() method of the MobileServer Thread class

            public void run()
        {
          try{

               boolean access_granted=false;
               while(!stop)
               {

                 char chr=(char)in.read();


                 if(chr!='%')  // read and append char by char from the InputStreamReader "in" until it encounters a '%'
                 str+=Character.toString(chr);

                 else terminate=true;

                 if(terminate)

                 {


                  if(entry)
                  {
                    int a=str.indexOf('&');
                    int b=str.indexOf('#');
                    String username=str.substring(0,a);
                    String password=str.substring((a+1),b);
                    String ip=str.substring((b+1),str.length());
                    System.out.println("IP ADDRESS: \""+ ip+"\"");
                    String usernameA[]=convertToArray(username);
                    String passwordA[]=convertToArray(password);
                    String user=decrypt(usernameA,portt);
                    String pass=decrypt(passwordA,portt);


                    boolean accessGranted=false;
                    int response=dbManager.verify_clientLogin(user,pass);

                if(response==RegisterInfo.ACCESS_GRANTED) { 
                System.out.println("access granted"); 
                 accessGranted=true;
                }
                   int retInt=-1;
                   if(accessGranted) retInt=1;

                   out.write(retInt);

                   entry=false;
                   terminate=false;


                  }
                  else
                  {
                     terminate=false;



                     try {
                  // str includes the original single base64String produced by the swift client app which is converted back to a byte array                              

  imageBytes = javax.xml.bind.DatatypeConverter.parseBase64Binary(str); 
}catch(ArrayIndexOutOfBoundsException l){ exception=true; }
str="";
                      if(!exception)
                       {


         //this byte array image data is converted to a image and played on the videoPlayer, and serial images played would be visible as a video stream
                         vidPlayer.playImage(imageBytes);

                          ioexcep=false;


                        }


                        else exception=false;



                   }
             }






         }





         }catch(Exception l){ l.printStackTrace(); }















        }