Capturing the camera on android in the background with preview after processing with OpenCV

My goal is to be able actively process an image in the background, but when the app is in the foreground I'd need a preview which shows the processed image, meaning the original frame and bounding boxes etc. added by opencv.

So I thought I'd use a bounded foreground service which would manage JavaCamera2View from opencv (It is very possible that there is a simpler way to communicate with the activity but that's the one I found so yeah). Then the activity would bind to the service, and use the camera view, but it didn't work or at least it all went fine until I tried to call enableView on camera view from the bounded service, then app would crash. Here's the code I used(I know it's garbage and the main activity does like half of the initialization of opencv which it shouldn't):

CameraService:

class CameraService: Service() {
    private val binder = CameraBinder(this)
    lateinit var openCvCameraView: CameraBridgeViewBase

    inner class CameraBinder (
        private val context: Context,
    ) : Binder() {
        val service = this@CameraService
    }

    override fun onBind(intent: Intent?): IBinder {
        return binder
    }

    private fun startCamera(): CameraBridgeViewBase {
        val view = JavaCamera2View(this, -1)
        view.visibility = SurfaceView.VISIBLE
        view.setCameraPermissionGranted()
        view.setCvCameraViewListener(DetectionRepository())
        return view
    }

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        when (intent?.action) {
            Actions.START.toString() -> startService()
            Actions.STOP.toString() -> stopSelf()
        }

        return super.onStartCommand(intent, flags, startId)
    }

    private fun startService() {
        val notification = NotificationCompat.Builder(
            this,
            TestApp.NotificationChannels.PROCESSING_CAMERA.toString()
        )
            .setSmallIcon(R.drawable.ic_launcher_foreground)
            .setContentTitle("Processing the camera feed")
            .setContentText("some text")
            .build()

        openCvCameraView = startCamera()
        startForeground(1, notification)
    }

    enum class Actions {
        START,
        STOP,
    }
}

Main activity:

class MainActivity : ComponentActivity() {
    private lateinit var cameraService: CameraService
    private var cameraServiceBound: Boolean = false

    private val connection = object : ServiceConnection {
        override fun onServiceConnected(className: ComponentName, service: IBinder) {
            val binder = service as CameraService.CameraBinder
            cameraService = binder.service
            cameraServiceBound = true
        }

        override fun onServiceDisconnected(arg0: ComponentName) {
            cameraServiceBound = false
        }
    }

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        enableEdgeToEdge()

        if (OpenCVLoader.initLocal()) {
            Log.i("LOADED", "OpenCV loaded successfully")
        } else {
            Log.e("LOADED", "OpenCV initialization failed!")
            (Toast.makeText(this, "OpenCV initialization failed!", Toast.LENGTH_LONG))
                .show()
            return
        }

        Intent(this, CameraService::class.java).also {
            it.action = CameraService.Actions.START.toString()
            startForegroundService(it)
            bindService(it, connection, Context.BIND_AUTO_CREATE)
        }

        if (!hasRequiredPermissions()) {
            ActivityCompat.requestPermissions(this, REQUIRED_PERMISSIONS, 0)
        }

        window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON)
        cameraService.openCvCameraView.setCameraPermissionGranted()
        cameraService.openCvCameraView.enableView() // crashes because of this


        setContent {
            AppTheme {
                Column {
                    if (cameraServiceBound) {
                        AndroidView(factory = { cameraService.openCvCameraView })
                    }
                }
            }
        }
    }

    override fun onStart() {
        Intent(this, CameraService::class.java).also {
            it.action = CameraService.Actions.START.toString()
            bindService(it, connection, Context.BIND_AUTO_CREATE)
        }
        super.onStart()
    }

    private fun hasRequiredPermissions(): Boolean {
        return REQUIRED_PERMISSIONS.all {
            ContextCompat.checkSelfPermission(
                applicationContext,
                it
            ) == PackageManager.PERMISSION_GRANTED
        }
    }

    companion object {
        val REQUIRED_PERMISSIONS = assemblePermissions()

        private fun assemblePermissions(): Array<String> {
            val basePerms = mutableListOf(
                Manifest.permission.CAMERA,
                Manifest.permission.FOREGROUND_SERVICE,
            )

            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.UPSIDE_DOWN_CAKE) {
                basePerms.add(Manifest.permission.FOREGROUND_SERVICE_CAMERA)
            }

            if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) {
                basePerms.add(Manifest.permission.POST_NOTIFICATIONS)
            }

            return basePerms.toTypedArray()
        }
    }
}

One more thing I've been considering is using CameraX and just converting each frame to a Mat using something similar to the implementation of JavaCamera2Frame in opencv. But I don't know how I would then change the output of the PreviewView to use the processed frames also I'd need to somehow convert back to the format CameraX is using.

I've looked around and most I've seen some people asking how to use opencv in the background but it I haven't seen anyone making a preview out of frames processed by opencv which is what I'm struggling to achieve.

I'm new to android development and opencv in general, I don't really know how to achieve my goal in this situation.

Is there a good solution to achieve this? Is this even possible?

Update

I was experiencing crashes with JavaCamera2View because I had a race condition. I tried to use the bounded cameraService when the connection wasn't established yet. I fixed this problem by moving the ui code into onServiceConnected as a temporary fix, it looks something like this:

override fun onServiceConnected(className: ComponentName, service: IBinder) {
            val binder = service as CameraService.CameraBinder
            cameraService = binder.service
            cameraServiceBound = true
            setContent {
              AppTheme {
                Column {
                        AndroidView(factory = {  cameraService.openCvCameraView })
                }
            }
        }
}

However this still didn't solve my issue as after this fix when the app was in the background the opencv would just stop processing frames, as if I had JavaCamera2View inside of the Activity. So this path led me to nowhere.

So I solved my own problem. If the app is in the foreground I need to show the preview, which I can most easily do with opencv, but then in the background I just want to process the frames which opencv can't do so I need to use CameraX. So why not do both.

I used a singleton usecase to keep track of whether the app is in foreground or background:

object AppStateUseCase {
    enum class AppVisibilityState {
        BACKGROUND,
        FOREGROUND,
    }

    private var _appVisibilityState = MutableStateFlow(AppVisibilityState.FOREGROUND)
    val appVisibilityState = _appVisibilityState.asStateFlow()

    fun updateAppVisibilityState(newState: AppVisibilityState) {
        _appVisibilityState.value = newState
    }
}

So I spawn the camera preview with this code which gets called from the main activity:

fun cameraPreview(
    context: Context,
    lifecycleScope: CoroutineScope
): JavaCamera2View {
    val view = JavaCamera2View(context, -1)
    lifecycleScope.launch { visibilityStateListener(view) }
    view.visibility = SurfaceView.VISIBLE
    if (!hasRequiredPermissions(context, REQUIRED_PERMISSIONS)) {
        throw IllegalStateException("startPreview camera called without necessary permissions")
    }
    view.setCameraPermissionGranted()
    view.setCvCameraViewListener(OpencvDetector())
    return view
}

private suspend fun visibilityStateListener(view: JavaCamera2View) {
    AppStateUseCase.appVisibilityState.collect {
        when (it) {
            AppStateUseCase.AppVisibilityState.BACKGROUND -> view.disableView()
            AppStateUseCase.AppVisibilityState.FOREGROUND -> view.enableView()
        }
    }
}

Then in the main activity the onStart and onStop update the state:

override fun onStart() {
    AppStateUseCase.updateAppVisibilityState(
        AppStateUseCase.AppVisibilityState.FOREGROUND
    )
    super.onStart()
}

override fun onStop() {
    AppStateUseCase.updateAppVisibilityState(
        AppStateUseCase.AppVisibilityState.BACKGROUND
    )
    super.onStop()
}

Now I don't bind to the service, I just use LifecycleService and manage CameraX with the IMAGE_ANALYSIS use case. The notable part is in analyzeInBackground which skips analysis when the app is in foreground because the main activity handles that. It might be worth to consider that by default opencv and camerax will give frames in different resolutions. CameraX defintily has a way to do that so it's worth checking out:

class CameraService: LifecycleService() {
    private lateinit var cameraProviderFuture : ListenableFuture<ProcessCameraProvider>

    private fun analyzeInBackground(imageProxy: ImageProxy) {
        if (AppStateUseCase.appVisibilityState.value
            == AppStateUseCase.AppVisibilityState.FOREGROUND) {
            imageProxy.close()
            return
        }
        
        Detector.analyze(imageProxy)
        imageProxy.close()
    }

    override fun onStartCommand(intent: Intent?, flags: Int, startId: Int): Int {
        when (intent?.action) {
            Actions.START.toString() -> startService()
            Actions.STOP.toString() -> stopSelf()
        }

        return super.onStartCommand(intent, flags, startId)
    }

    private fun startService() {
        val notification = NotificationCompat.Builder(
            this,
            EmpressApp.NotificationChannels.PROCESSING_CAMERA.toString()
        )
            .setSmallIcon(R.drawable.ic_launcher_foreground)
            .setContentTitle("Processing the camera feed")
            .setContentText("")
            .build()

        cameraProviderFuture = ProcessCameraProvider.getInstance(this)
        cameraProviderFuture.addListener(Runnable {
            val cameraProvider = cameraProviderFuture.get()
            bindAnalyzer(cameraProvider)
        }, ContextCompat.getMainExecutor(this))

        startForeground(1, notification)
    }

    enum class Actions {
        START,
        STOP,
    }

    private fun bindAnalyzer(cameraProvider : ProcessCameraProvider) {
        val cameraSelector : CameraSelector = CameraSelector.Builder()
            .requireLensFacing(CameraSelector.LENS_FACING_BACK)
            .build()

        val imageAnalysis = ImageAnalysis.Builder()
            .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
            .build()

        imageAnalysis.setAnalyzer(
            ContextCompat.getMainExecutor(this)
        ) { imageProxy ->
            analyzeInBackground(imageProxy)
        }

        cameraProvider.bindToLifecycle(this , cameraSelector, imageAnalysis)
    }
}

Now the last important thing left is to convert from ImageProxy to Mat when analzying in the background:

         private fun imageProxyToMat(imageProxy: ImageProxy): Mat {
            if (imageProxy.format != PixelFormat.RGBA_8888) {
                throw IllegalArgumentException("image proxy has the wrong format")
            }

            val mat = Mat(imageProxy.height, imageProxy.width, CvType.CV_8UC4)
            Utils.bitmapToMat(imageProxy.toBitmap(), mat)
            Imgproc.cvtColor(mat, mat, Imgproc.COLOR_RGBA2BGR)
            Core.rotate(mat, mat, Core.ROTATE_90_CLOCKWISE)
            return mat
        }

And here is the Detector and OpencvDetector class which were used through out the code for the sake of completeness:

class OpencvDetector: CvCameraViewListener2 {
    override fun onCameraViewStopped() {}

    override fun onCameraViewStarted(width: Int, height: Int) {}

    override fun onCameraFrame(inputFrame: CameraBridgeViewBase.CvCameraViewFrame): Mat {
        return Detector.analyze(inputFrame.rgba())
    }
}

class Detector {
    companion object {
        fun analyze(input: Mat): Mat {
            Log.i("foo", "analyzing")
            return input
        }

        fun analyze(input: ImageProxy): Mat {
            return analyze(imageProxyToMat(input))
        }
    }
}

I may have explained it a little too much in depth than needed but I was excited that I found the solution and also wanted to share if anyone finds it useful.