Android Gstreamer local display video lags after starting rtmpsink

I am working on Android app

1. where I have incoming video source in rtsp stream( h265/hevc )

2. source rtsp server have problem only allows one connection

3. I want to play rtsp stream and also stream it to rtmp server (antmedia).

   Simple view is as follows

   [rtsp server] ---> [split] ---> [display]
                              L--> [rtmp stream]
   

Current progress:

I used gstreamer and its tutorial 3 code and modify its pipeline as follows

data->pipeline = gst_parse_launch ("rtspsrc location=rtsp://<local_server_ip>:8554/main.265 latency=50 udp-reconnect=1 timeout=5000000 ! tee name=t t. ! queue ! parsebin sink-caps=\"video/x-h265,width=1280,height=720,stream-format=hvc1\" ! decodebin3 ! autovideosink  sync=false t. ! valve drop=true name=liveStrmValve ! queue max-size-buffers=0 silent=true ! parsebin sink-caps=\"video/x-h265,width=1280,height=720,stream-format=hvc1\" ! avdec_h265 ! videoscale ! videorate ! video/x-raw,width=1280,height=720,framerate=25/1 ! x264enc speed-preset=ultrafast tune=zerolatency ! h264parse ! queue ! flvmux streamable=true skip-backwards-streams=true ! rtmpsink location=\"rtmp://<server_ip>/LiveApp/<key> live=true\" sync=false async=false", &error);

Where streaming is controlled by valve name=liveStrmValve, property drop.

Problem I am facing :

1. If the valve is droping(drop=false). local video playing is smooth and realtime.

2. If I start valve (from java ui button to jni funtion). valve start and I get stream at Antmedia server stream is good having 8sec delay.

3. But while streaming local display video start laging. and lag get increases as there is lot of movment in frames.

4. if I stop live streaming to rtmp server( valve drop=true). the lagged video start playing fast and match the real time feed and work smoothly.

Android :

armeabi-v7a
minSdk 25
targetSdk 33
ndkVersion '21.3.6528147'

Gstreamer : gstreamer-1.0-android-universal-1.22.6

My implementation:

• I am getting rtsp stream encoded h264/hevc. I have used hardwared decoder by changing rank before creating pipe. this is only to display video feed on autovideosink

  tee t. ->parsebin -> decodebin3 -> autovideosink

• for stream video to server, I have tee split stream. then decode

  t. -> h265decode -> video rate+scale -> encode h264 -> parse -> flvmix -> rtmpsink

Some errors and warning by gstreamer:

After pipe start
w : ../gst-libs/gst/video/video-info.c:762:gst_video_info_to_caps invalid matrix 0 for RGB format, using RGB

w : ../libs/gst/base/gstaggregator.c:2146:gst_aggregator_query_latency_unlocked:<flvmux0> Latency query failed

w : amcvideodec-omxqcomvideodecoderhevc0: Too old frames, bug in decoder -- please file a bug

w : ../gst-libs/gst/video/gstvideodecoder.c:3154:gst_video_decoder_prepare_finish_frame:<amcvideodec-omxqcomvideodecoderhevc0> decreasing timestamp (0:00:00.056677084 < 0:00:03.382351466)

After stream start
e : :0: Could not find ref with POC 54
w : ../gst-libs/gst/video/video-info.c:201:validate_colorimetry Need to specify a color matrix when using YUV format (I420)
w : ../gst-libs/gst/video/video-info.c:516:gst_video_info_from_caps invalid colorimetry, using default

Need help :

1. what could be the cause of local video lagging if stream is live

2. is it possible to use separate clock for each branch from tee

3. I have tried to send video on udpsink and receive it on other branch udpsrc then stream to rtmp server but there is glitches in video at antmedia server.

4. is there any other library I can use here. single connection, split stream display and stream to rtmp server at same time.

I come up with following solution
Pipe 1 (main pipe display video)

 String RtspH264Url= "rtspsrc location="
         + CameraSettingParameter.getRtspURL() + // camera RTSP Url
         " latency=100 udp-reconnect=1 " +
         "timeout=5000000 wait-for-keyframe=1 name=rtspSrc ! " +
         "rtpjitterbuffer drop-on-latency=true " +
         "latency=100 ! parsebin " +
         "sink-caps=\"video/x-h264,width=1280,height=720,stream-format=byte-stream\" ! " +
         "tee name=t t. ! queue ! decodebin3 ! autovideosink name=video-sink sync=false " +
         "t. ! queue ! tcpserversink port=5002 sync=true ";

Now I get video on tpc port 5002
Pipe 2 (stream video to server)

String streamingH264Url =  "tcpclientsrc  port=5002 ! " +
    "capsfilter caps=\"video/x-h264, " +
    "media=(string)video," +
    "clock-rate=(int)90000, " +
    "width=(int)1280, " +
    "height=(int)720," +
    "framerate=30/1, " +
    "payload=(int)96, " +
    "encoding-name=(string)H264, " +
    "stream-format=(string)byte-stream\" ! " +
    "queue " +
    "max-size-buffers=600 " +
    "max-size-time=90000000000 ! " +
    "decodebin ! " +
    "videoconvert ! " +
    "videoscale ! " +
    "videorate ! " +
    "video/x-raw," +
    "width="+ videoWidth +","+//1280
    "height="+ videoHeight+","+ //720
    "framerate="+String.valueOf(CameraSettingParameter.getFrameRate())+"/1 ! " +
    "x264enc speed-preset=ultrafast " +
    "tune=zerolatency ! " +
    "h264parse ! " +
    "flvmux streamable=true ! " +
    "rtmpsink location=\""+ streamingServerUrl +" live=true\" sync=false";

Now both pipe line work independently. Without making lags in clocks.