How to make Gstreamer RTSP -> UDP -> RTSP low latency pipeline?

I need to forward an rtsp stream from an axis camera through an udp port. I have this pipeline on the send side:

gst-launch-1.0 rtspsrc location='rtsp://XXXX:XXXXX@192.168.2.68/axis-media/media.amp'  latency=0 ! \
rtph264depay ! \
h264parse ! \
rtph264pay config-interval=1 ! \
udpsink host=127.0.0.1 port=5005 sync=false

then on the receiving side, if I run this pipeline:

gst-launch-1.0 udpsrc port=5005 buffer-size=200000 caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" ! \
rtpjitterbuffer do-lost=true latency=450 ! \
rtph264depay ! \
h264parse ! queue2 use-buffering=true max-size-buffers=1000 max-size-bytes=0 max-size-time=0 ! \
avdec_h264 max-threads=8 ! \
videoconvert ! \
autovideosink sync=false

I can see the stream perfectly and it adds basically 0 latency. Wonderful.

I then forward it to my mediamtx rtsp server with this pipeline (basically the above with rtspclientsink instead of autovideosink) :

gst-launch-1.0 udpsrc port=5005 buffer-size=200000 caps="application/x-rtp, media=(string)video, encoding-name=(string)H264, payload=(int)96" ! \
rtpjitterbuffer do-lost=true latency=400 ! \
rtph264depay ! \
h264parse ! queue2 use-buffering=true max-size-buffers=1000 max-size-bytes=0 max-size-time=0 ! \
avdec_h264 max-threads=8 ! \
videoconvert ! \
rtspclientsink protocols=tcp location=rtsp://127.0.0.1:8554/media1.amp latency=0

but then reading it with:

gst-launch-1.0 playbin uri=rtsp://127.0.0.1:8554/media1.amp latency=0 buffer-size=50000000 buffer-duration=5000000000

suddenly I have 2 seconds latency. Also, trying to read the uri with ffplay will yield green lines/broken stream and VLC directly refuses to read it at all.

What am I doing wrong? is there a better way to do this?

Other data points:

If I directly sink the original stream to the destination rtspclientsink, playbin will read it nicely, no latency added. But I need the udp bridge.
I can sink directly the rtsp stream to udp, but then there is no way for me to read it back from udpsrc; sinking udpsrc to fakesink shows timestamp going, but no data will actually go through anything else.
rtspsrc will also refuse to read the rtsp stream from mediamtx (rtsp://127.0.0.1:8554/media1.amp), no matter what pipeline I try.

Solution

So, in the end I found how to do this.

RTSP -> UDP

gst-launch-1.0 rtspsrc location='rtsp://user:pwd@ip:port/axis-media/media.amp'  latency=0 ! \
rtph264depay ! \
h264parse ! \
rtph264pay config-interval=1 ! \
udpsink host=127.0.0.1 port=5005 sync=false

UDP -> RTSP

gst-launch-1.0 udpsrc port=5006 caps="application/x-rtp,media=(string)video,encoding-name=(string)H264,payload=(int)96" ! rtpjitterbuffer latency=50 ! \
    queue ! rtph264depay ! rtspclientsink protocols=tcp location=rtsp://ip:port/media.amp

of course set up mediamtx to receive the media.amp stream, and configure it to receive on a port; 554 can't be opened without sudo, at least on Linux. I've seen 8554 as sort of a standard-ish fallback.

Good luck to whoever needs this.