What is the point of having this rectangular thingy like https://www.microsemi.com/images/soc/partners/solution/ip/Trace_small.jpg? How comes that gdbserver is able to debug over the Ethernet without any additional H/W and this TRACE32 behemoth S/W itself cannot decode/encode signals coming out of and to JTAG port? Isn't JTAG a port itself? Doesn't it send signals? Why cannot this piece of S/W interpret them? Why is this thingy needed (which BTW once works, once doesn't and in general is black magic). Is there a reason for existing of a specific device between JTAG and USB port (having in mind that TRACE32 installations has 800 MB...)
There are probably certain aspects to consider:
"Run mode debugging" means "Debug an application on a system running an operating system". That is what is also happening when you debug an application on your Windows/Linux/Mac machine. When you hit a breakpoint in your application the CPU is still running. It is only the debugged application which is stopped.
So if your embedded system is running an operating system your GDB might be able to connect via Ethernet to a gdbserver running on your target OS, which allows your to debug an application on your device.
"Stop mode debugging" means "Debugging all software on a CPU by controlling the run-state of the CPU". So if you hit a breakpoint on a CPU with stop mode debugging, your entire CPU will stop. This allows you to debug bare metal applications or an operation system itself or an application in the context of the operation system or even a hypervisor.
For stop mode debugging you usually need a chip with a JTAG interface (or SWD or similar) and an in-circuit debugger. Basically something which allows you to control the CPU on a very low level. In former times this was done by using an in-circuit emulator (instead of JTAG) which replaced the CPU with a special bond-out chip, which allowed also to control the chip on a very low level. To make thinks more confusing some vendors call their JTAG probes also "in-circuit emulator".
For stop mode debugging you need a probe which converts the interfaces of your PC to the low level debug interface of your chip. So basically some USB to JTAG converter. Or and Ethernet to JTAG converter.
The simplest probe I can think of is simply some device which allows you to control some GPIOs (General Purpose Input Output) pins via USB. Then all the JTAG communication protocol and higher debug protocol is totally done in software. Advantage: Very flexible. Disadvantage: Very slow.
More advanced probes know how to do JTAG and thus, only the high level debug protocol has to be handled via USB, while the low level JTAG communication is done by the probe itself. These probes are often still quite slow, since USB is not so efficient when you need short latencies.
High end probes usually handle the debug protocol itself, which is individual for each CPU architecture or sometimes even for a single chip. So the host PC running the debug software sends only a high level command like "do a single step", while all the rest is handled by the probe itself. This boosts performance especially with complex multicore chips which required often a lot of JTAG communication until even a simple task completes.
Simple USB to JTAG converters are often already on the PCB of cheap evaluation boards. In theory you could also integrate such a converter directly in the chip itself, but this is usually not done by the chip manufacturers, since it would increase the costs of every single chip. In the professional sector high end debuggers are pretty common, because companies don't want to have their developers sitting in front of their PC just waiting for a slow debugger to finish an application download.
In general I assume that the faster, more flexible and more feature-rich a debugger is, the bigger and more expensive it gets. So it depends a lot on your needs.
All JTAG debuggers allow you to stop and restart your CPU, set breakpoints and read and write memory and CPU registers. That is the stop more debugging I've mentioned above.
Some debug probes allow you also to record the code execution and data accesses by the CPU while the CPU is running and without stopping it. This is called a Real Time Trace. For such a trace recording you need both a debug probe and a chip which supports this.
E.g. on ARM Cortex chips this feature is called the ETM, which is not available with Cortex-M0/M0+ chips but usually available with Cortex-A/R chips and Cortex-M3 (and bigger) chips when the chip has 100 pins and more.
Tools which support trace are usually bigger and more expensive than debug probes without trace support.
Things which have an influence on the price of a debugger with trace recorder:
The device from Lauterbach, which you are referring to, supports the tracing of Cortex-M chips with a total of 1600 MBit/s on the trace port. The official product page is here https://www.lauterbach.com/microtrace_cortexm.html
You wrote
which BTW once works, once doesn't and in general is black magic
If your tool is not working, I suggest to request support from your tool vendor. For Lauterbach visit https://www.lauterbach.com/tsupport.html
JTAG debugging itself is really not black magic: The JTAG protocol itself is an IEEE standard and the debug protocol (on the next level) is often described in the chip manufacturers manuals accessible to the public. However it is of course a lot of engineering.
