Does anybody know weather VNC (RFB) supports virtual channels and add-ins to them like it is in the RDP (Microsoft Terminal Services)? I just want to transfer my own data across a VNC connection...
VNC/RFB does not have virtual channels unfortunately.
Here is the best reference I've found to the RFB protocol: http://tigervnc.org/cgi-bin/rfbproto
Without knowing more about what you are trying to send and which direction(s), there are a few of options that come to mind:
The tight encoding has file-transfer support. There is a poorly formatted specification for the full tight encoding here: http://vnc-tight.svn.sourceforge.net/viewvc/vnc-tight/trunk/doc/rfbtight.odt?revision=3619
If you have control of both client and server, then you could define a custom encoding that allows you to send your data. The client would advertise that it supports the encoding and if the server supports it then it will start using it.
You could use the clipboard messages (ClientCutText and ServerCutText) and if you need to send binary data that create a custom encoding the data as ISO 8859-1 (Latin-1). The downside is that if the server doesn't support it and the client sends the data it will get pasted to the server.
If you just need to send from the server to the client, then you could use a framebufferUpdate message that sends data outside the current viewport (i.e. 123 pixels beyond the right side of the viewport). Clients without support may not handle this well though.
Another option if you just need to send from the server to the client, is that you could send a framebufferUpdate within the viewport with a special marker and then immediately send a framebufferUpdate (even in the same packet) with the real visible data to replace it. This would work with existing clients (a bit more overhead). Clients might see brief flicker though.