I have read the paper of NEAT Algorithm here. And want to try it out myself. But I still have one question about the algorithm and that is:
How to determinate the direction of a connection during the add connection mutation operation if 2 hidden nodes are selected?
Let's say I have a network where node 1,2 are input nodes, 3,4,5 are hidden nodes and 6 is the output.
Node 3 and 4 are directly connected with the input 1,2 where 5 currently connected only node 4 and direct it to output node 6.
It is simple to determinate the in and out node of a new connection if one of the nodes is input or output. But if in the case above, the code decided to add a connection between node 3 and node 5. which node should I use as in-node and which as out? or it doesn't matter?
It does not matter. Normally.
The typical solution would be to select first a node from (with some restrictions like: avoid output nodes) then a node to (with some different restrictions, like: avoid input nodes).
Sometimes you will want extra checks. For example, you may want to work with feed-forward-only networks, where connection loops are not allowed. In this case you will need to check that the selected from-to pair of nodes does not create a loop.
However, I strongly recommend that you use and modify an existing implementation of NEAT (check Colin Green, who takes care of some very high quality NEAR repositories). You will find that NEAT is surprisingly complex to implement if you want all the tricks and good practices.