Why does my Brain.js neural network get stuck in the middle?

I'm trying to use Brain.js for text generation purposes.

See my WIP example at: https://codepen.io/tomsoderlund/pen/WEPqzE (see also console output).

I basically:

1. Generate an array of all the words: wordsInOrder
2. Create a dictionaryWords array with sorted unique words.
3. I create my training set from wordsInOrder like this: { input: [0.0326], output: [0.9565] }, where input is the current word's dictionary index (normalized), and output is the following word's dictionary index.

Then I generate new words by:

1. Picking a random word from dictionary.
2. Then running the brainJsNetwork.run([wordValue]) function to generate the following word.
3. Repeat from step 1 over again.

However, it seems to get stuck on words in the middle of the dictionary, with wordValue’s around 0.5:

Any clues what the problem is?

I suspect this is due to your training set. This is supposed to map a certain input to an output that's correct. Like in brainjs color contrast example:

net.train([{input: { r: 0.03, g: 0.7, b: 0.5 }, output: { black: 1 }},
       {input: { r: 0.16, g: 0.09, b: 0.2 }, output: { white: 1 }},
       {input: { r: 0.5, g: 0.5, b: 1.0 }, output: { white: 1 }}]);

For a list of inputs, it gives the correct categorization. Then afterwards if you run the trained network, it gives the likelihood of the categories for the input you give it:

var output = net.run({ r: 1, g: 0.4, b: 0 });  // { white: 0.99, black: 0.002 }

You create the training set from wordsInOrder. That means there are some words which occur multiple times in your training set. A word like 'made' is in your training set multiple times, with different outputs:

made -> If (or the respective wordindex values, normalized to be between 0-1)
made -> It's
made -> outside
made -> in

The trained network will try to compensate for the different possible outcomes and will average the likelihood it outputs. If you then take that output to lookup the word in the dictionaryWords array, you are morelikely to end up with words thate are in the middle of the array (like 'not' and 'necessarily')

You need to take into account that the neural network will return a likelihood of the input belonging to a certain category. so if you want to use it for predicting the next word, you have to encode the training data differently. There are 4 valid next words for 'made' so you would have to encode them like ...

{input: { (wordindex of 'made' }, output: { if: 1, its: 1, outside:1, in:1 }}

Of course this means your output will have a likelihood score for all the 92 unique words in the dictionaryWords array. I am not sure if this simple neural network can be used with an output with 92 dimensions.

Have you looked at markov chains for generating text? It makes it more easy to model which transitions (from one word to the next word) are more likely than others.

Here is an explanation and a javascript implementation.