I am in the process of designing a chess game which will feature an AI opponent to play against. The user will also have the ability to create custom chess pieces for their starting lineup by combining behaviours of current pieces they have unlocked (for example a Heroine; a hybrid of Queen and Horse) plus completely new pieces, and will also feature new board shapes different to the original 8x8.
The question then is how to design the AI. I decided I will start simple and try the minimax algorithm with alpha-beta pruning, but the problem arises in determining how to evaluate the position.
Evaluating a position with new pieces and new board shapes that can't be pre-determined at compile-time is something I have not seen people mention much, so I am stuck on how to even approach it. A simple point-wise sum of each piece value doesn't really work either because you can't score how the custom pieces. I am aware that Alpha Zero and other chess engines use a large neural network for evaluating their position.
Edit: I discussed possible architectures with a friend of mine, and we considered a theoretical architecture where the behaviour of each custome piece is encoded in a binary string, which is then combined with the board layout (which would also be a binary string) to create the first input layer of a MLP for the evaluation function, which would have a single output neuron, which outputs a number which is the evaluation of the position (positive for white and negative for black as usual).
The evaluation function is then put in a minimax to create an AI bot, and using this we make it play against itself, and to train the evaluation function we get the result of the game (1 for white wins and 0 for black) and use that to create a cost function - if white wins, then the eval function for white should have been evaluating the position as positive most of the time.
Of course, this doesn't protect against sudden mistakes the opponent may make, which completely change the tide of the game, even though they may have been winning most of the time. To rectify this, we play the same model against itself with the same game multiple times, butting adding random noise to the moves each time.
I have no clue if this would work, and I'm hoping some AI experts could point out any flaws ro alternatives.
General Game Playing (GGP) is an AI field where the rules are not known in advance of game play. So if the player unlocked new custom piece moves, you could encode those into Game Description Language (GDL) and have it play. There are many open source players available, like Cadiaplayer, Sancho, and some others.