In a traditional Web application, the backend service should never trust the payload sent from the user frontend. Though the frontend pages use carefully designed UI to force users to send data in a valid/expected format, they can't prevent advanced users from directly calling API endpoints and sending invalid/malicious data. So backend service should always validate the data.
In Telegram bot, my bot send users a message with inline keyboard with a few buttons. Clicking a button will send a callback query to my bot server. I'm wondering if I should validate the carried data of the callback query? Can users construct a custom callback query that can carry any data they want?
For example, if I let users do a multiple choices problem and give them 4 buttons. The callback query data of each button is A, B, C and D respectively. Can my bot server assume it'll always receive one of A, B, C and D? Or are users able to send a E (or any other strings they want) to my bot server?
Generally speaking, by tapping the buttons users will only be able to send the callback data you've specified in your bot's code - in the example case A, B, C, and D. Users will not be able to change the data from Telegram app.
Well, on a wider perspective, a "dedicated geeky exploiter user" can send any string to your bot as callback query data using the MTProto API (messages.getBotCallbackAnswer)
Anyways, your bot will simply receive a CallbackQuery update. It's up to your code whether to process or skip the query. I'd say it's best to match your query data with RegEx or other methods to match the query data with expected patterns. Just skip the rest with answering the callback queries.