Could you help me understand if NextJS can be used with excel taskpane when app is being developed using ReactJS, if so could you guide me with any example in the web?
Yes. You can use NextJS. An Office Add-in is essentially just a web app. So, with few exceptions, you can do anything in an add-in that you can do in a web app.