I have a Microsoft Excel add-in. When launching, it opens a sidebar. The sidebar hosts a website which is the core of the add-in. The website can interact with the contents of the workbook. In one webpage, users can sign in. And there are two third-party authentication buttons: Microsoft and Google. Clicking on one button opens a Office.context.ui.displayDialogAsync where users can enter their Microsoft/Google credentials, if succeeded, the dialog will be closed automatically.
I started to code a Google Sheets add-on for similar usage. When launching, it opens a modeless dialog (rather than a sidebar that most add-ons launch) by SpreadsheetApp.getUi().showModelessDialog. The dialog hosts the website which is the core of the add-on. The website can interact with the contents of the workbook.
Now, I'm thinking of how to enable the 2 third-party authentication buttons for Microsoft and Google. There is no Office.js any more. Is opening another dialog on top of the modeless dialog a conventional way of hosting third-party authentication? Should I use showModalDialog?
No. You shouldn't use dialogs or sidebars for authentication. You should open a separate webpage using window.open or similar methods. This is because all dialogs and sidebars are sandboxed in a iframe. Oauth will be disabled inside iframes for security.