How to clone Trix default link button?

I would like to add a button to Trix editor that would insert YouTube video using custom HTML attribute. It seems to me that default link button is exactly what I need. I would need to just modify little bit what is done once someone click the link button after text is entered.

The standard link button

has Trix menu button
has input field
has input value validation
link and unlink functionality. I might want to remove the unlink button
inserts text into editor
inserted text / area is not editable

I was searching the net but did not find any example how to achieve what I want. The closest solution is https://www.kindleman.com.au/blog/advanced-actiontext-modifications-for-cms-use/ the embed, image, and link buttons are basically what I want. But it is coded in RoR. I know nothing about RoR.

Could someone help me to create my link (embed) button in Trix?

I know how to add a button. But I do not know how to open modal upon clicking the button. I know how to insert text.

var buttonHTML = '<button type="button" data-trix-attribute="red">RED</button>'

event.target.toolbarElement.
querySelector(".trix-button-group").
insertAdjacentHTML("beforeend", buttonHTML)

Solution

adding a button to the toolbar: There are two important attributes that need to match the dialog: data-trix-attribute="embed_URL" and data-trix-action="embed". [source file]
Adding embed dialog similar to the link dialog: I matched the data-trix-dialog="embed" and data-trix-dialog-attribute="embed_URL" attribute values with the button.
URL validation: I've added data-trix-method="setAttribute" to the Add button to validate URL and close dialog by calling SetAttribute from toolbar_controller.js ( to find out how it works check StimulusJS syntax )

For more details, see the comments in the code.

        /* you also can use trix-before-initialize i*/
        document.addEventListener("trix-initialize", function(e) {

            /* toolbar elements ( src/trix/config/toolbar.js )  */
            const toolbarContainer = e.target.toolbarElement.getElementsByClassName('trix-button-group--block-tools')[0];
            const trixDialogs = e.target.toolbarElement.getElementsByClassName('trix-dialogs')[0];

            /* add a buttom to toolbar  */
            const Button  = `<button id="embed-btn" type="button" class="trix-button" data-trix-attribute="embed_URL" data-trix-action="embed" title="Embed" tabindex="-1">Embed</button>`;
            toolbarContainer.insertAdjacentHTML("beforeend", Button);

            /* add a dialog HTML to toolbar */
            const embedDialogHTML = `
                <div class="trix-dialog trix-dialog--embed" data-trix-dialog="embed" data-trix-dialog-attribute="embed_URL">
                    <div class="trix-dialog__link-fields">
                    <input type="url" name="embed_URL" class="trix-input trix-input--dialog" placeholder="Enter a URL…" aria-label="URL" required="" data-trix-input="">
                    <div class="trix-button-group">
                        <input type="button" class="trix-button trix-button--dialog" value="Add" data-trix-action="x-add-embed" data-trix-method="setAttribute">
                    </div>
                    </div> 
                </div>
            `;
            trixDialogs.insertAdjacentHTML("beforeend", embedDialogHTML);


            const embedURL = trixDialogs.querySelector('.trix-dialog--embed .trix-input');
            
            document.addEventListener("trix-action-invoke", function(event) {
                    
                /* it will call from trix-button attribute data-trix-action="x-add-embed" */
                if(event.actionName === "x-add-embed"){
        
                    if(!embedURL.checkValidity())
                        return;
        
                    const youtubeIdRegex = /^(?:(?:https|http):\/\/)?(?:www\.)?(?:youtube\.com|youtu\.be).*(?<=\/|v\/|u\/|embed\/|shorts\/|watch\?v=)(?<!\/user\/)(?<id>[\w\-]{11})(?=\?|&|$)/;
                    const param = embedURL.value.match(youtubeIdRegex)?.groups?.id;
        
                    /* validate youtube link */
                    if(!param) {
                        alert('Youtube your is not valid');
                        embedURL.value = '';
                        return;
                    }
        
                    const embedHTML = `
                    <iframe width="320" height="215" src="https://www.youtube.com/embed/${param}/?controls=1"></iframe>`;
        
                    /*  The HTML inside a content attachment is not subject to Trix’s document conversion rules and will be rendered as-is. */
                    var attachment = new Trix.Attachment({ content: embedHTML })
                    e.target.editor.insertAttachment(attachment);
                }
            })

           window.addEventListener("load",() => {
                document.getElementById('result').onclick = () => {
                        console.log(document.getElementById('x').value);
                        document.getElementById('output').innerHTML = document.getElementById('x').value;
                };
           });


        });

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Trix editor Youtube feature</title>
    <link rel="stylesheet" type="text/css" href="https://unpkg.com/trix@2.0.8/dist/trix.css">
    <script type="text/javascript" src="https://unpkg.com/trix@2.0.8/dist/trix.umd.min.js"></script>
</head>
<body>
    <input id="x" type="hidden" name="content">
    <trix-editor input="x"></trix-editor>
    <br />
    <button id="result"> Preview </button>
    <b>output:</b>
    <div style="border:1px gray solid ; margin:10px; padding: 10px" id="output">

    </div>
</body>
</html>

if you are going to add more botton and dialog instead of using insertAdjacentHTML it's better to change the hole toolbar ( toolbar.js )

Trix.config.toolbar.getDefaultHTML = function() {
            return 'toolbar HTML' // src/trix/config/toolbar.js
}