Updating Taxonomy field throws exception when "castTo TaxonomyField"

I have a list that contains a Taxonomy Field. I want a function that updates an item from that list (I have the id) and changes that field to one of its values (always the same one). I've searched and using functions I used in the past this is the code that I am using right now:

function SetSingleTaxonomyField(siteUrl, listName, fieldInternalName, itemId, term) {
        return new Promise(function (resolve, reject) {
            var context = new SP.ClientContext(siteUrl);
            var list = context.get_web().get_lists().getByTitle(listName);
            var listItem = list.getItemById(itemId);
            context.load(listItem);

            var categoryField = list.get_fields().getByInternalNameOrTitle(fieldInternalName);
            var taxonomyValue = set_taxonomyField(term);
            var taxField = context.castTo(categoryField, SP.Taxonomy.TaxonomyField);
            taxField.setFieldValueByValue(listItem, taxonomyValue);
            listItem.update();
            context.load(listItem);
            context.executeQueryAsync(function () {
                resolve();
                //console.log('Field successfully updated.');
            }, function (sender, args) {
                reject('An error occurred:' + args.get_message());
                //console.log('An error occurred:' + args.get_message());
            });
        });
    }

The line above:

var taxonomyValue = set_taxonomyField(term);

calls this function:

        function set_taxonomyField(term) {
        var taxonomyValue = new SP.Taxonomy.TaxonomyFieldValue();
        if (term !== undefined) {
            taxonomyValue.set_label(term.Title);
            taxonomyValue.set_termGuid(term.Id);
            taxonomyValue.set_wssId(-1);
        }
        return taxonomyValue;
    }

The exception triggers at

     var taxField = context.castTo(categoryField, SP.Taxonomy.TaxonomyField);

This is the error I get:

Possible Unhandled Promise Rejection: Error: Sys.ArgumentException: Value does not fall within the expected range.
Parameter name: type
at Function.Error.create (ScriptResource.axd?d=N...fc8ae3:5)
at Function.Error.argument (ScriptResource....0&t=72fc8ae3:5)
at SP.ClientContext.castTo (sp.runtime.js:2)"

I have other similar functions but this is the only one that I have all "hardcoded" (the term name, field internal name and term id). But checking the other functions using the same term, the values feel correct.

I have no idea how to solve this, thanks in advance.

Today I had the same error than months ago, I finally fixed it. The problem was that I was loading in various places the file "SP.Taxonomy", so when using the SP.Taxonomy.TaxonomyField it was sending a wrong argument (not sure why, but removing all but one reference to SP.Taxonomy fixed it).