Check if pubkey belongs to twisted Edwards25519

I want to check if some pubkey belongs to twisted edwards25519 (I guess this is used for ed25519 ?) The problem is that I have in theory some valid pubkeys like:

hash_hex = "3afe3342f7192e52e25ebc07ec77c22a8f2d1ba4ead93be774f5e4db918d82a0"

or

hash_hex = "fd739be0e59c072096693b83f67fb2a7fd4e4b487e040c5b128ff602504e6c72"

and to check if they are valid I use from libsodium:

auto result = crypto_core_ed25519_is_valid_point(reinterpret_cast<const unsigned char*>(hash_hex.c_str()));

and the thing is that for those pubkeys which should be in theory valid, I have in both cases 0 as a result, which means that checks didn't pass (according to https://doc.libsodium.org/advanced/point-arithmetic#point-validation). So my question is if I am using this function wrong ? should that key be delivered in another form ? or maybe somehow those keys are not valid for some reasons (I have them from some coin explorer, so in theory they should be valid) ? is there some online tool where I can check if those pubkey belongs to that eliptic curve ?

You need to convert your hex string into binary format. Internally, the ed25519 functions work on a 256 (crypto_core_ed25519_BYTES (32) * 8) bit unsigned integer. You can compare it with an uint64_t, which consists of 8 octets. The only difference is that there is no standard uint256_t type, so a pointer to an array of 32 unsigned char is used instead. I use std::uint8_t instead of unsigned char below, so if std::uint8_t is not an alias for unsigned char the program should fail to compile.

Converting the hex string to binary format is done like this.

• A nibble is 4 bits, which is what a single hex digit can represent. 0 = 0b0000 and f = 0b1111.
• You lookup each hex character in a lookup table to easily convert the character into the value 0 - 15 (decimal), 0b0000 - 0b1111 (binary).
• Since an uint8_t requires two nibbles, you combine them two and two. The first nibble is left shifted to form the high part of the final uint8_t and the second nibble is just bitwise OR:ed (|) with that result. Using fe as an example:

  f = 0b1111
  e = 0b1110
  

  shift f left 4 steps:

  f0 = 0b11110000
  

  bitwise OR with e

    0b11110000
  | 0b00001110
  ------------
    0b11111110 = 254 (dec)
  

Example:

#include <sodium.h>

#include <cstdint>
#include <iostream>
#include <string_view>
#include <vector>

// a function to convert a hex string into binary format
std::vector<std::uint8_t> str2bin(std::string_view hash_hex) {
    static constexpr std::string_view tab = "0123456789abcdef";

    std::vector<std::uint8_t> res(crypto_core_ed25519_BYTES);

    if(hash_hex.size() == crypto_core_ed25519_BYTES * 2) {
        for(size_t i = 0; i < res.size(); ++i) {
            // find the first nibble and left shift it 4 steps, then find the
            // second nibble and do a bitwise OR to combine them:
            res[i] = tab.find(hash_hex[i*2])<<4 | tab.find(hash_hex[i*2+1]);
        }
    }
    return res;
}

int main() {
    std::cout << std::boolalpha;

    for(auto hash_hex : {
        "3afe3342f7192e52e25ebc07ec77c22a8f2d1ba4ead93be774f5e4db918d82a0",
        "fd739be0e59c072096693b83f67fb2a7fd4e4b487e040c5b128ff602504e6c72",
        "this should fail" })
    {
        auto bin = str2bin(hash_hex);
        bool result = crypto_core_ed25519_is_valid_point(bin.data());

        std::cout << "is " << hash_hex << " ok: " << result << '\n';
    }
}

Output:

is 3afe3342f7192e52e25ebc07ec77c22a8f2d1ba4ead93be774f5e4db918d82a0 ok: true
is fd739be0e59c072096693b83f67fb2a7fd4e4b487e040c5b128ff602504e6c72 ok: true
is this should fail ok: false

Also note: libsodium comes with helper functions to do this conversion between hex strings and binary format:

char *sodium_bin2hex(char * const hex, const size_t hex_maxlen,
                     const unsigned char * const bin, const size_t bin_len);

int sodium_hex2bin(unsigned char * const bin, const size_t bin_maxlen,
                   const char * const hex, const size_t hex_len,
                   const char * const ignore, size_t * const bin_len,
                   const char ** const hex_end);