I know that AMD64 aka. x86-64 is AMD's own proprietary technology and can be licensed by 3rd parties, and they do, like Intel, VIA, etc.
I also know that the "big thing" about AMD64 ISA is that it extends the x86 ISA, thus compatibility is a fundamental advantage over the Intel's IA-64.
But (my question comes now ;)) as AMD64 relies on the basic x86 instruction set, does this mean if AMD would not get a license to that from Intel, AMD64 would be just an extension to x86 without the x86 instruction set itself, or does AMD64 "reimplement/redefine" the whole x86 ISA making the x86 license unnecessary in this regard? (I guess licensing the x86 by AMD is not just about having a complete ISA with the AMD64, so this question is just a "what if"-kind to let me better understand how AMD64 depends on or free from x86.)
If a manufacturer wants to make a CPU purely with AMD64 ISA, is it possible to make an OS that runs on it? Will it involve x86 instruction set? Or AMD64 cannot be defined without x86 so there's a bunch of basic instructions that are not part of the AMD64, thus without them there's no way a CPU can work at all?
Unlike AArch64 vs. ARM 32-bit, it's not even a separate machine-code format. I think you'd have a hard time justifying an x86-64 as separate from x86, even if you left out "legacy mode" (i.e. the ability to work exactly like a 32-bit-only CPU, until/unless you enable 64-bit mode).
x86-64's 64-bit mode uses the same opcodes and instruction formats (with mostly just a new prefix, REX). https://wiki.osdev.org/X86-64_Instruction_Encoding. I doubt anyone could argue it was substantially different from x86, or whatever the required standard is for patents. (Although patents on that might be long expired, if they were for 8086).
Especially given that long mode includes 32/16-bit "compat" sub-modes (https://en.wikipedia.org/wiki/X86-64#Operating_modes), and uses Intel's existing PAE page-table format.
But note that a lot of the patent-sharing stuff between Intel and AMD is for implementation techniques, for example a "stack engine" that handles the modification-to-stack-pointer part of push/pop/call/ret, letting it decode to 1 uop and avoiding a latency chain through RSP. Or IT-TAGE branch prediction (Intel Haswell, AMD Zen 2). Or perhaps the whole concept of decoding to uops, which Intel first did with P6 (Pentium Pro) in ~1995.
Presumably there are also patents on ISA extensions like SSE4.1 and AVX that would make unattractive to sell a CPU without, for most purposes. (SSE2 is baseline for x86-64, so you need that. Again, the instructions and machine-code formats are identical to 32-bit mode.)
BTW, you'd have to invent a way for it to boot, starting in long mode which requires paging to be enabled. So maybe boot with a direct-mapping of some range of addresses? Or invent a new sub-mode of long mode that allows paging to be disabled, using physical addresses directly.
The firmware could handle this and boot via 64-bit UEFI, potentially allowing 64-bit OSes to run unmodified as long as they never switch out of long mode.
Note that AMD, when designing AMD64, intentionally kept x86's variable-length hard-to-decode machine-code format as unchanged as possible, and made as few other changes as possible.
This means the CPU hardware doesn't need separate decoders, or separate handling in execution units, to run in 64-bit mode. AMD weren't sure AMD64 was going to catch on, and presumably didn't want to be stuck needing a lot of extra transistors to implement 64-bit mode when hardly anybody was going to take advantage of it.
(Which was definitely true even in practice for their first generation K8 chips; it was years before 64-bit Windows was common, and GNU/Linux users running still-evolving amd64 ports of distros were only a small fraction of the market back in 2003.)
Unfortunately this means that unlike AArch64, AMD64 missed the opportunity to clean up some of x86's minor warts (like providing setcc r/m32 instead of the inconvenient setcc r/m8 is my favourite example of something I would have changed for the semantics of an opcode in 64-bit mode vs. 16 and 32.)
I can see why they didn't want to totally redesign the machine-code format and need an entirely new decoding method; as well as costing silicon, that would force toolchain software (assemblers / disassemblers) to change more, instead of mostly minor changes to existing tools. That would slightly raise the barrier to adoption of their extension to x86, which was critical for them to beat IA-64.
(IA-64 was Intel's 64-bit ISA at the time, whose semantics are very different from x86 and thus couldn't even share much of a back-end. It would have been possible to redesign machine-code for mostly the same instruction semantics as x86 though. See Could a processor be made that supports multiple ISAs? (ex: ARM + x86) for more about this point: separate front-ends to feed a common back-end can totally work if the ISAs are basically the same, like just a different machine-code format for most of the same semantics.)