Please help me fix my confusion:
In Elf64 File at first we the elf header, then immediately all program header and then all section headers?
Then why for example in elf header we have a index to where the first program header starts if it's always 64 bytes after the start of the file, that seems to be extra garbage information?
If nothing else, it allows the header's size to be made larger in future versions of the format without losing backwards compatibility.
In Elf64 File at first we the elf header, then immediately all program header and then all section headers?
No, the program header, the section headers and the sections are wherever the headers say they are. There's no requirement for them to be immediately after one another or in any particular order.