I understand that on the wire, most integers are in big endian format.
But why is it the burden of the application to do the byte swapping in structures like sockaddr_in and not the kernels, where all the low level work actually happens? It would make more sense if the userspace API was more platform agnostic and should not deal with this.
Why was the Berkeley socket API designed like this?
The reason is probably historical.
The socket API was invented (in the 1980s) when Sun-3 (MC68030) and Sun-4 (Sparc) workstations were kings. The endianness of these (slow by today's standards) processors mattered.
I forgot the details, and probably BSD sockets conventions have been invented for some PDP-11 or VAX-780.
But why is it the burden of the application to do the byte swapping in structures like sockaddr_in and not the kernels
Probably because in the 1980s you did not want the computer (a thousand times slower than your mobile phone) to spend too much (uninterruptible) time in kernel-land.
That question should really be asked on https://retrocomputing.stackexchange.com/ (and its answer lies inside the source code of some 1980s era Unix kernel)