Converting an equiangular cubemap to an equirectangular one

I am making a retro-style game with OpenGL, and I want to draw my own cubemaps for it. Here is an example of one:

As you can tell, there is no perspective warping anywhere; each face is fully equiangular. When using this as a cubemap, the result is this:

As you can see, it looks box-y, and not spherical at all. I know of a solution to this, which is to remap each point on the cubemap to a a sphere position. I have done this manually by creating a sphere mesh and mapping the cubemap texture onto it (and then rendering that to an environment map), but this is time-consuming and complicated.

I seek a different solution: in my fragment shader, I hope to remap the sampling ray to a sphere position, instead of a cube position. Here is my original fragment shader, without any changes:

#version 400 core

in vec3 cube_edge;

out vec3 color;

uniform samplerCube skybox_sampler;

void main(void) {
    color = texture(skybox_sampler, cube_edge).rgb;
}

I can get a ray that maps to the sphere by just normalizing cube_edge, but that doesn't change anything, for some reason. After messing around a bit, I tried this mapping, which almost works, but not quite:

vec3 sphere_edge = vec3(cube_edge.x, normalize(cube_edge).y, cube_edge.z);

As you can see, some faces become spherical in nature, whereas the top face warps inwards, instead of outwards.

I also tried the results from this site: http://mathproofs.blogspot.com/2005/07/mapping-cube-to-sphere.html, but the faces were not curved outwards enough.

I have been stuck on this for so long now - if you know how I can change my cube to sphere mapping in my fragment shader, or if that's even possible, please let me know!

As you can tell, there is no perspective warping anywhere; each face is fully equiangular.

This premise is incorrect. You hand-drew some images; this doesn't make them equiangular.

'Equiangular cubemap' (EAC) specifically means a cubemap remapped by this formula (section 2.4):

u = 4/pi * atan(u)
v = 4/pi * atan(v)

Let's recognize first that the term is misleading, because even though EAC aims at reducing the variation in sampling rate, the sampling rate is not constant. In fact no 2d projection of any part of a sphere can truly be equi-angular; this is a mathematical fact.

Nonetheless, we can try to apply this correction. Implemented in GLSL fragment shader as:

d /= max(abs(d.x), max(abs(d.y), abs(d.z));
d = atan(d)/atan(1);

gives the following result:

Compare it with the uncorrected d:

As you can see the EAC projection shrinks the pixels in the middle by a little bit, and expands them near the corners, so that they cover more equal area. However, EAC is not a smooth mapping near the edges of the cube -- i.e. it can make straight lines pointy -- which seems like the artifact you want to avoid.

Instead, it appears that you want a cylindrical projection around the horizon, which is smooth. It can be implemented like so:

d /= length(d.xy);
d.xy /= max(abs(d.x), abs(d.y));
d.xy = atan(d.xy)/atan(1);

And it gives the following result:

However there's no artifact-free way to fit the top/bottom square faces of the cube onto the circular faces of the cylinder -- which is why you see the artifacts there.

Bottom-line: you cannot fit the image that you drew onto a sphere in a visually pleasing way. You should instead re-focus your effort on alternative ways of authoring your environment map. I recommend you try using an equidistant cylindrical projection for the horizon, cap it with solid colors above/below a fixed latitude, and use billboards for objects that cannot be represented in that projection.

For example, using the following 2D texture:

And the following formula:

float tau = 6.283185;
float u = atan(d.y, d.x);
float v = atan(d.z, length(d.xy));
OUT = texture(TEX, vec2(u/tau, 2*v/tau + 0.5));

I can get these results:

I think this will get you further than any cubemap based tweaks.