transformation issues when implementing fisheye distortion in a radial tree

Basically, I am attempting to apply the d3 fisheye distortion algorithm to a radial tree. I believe the issues I am encountering revolve around the fact that the coords being fed to the fisheye distortion are the coords computed by the d3.layout.tree. But the actual coords have been adjusted by the g transform. So, the coords resulting from the fisheye distortion need to be adjusted back to the g transform.

For example:

// re-setting the projection according to fisheye coords
diagonal.projection(function(d) { d.fisheye = fisheye(d); return [d.fisheye.y, d.fisheye.x / 180 * Math.PI]; })

I have been attempting this...here is the fiddle.

I am somewhat close...help is appreciated.

Following the direction I'd suggested in the comments, this is the result:

https://jsfiddle.net/xdk5ehcr/

Instead of using rotations and translations to position the nodes, I created two trigonometry-based functions to calculate horizontal and vertical position from the data (x,y) values, which are treated as polar coordinates.

Then I had to set the fisheye function to use my positioning functions as "accessor" functions instead of reading d.x and d.y directly. Unfortunately, the basic plug-in you were using for the fisheye didn't include a way to get and set x/y accessor functions, so I had to modify that too. I was surprised it wasn't already in the code; it's a standard functionality on most d3 layout objects.

(When I get github set up, I will have to make a pull request to add it in. I'll need to figure out how the fisheye scale/zoom function works, though -- I took that out of this example since you weren't using it.)

The positioning functions were as follows:

function getHPosition(d){
    //calculate the transformed (Cartesian) position (H, V)
    //(without fisheye effect)
    //from the polar coordinates (x,y) where 
    //x is the angle
    //y is the distance from (radius,radius)
    //See http://www.engineeringtoolbox.com/converting-cartesian-polar-coordinates-d_1347.html

    return (d.y)*Math.cos(d.x);
}
function getVPosition(d){
    return (d.y)*Math.sin(d.x);
};

The functions are used to set the original position of the nodes and links, and then once the fisheye kicks in it uses these functions internally, returning the results (with distortion if appropriate) as d.fisheye.x and d.fisheye.y.

For example, for links that means the projection setting the d3.svg.diagonal function like this for initialization:

var diagonal = d3.svg.diagonal()
    .projection(function(d) { 
        return [getHPosition(d), getVPosition(d)]; 
});

But like this for update:

diagonal.projection(function(d) { 
    d.fisheye = fisheye(d); 
    return [d.fisheye.x, d.fisheye.y]; 
});

There are a couple other little changes:

I simplified the dimensions of the plotting area a bit.

I added a background rectangle with pointer-events:all; so that the fisheye doesn't turn on and off as the mouse moves between nodes and empty background.

I didn't bother rotating the text (since the node groups are no longer rotating, it doesn't happen by default), but you could easily add in a rotate transformation on the individual text elements.

Finally, and this one stumped me for longer than I'd like to admit, the angles have to be in radians for the Javascript trig functions. Couldn't figure out why my layouts were so ugly, with overlapping lines. I thought it was something to do with switching between d3.svg.diagonal() and d3.svg.diagonal.radial() and spent a lot of time trying to do inverse-trig and all sorts of things...