Rotating a curve using python

I have written following code to rotate a curve by specified angle and return new equation. I know when we want to rotate the axes by an angle A then new coords become X=xcosA-ysinA and Y=xsinA+ycosA. when i test my fxn on the hyperbola x^2+y^2=1 . Expected equations is 2xy-1=0 but my fxn gives -2xy-1=0 Where I am doing wrong?

import sympy as sp
@display_mathjax_and_replace
def rotate(f,theta):
   x, y, a, b = sp.symbols('x y a b')
   rotation_matrix = sp.Matrix([[sp.cos(theta),-sp.sin(theta)],[sp.sin(theta), sp.cos(theta)]])
   transformed_coords = rotation_matrix * sp.Matrix([a, b])
   g = f.subs({x: transformed_coords[0], y: transformed_coords[1]})
   return g

I have created following decorator to present it beautifully:

def display_mathjax_and_replace(func):
  def wrapper(*args, **kwargs):
    result = func(*args, **kwargs)
    result = result.subs({'a': 'x', 'b': 'y'})
    simplified_expr = sp.simplify(result)
    display(Math(sp.latex(sp.Eq(simplified_expr, 0))))
  return wrapper

I call my code :

x,y = sp.symbols('x y')
f = x**2 - y**2 -1
rotate(f,sp.pi/4)

Output: -2xy-1

Expected output:2xy-1

You swapped the sign on the sin terms of the transformation matrix. Here is how it should look like for a positive rotation:

def rotate_1(f, theta):
   x, y, a, b = sp.symbols('x y a b')
   rotation_matrix = sp.Matrix([[sp.cos(theta),sp.sin(theta)],[-sp.sin(theta), sp.cos(theta)]])
   transformed_coords = rotation_matrix * sp.Matrix([a, b])
   g = f.subs({x: transformed_coords[0], y: transformed_coords[1]})
   return g

rotate(f, sp.pi/4).simplify()
# out: 2*a*b - 1

If you use SymPy Plotting Backend and an interactive environment like Jupyter Notebook, you can perform nice visualization that helps you catch any eventual mistake. For example:

%matplotlib widget
from sympy import *
from spb import *

def symbolic_rotate(f, theta, x, y):
    a, b = var("a, b")
    rotation_matrix = Matrix([[cos(theta),sin(theta)],[-sin(theta), cos(theta)]])
    transformed_coords = rotation_matrix * Matrix([a, b])
    g = f.subs({x: transformed_coords[0], y: transformed_coords[1]})
    g = g.subs({a: x, b: y})
    return g

var("x, y, theta, a, b")
f = x**2 - y**2 -1

graphics(
    implicit_2d(f, (x, -10, 10), (y, -10, 10), label="f"),
    implicit_2d(symbolic_rotate(f, theta, x, y), (x, -10, 10), (y, -10, 10), label="f rot",
        params={theta: (pi/6, 0, 2*pi)}),
    aspect="equal"
)

symbolic_rotate creates a new expression based on a symbolic angle of rotation. Then, we can plot the original hyperbole and the rotated one and play with a slider. Here, you can visually verify if the transformation matrix is correct.