So I would like to create a compositor for wayland which supports 3D effects for windows(something resembling compiz, but on wayland). I already saw this question: Where do I start if I want to write a wayland compositor? but the only answer points to SWC(https://github.com/michaelforney/swc), which is not applicable in my case as I want to use OpenGL and because SWC doesn't support 3D easily. So is there some project/library/book/tutorial/etc where I can learn the necessary things for writing my own WM on wayland? Thanks in advance.
The only purpose of the wayland protocol is the communication between client and server. The server provides the client with input events and the client provides the server with a buffer (that can be mapped to an OpenGL texture). Where the server/compositor gets its input events from and what it does with the buffer is completely up to the compositor.
So the compositor itself needs a source for input events and a way to draw its result. That's why many wayland compositors have multiple backends: they can run on top of X11, directly on top of the Linux kernel or even on top on another wayland server.
The answer to your question really depends on where you want to run your compositor. Writing a compositor that runs on top of X11 might be the easiest way to get started if you're already familiar with how to get an OpenGL app up and running there. If you want to run your compositor directly on top of the Linux kernel you'd probably want to look into evdev and libinput for input and DRM/KMS together with EGL on top of GBM in order to create an OpenGL context and show the result on your monitor. There are also rendering libraries (e.g. evas) that can run directly on top of the Linux kernel but I don't know how far they let you inject your own OpenGL code.
Once you have decided where you want to run your compositor you can start by just writing a regular OpenGL app and then go on and integrate a wayland server in order to display and interact with actual client windows.