I want to use a custom shaped window using gtk3. I found gtk_widget_shape_combine_mask_() in gtk2. The closest thing I can find in gtk3 is gdk_window_shape_combine_region_(), which only allows for rectangular shaped regions. Why was the support dropped? Or was it changed and implemented in a different way?
What do you mean with "why was the support dropped?".
gtk_widget_shape_combine_mask() only supports rectangular shapes, too. That's what a GdkBitmap is: An image with a bit depth of 1. This means that a pixel is either included or not.
This is basically the same as a cairo_region_t. The difference is only in how the data is stored: Instead of a bitmap, cairo_region_t uses a list of rectangles to describe the same result.
"Or was it changed and implemented in a different way?"
Yes and you already found the replacement yourself.