I'm looking for a simple 3D engine or toolkit that I can experiment computer graphics algorithms with. E.g. applying transformation matrices, seeing what happens.
Can anyone make a suggestion?
One of the most well known material to study and learn about rendering is the book from Matt Pharr and Greg Humphreys called Physically Based Rendering. It is a book but it also comes with the source code of a renderer (open source). Many existing renderers have used PBRT (that's the name of the renderer) as their starting point.
Another source to learn about CG programming techniques is www.scratchapixel.com. This one has the advantage to be online. While it doesn't really provide you with the source code of a complete renderer, it still provides you with many examples in C++ that show how rendering techniques can be implemented.
I actually found scratchapixel useful to read before getting to a book like PBRT, which is more abstract and more technical in a way.
Alternatively, you can use something like the OpengGL API to write graphics applications. However, it will be harder for you to explore techniques such as Ray Tracing with OpenGL. And it's more about learning the API than really understanding how the algorithms used by OpenGL under the hood work (while this is what the two ressources mentioned above help you with). But it is perfect for what you want to do (playing with matrices, etc.)