Background: I'm writing a program that creates generative art. I care about creating one final static image, and I don't need to render a bunch of frames per second. So far it's been 2D, and I'm on a Mac so I've been using the Core Graphics (aka Quartz) 2D drawing API. I've reached it's limits so I started messing with OpenGL, but I'm not happy with the antialiasing so far.
I'm wondering if I should invest in learning it, or whether it's not built for what I want. Is OpenGL more about creating moving graphics as fast as possible, mainly for games? If I want the highest quality rendering (high resolution, smooth curves, best antialiasing, arbitrary lighting and shading algorithms) do I need to write my own renderer, or does it make sense to learn OpenGL? Will I be able to use it as a base?
OpenGL is not a general purpose graphics library.
OpenGL is a API with the design being focused on controlling GPUs for purposes of drawing realtime graphics. If you know how to use it you can use OpenGL to generate high quality, close to photorealistic drawings. But it takes a lot of effort to do this.
Antialiasing is actually rather easy to do with high quality: Select a multisampled frambuffer format with a high subsampling density, enable multisampling and render.
However your use case sounds more like the task for an offline renderer like Renderman, Pixie, Yafa-Ray, and similar.