I would love to set up my own server with deis.io.
I currently have a Ubuntu 12.04 installation. I have root access to the server, but I think I can't install another OS. So how could I run CoreOS?
Would love to use this for productive apps. Thats why I don't know if I should use Vagrant.
Is there a tutorial out there?
You could use vagrant to set it up. It's not going to be great performance but it will work. Vagrant requires VirtualBox so you'll first have to install that, and then Vagrant. You will also need to install fleetctl, you will need at least 0.5.0 which you can find on their GitHub repo.
Once you have those installed, you can then clone the Deis GitHub repo, and follow the instructions found here: https://github.com/deis/deis/blob/master/README.md
If you're stuck, jump on IRC and head over to #deis and we'll help you out!