I'm in the need to do parameter optimization for my latest research project. I have an algorithm which has currently 5 parameters (four double [0,1] and one nominal with 3 values). The algorithm uses those parameters to calculate some stuff and afterwards I calculate the Precision, Recall & FMeasure. A single run takes about 1,8s. Currently I'm going through each parameter with a 0.1 step size which shows me approximately where the global maxima is. But I want to find the precise global maximum. I've looked into Gradient Descent but I don't really know how to apply this to my algorithm (if it's even possible). Could anybody please guide me a little how I would implement such an algorithm since I'm very new to this kind of work.
Cheers, Daniel
You can certainly do better than a grid search.
Before applying an algorithm like gradient descent, you have to be sure that your parameter space does not contain local maxima or that at least your starting point is close to the global maximum and your step size is appropriate enough to bring you to it.
In your case, I would recommend starting by drawing as many random samples as you can. This is a much better way of exploring the parameter space than a grid search. Once you collect enough data this way, you can use a mode-finding algorithm, such as mean shift or one of its faster derivatives, or go straight to optimization. Since you don't have the Jacobian of your parameter space, you could use the Broyden's method, which iteratively approximates it or a secant method, such as BFGS.
Also, see this related question: How can I adjust parameters for image processing algorithm in an efficient way?