pythonscipyregression

how can I do a maximum likelihood regression using scipy.optimize.minimize


How can I do a maximum likelihood regression using scipy.optimize.minimize? I specifically want to use the minimize function here, because I have a complex model and need to add some constraints. I am currently trying a simple example using the following:

from scipy.optimize import minimize

def lik(parameters):
    m = parameters[0]
    b = parameters[1]
    sigma = parameters[2]
    for i in np.arange(0, len(x)):
        y_exp = m * x + b
    L = sum(np.log(sigma) + 0.5 * np.log(2 * np.pi) + (y - y_exp) ** 2 / (2 * sigma ** 2))
    return L

x = [1,2,3,4,5]
y = [2,3,4,5,6]
lik_model = minimize(lik, np.array([1,1,1]), method='L-BFGS-B', options={'disp': True})

When I run this, convergence fails. Does anyone know what is wrong with my code?

The message I get running this is 'ABNORMAL_TERMINATION_IN_LNSRCH'. I am using the same algorithm that I have working using optim in R.


Solution

  • Thank you Aleksander. You were correct that my likelihood function was wrong, not the code. Using a formula I found on wikipedia I adjusted the code to:

    import numpy as np
    from scipy.optimize import minimize
    
    def lik(parameters):
        m = parameters[0]
        b = parameters[1]
        sigma = parameters[2]
        for i in np.arange(0, len(x)):
            y_exp = m * x + b
        L = (len(x)/2 * np.log(2 * np.pi) + len(x)/2 * np.log(sigma ** 2) + 1 /
             (2 * sigma ** 2) * sum((y - y_exp) ** 2))
        return L
    
    x = np.array([1,2,3,4,5])
    y = np.array([2,5,8,11,14])
    lik_model = minimize(lik, np.array([1,1,1]), method='L-BFGS-B')
    plt.scatter(x,y)
    plt.plot(x, lik_model['x'][0] * x + lik_model['x'][1])
    plt.show()
    

    Now it seems to be working.

    maximum likelihood regression

    Thanks for the help!