tried to understand why my optimization is it taking so long, I put time stamps in my code. the problem is between the stamp just before the return and one in the beginning of the function "start iteration". while times of operations within the function stay more or less the same, this timing just keeps increasing.. tried profiling memory, it is flat from the time dataframes are built to when I stop it.. ideas?
this is the code:
def objective(df_initial_guess, shape, now):
print("start iteration -", time.time()-now)
now = time.time()
df_initial_guess = df_initial_guess.reshape(shape)
global df_training_data # Use the global keyword
print("initial guess and training data organized -", time.time()-now)
now = time.time()
......
print("formula spectrum calculation done -", time.time()-now)
now = time.time()
data=np.array(data)*100
df_training_data[57]=data
print("anomaly calculation done -", time.time()-now)
now = time.time()
print()
return df_training_data.loc[df_training_data.iloc[:,7] >= 1, 57].mean()
result=minimize(objective,df_initial_guess, args=(shape,now,))
and this is what I get in the terminal:
start iteration - 0.001001119613647461
initial guess and training data organized - 0.0
Pure color's constants were calculated - 0.2595968246459961
formulas' constants were calculated - 10.22897744178772
formula spectrum calculation done - 160.1121380329132
anomaly calculation done - 0.701399564743042
start iteration - 171.34229683876038
initial guess and training data organized - 0.015207767486572266
Pure color's constants were calculated - 0.18435430526733398
formulas' constants were calculated - 10.336299896240234
formula spectrum calculation done - 145.85290336608887
anomaly calculation done - 1.0375640392303467
start iteration - 328.769579410553
initial guess and training data organized - 0.0027430057525634766
Pure color's constants were calculated - 0.19941067695617676
formulas' constants were calculated - 14.058949947357178
formula spectrum calculation done - 133.65063190460205
anomaly calculation done - 0.6168942451477051
start iteration - 477.33042335510254
initial guess and training data organized - 0.00693202018737793
Pure color's constants were calculated - 0.1299881935119629
formulas' constants were calculated - 8.694848537445068
formula spectrum calculation done - 134.9823079109192
anomaly calculation done - 0.5860121250152588
start iteration - 621.7472784519196
initial guess and training data organized - 0.009764671325683594
Pure color's constants were calculated - 0.12981104850769043
formulas' constants were calculated - 8.949415445327759
formula spectrum calculation done - 146.67141890525818
anomaly calculation done - 0.46207499504089355
start iteration - 777.9782621860504
initial guess and training data organized - 0.0010035037994384766
Pure color's constants were calculated - 0.1138761043548584
formulas' constants were calculated - 7.141656398773193
formula spectrum calculation done - 114.04382729530334
anomaly calculation done - 0.48108625411987305
start iteration - 899.7692215442657
initial guess and training data organized - 0.0010166168212890625
Pure color's constants were calculated - 0.11710047721862793
formulas' constants were calculated - 7.446816444396973
described above in the main text
Unless I'm wrong, the args arguments are fixed so now is never reevaluated:
Documentation of minimize:
The objective function to be minimized:
fun(x, *args) -> float
where x is a 1-D array with shape (n,) and args is a tuple
of the fixed parameters needed to completely specify the function.
To check that, insert print('now -', now) at the beginning of your minimize function:
def objective(df_initial_guess, shape, now):
print('now -', now)
print("start iteration -", time.time()-now)
...
Why don't you do:
def objective(df_initial_guess, shape):
now = time.time()
print("start iteration -", time.time()-now)
...
result = minimize(objective, df_initial_guess, args=(shape,))