When comparing REPTree and Linear Regression model, what is the most important factor to take into account when deciding which model is better?
Is it the correlation coefficient or relative absolute error between the two? Also, is the higher the better within those two?
Never rely on just a single summary statistic and always plot your predictions. Summary statistics can be misleading when there are outliers (see Anscombe's quartet). Correlation coefficient of 1 means perfect correlation. For errors, the smaller, the better.
For comparing classifiers, use the Weka Experimenter. The Weka manual describes this tool.