Similar articles

Abstract: An effective ensemble should consist of a set of networks that are both accurate and diverse. We propose a novel clustering-based selective algorithm for constructing neural network ensemble, where clustering technology is used to classify trained networks according to similarity and optimally select the most accurate individual network from each cluster to make up the ensemble. Empirical studies on regression of four typical datasets showed that this approach yields significantly smaller ensemble achieving better performance than other traditional ones such as Bagging and Boosting. The bias variance decomposition of the predictive error shows that the success of the proposed approach may lie in its properly tuning the bias/variance trade-off to reduce the prediction error (the sum of bias² and variance).

[1] Bauer, E., Kohavi, R., 1999. An empirical comparison of voting classification algorithms: Bagging, Boosting, and variants. Machine Learning, 36(1-2):105-139.

[2] Blake, C., Keogh, E., Merz, C.J., 1998. UCI Repository of Machine Learning Databases. http://www.ics.uci.edu/~mlearn/MLRepository.html. Department of Information and Computer Science, University of California, Irvine, CA.

[3] Breiman, L., 1996. Bagging predictors. Machine Learning, 24(2):123-140.

[4] Breiman, L., Friedman, J.H., 1985. Estimating optimal transformations in multiple regression and correlation (with discussion). Journal of the American Statistical Association, 80:580-619.

[5] Drucker, H., 1999. Boosting Using Neural Networks. In: Sharkey, A. (Ed.), Combining Artificial Neural Nets: Ensemble and Module Multi-net Systems. Springer-Verlag, London, p.42-49.

[6] Friedman, J.H., Grosse, E., Stuetzle, W., 1983. Multidimensional additive Spline approximation. SIAM Journal of Scientific and Statistical Computing, 4:292-301.

[7] Fu, Q., Hu, S.X., Zhao, S.Y., 2004. A PSO-based approach for neural network ensemble. Journal of Zhejiang University (Engineering Science), 38(12):1596-1600 (in Chinese).

[8] German, S., Bienenstock, E., Doursat, R., 1992. Neural networks and the bias/variance dilemma. Neural Computation, 4(1):1-58.

[9] Hansen, J.V., 2000. Combining Predictors: Meta Machine Learning Methods and Bias/variance and Ambiguity Decomposition. Ph. D Dissertation, Department of Computer Science, University of Aarhus, Denmark.

[10] Hansen, L.K., Salamon, P., 1990. Neural network ensembles. IEEE Transaction on Pattern Analysis and Machine Intelligence, 12(10):993-1001.

[11] Krogh, A., Vedelsdy, J., 1995. Neural Network Ensembles Cross Validation, and Active Learning. In: Tesauro, G., Touretzky, D., Leen, T. (Eds.), Advances in Neural Information Processing Systems, Volume 7. MIT Press, Cambridge, MA, p.231-238.

[12] Lazarevic, A., Obradovic, Z., 2001. Effective pruning of neural network classifier ensembles. Proc. International Joint Conference on Neural Networks, 2:796-801.

[13] Liu, Y., Yao, X., 2000. Evolutionary ensembles with negative correlation learning. IEEE Trans. Evolutionary Computation, 4(4):380-387.

[14] Melville, P., Mooney, R., 2003. Constructing Diverse Classifier Ensembles Using Artificial Training Examples. Proc. of the IJCAI-2003, Acapulco, Mexico, p.505-510.

[15] Navone, H.D., Verdes P.F., Granitto, P.M., Ceccatto, H.A., 2000. Selecting Diverse Members of Neural Network Ensembles. Proc. 16th Brazilian Symposium on Neural Networks, p.255-260.

[16] Opitz, D., Shavlik, J., 1996. Actively searching for an effective neural network ensemble. Connection Science, 8(3-4):337-353.

[17] Ridgeway, G., Madigan, D., Richardson, T., 1999. Boosting Methodology for Regression Problems. Proc. 7th Int. Workshop on Artificial Intelligence and Statistics. Fort Lauderdale, FL, p.152-161.

[18] Rosen, B.E., 1996. Ensemble learning using decorrelated neural network. Connection Science, 8(3-4):373-384.

[19] Schapire, R.E., 1990. The strength of weak learn ability. Machine Learning, 5(2):1971-227.

[20] Zhou, Z.H., Wu, J.X., Jiang, Y., Chen, S.F., 2001. Genetic algorithm based selective neural network ensemble. Proc. 17th International Joint Conference on Artificial Intelligence, 2:797-802.