# combining classifiers

Introduction of stacked and parallel combining, fixed and trained combiners. It hase to be realized that a combined a set of classifiers can only be better than the best of the constituting base classifiers if this set does not contain an 'optimal' classifier that can solve the problem by its own.

PRTools and PRDataSets should be in the path

Download the m-file from here. See http://37steps.com/prtools for more.

## Contents

delfigs; randreset; % for reproducability reps = 10; % repeat experiment reps times.

## stacked combining

Three simple classifiers are combined for a 10-class (digits) 64-dimensional problem. Learning curves are computed for the three classifiers and three combiners, two fixed (mean and product) and one trained (fisher).

r = {nmc ldc qdc([],[],1e-2)}; % three untrained base classifiers u = [r{:}]*classc; % untrained stacked combiner a = mfeat_kar; % the dataset, 10 x 200 objects in 64 dimensions trainsize = [0.07 0.1 0.15 0.2 0.3 0.5 0.7 0.9999]; % fractions e = zeros(reps,numel(trainsize),numel(r)+3); for n=1:reps [tt,s] = gendat(a,0.5); % 50-50 split in train and test set for i=1:numel(trainsize) randreset(n); % takes care that larger training sets t = gendat(tt,trainsize(i)); % include the smaller ones. w = t*u; % train combiner for j=1:numel(w.data) % test base classifiers e(n,i,j) = s*w.data{j}*testc; end e(n,i,j+1) = s*w*meanc*testc; % test mean combiner e(n,i,j+2) = s*w*prodc*testc; % test product combiner e(n,i,j+3) = s*(t*(w*fisherc))*testc; % train and test fisher combiner end end % plot and anotate tsizes = ceil(trainsize*size(t,1)); figure; plot(tsizes,squeeze(mean(e))); set(gca,'xscale','log'); axis([min(tsizes),ceil(max(tsizes)),0.03,0.15]); set(gca,'xtick',tsizes) names = char(char(getname(r)),char('mean combiner','product combiner','fisher combiner')); legend(names); xlabel('training set size'); ylabel(['Average error ( ' num2str(reps) ' exp)']); title(['Stacked combining on ' getname(a)]) fontsize(14); linewidth(2);

The combiners perform almost everywhere better than the base classifiers. The product rule needs more accurate confidences and thereby a larger training set. The fisher combiner, in fact optimizing a weighted version of the mean combiner, needs more data as well.

## parallel combining

A simple classifier is trained for three aligned feature sets of a 10-class (digits) 64-dimensional problem. Learning curves are computed for the three classifiers and three combiners, two fixed (mean and product) and one trained (fisher).

a = {mfeat_kar; mfeat_zer; mfeat_fac}; % three feature sets classf = qdc([],[],1e-2)*classc; % the classifier, regularized qdc trainsize = [0.07 0.1 0.15 0.2 0.3 0.5 0.7 0.9999]; % fractions e = zeros(reps,numel(trainsize),numel(r)+3); for n=1:reps [tt,s] = gendat(a,0.5); % 50-50 split in train and test set for i=1:numel(trainsize) randreset(n); % takes care that larger training sets t = gendat(tt,trainsize(i)); % include the smaller ones. w = t*classf; % train the classifier for 3 feature sets for j=1:numel(a) % test them e(n,i,j) = s{j}*w{j}*testc; end w = parallel(w); e(n,i,j+1) = [s{:}]*[w]*meanc*testc; % test mean combiner e(n,i,j+2) = [s{:}]*[w]*prodc*testc; % test product combiner e(n,i,j+3) = [s{:}]*([t{:}]*(w*fisherc))*testc; % train and test fisher combiner end end % plot and annotate tsizes = ceil(trainsize*size(t{1},1)); figure; plot(tsizes,squeeze(mean(e))); set(gca,'xscale','log'); axis([min(tsizes),ceil(max(tsizes)),0.01,0.35]); set(gca,'xtick',tsizes) names = char(char(getname(a)),char('mean combiner','product combiner','fisher combiner')); legend(names); xlabel('training set size'); ylabel(['Average error ( ' num2str(reps) ' exp)']); title(['Parallel combining on MFEAT']) fontsize(14); linewidth(2);

The three dataset perform rather differently. For sufficiently large training sets the combiners perform better.

## Show that trained combining is not useful here

axis([400 1000 0.02 0.06])

This is the zoomed version of the large training set sizes. It shows that for the trained fisher combiner the training set is still not sufficiently large and does not improve the results of the best feature set.