We present an open, parametric system for automatic detection of EEG artifacts in polysomnographic recordings. It relies on independent parameters reflecting the relative presence of each of the eight types of artifacts in a given epoch. An artifact is marked if any of these parameters exceeds a threshold. These thresholds, set for each parameter separately, can be adjusted via "learning by example" procedure (multidimensional minimization with computationally intensive cost function), which can be used to automatically tune the parameters to new types of datasets, environments or requirements. Performance of the system, evaluated on 103 overnight polysomnographic recordings, revealed concordance with decisions of human experts close to the inter-expert agreement. To make this statement well defined, we review the methodology of evaluation for this kind of detection systems. Complete source code is available from http://eeg.pl; a user-friendly version with Java interface is available from http://signalml.org.