Improving seizure detection performance relies not only on novel signal processing approaches but also on new accurate, reliable and comparable performance reporting to give researchers better and fairer tools for understanding the true algorithm operation. This paper investigates the sensitivity of current performance metrics to the duration of data that must be marked as candidate seizure activity before a seizure detection is made. The results demonstrate that not all metrics are insensitive to this high level choice in the algorithm design, and provide new approaches for comparing between reported algorithm performances in a robust and reliable manner.