An operational forecast system for harmful algal blooms (HABs) in southwest Florida is analyzed for forecasting skill. The HABs, caused by the toxic dinoflagellate, Karenia brevis, lead to shellfish toxicity and to respiratory irritation. In addition to predicting new blooms and their extent, HAB forecasts are made twice weekly during a bloom event, using a combination of satellite derived image products, wind predictions, and a rule-based model derived from previous observations and research. These forecasts include: identification, intensification, transport, extent, and impact; the latter being the most significant to the public. Identification involves identifying new blooms as HABs and is validated against an operational monitoring program involving water sampling. Intensification forecasts, which are much less frequently made, can only be evaluated with satellite data on mono-specific blooms. Extent and transport forecasts of HABs are also evaluated against the water samples. Due to the resolution of the forecasts and available validation data, skill cannot be resolved at scales finer than 30 km. Initially, respiratory irritation forecasts were analyzed using anecdotal information, the only available data, which had a bias toward major respiratory events leading to a forecast accuracy exceeding 90%. When a systematic program of twice-daily observations from lifeguards was implemented, the forecast could be meaningfully assessed. The results show that the forecasts identify the occurrence of respiratory events at all lifeguard beaches 70% of the time. However, a high rate (80%) of false positive forecasts occurred at any given beach. As the forecasts were made at half to whole county level, the resolution of the validation data was reduced to county level, reducing false positives to 22% (accuracy of 78%). The study indicates the importance of systematic sampling, even when using qualitative descriptors, the use of validation resolution to evaluate forecast capabilities, and the need to match forecast and validation resolutions.