Assessments of watershed condition for aquatic and riparian species often have to rely on expert opinion because of the complexity of establishing statistical relationships among the many factors involved. Such expert-based assessments can be difficult to document and apply consistently over time and space. We describe and reflect on the process of developing a computer-based decision support application from expert judgments for assessing aquatic and riparian conditions over the 100,000 km² managed by the US federal government under the Northwest Forest Plan. The decision support system helped structure and document the assessment process and provided consistency and transparency to the evaluation methodology. However, many decisions and trade-offs were required in the expert engagement and model-building processes. Knowledge elicitation in an interactive group had a number of benefits over nominal group or Delphi processes, but efficient knowledge capture required considerable planning and expertise in the subject matter and modeling process. Communicating model results for validation was problematic and only effectively accomplished via in-person workshops. The choice to use different expert groups for each biophysical province provided more opportunities for participation and promoted greater ownership in the assessment, but it also led to increased variation among the resulting model structures. We propose three possible approaches for better managing the consistency of assessment models when multiple expert groups are involved.