Predicting hearing outcomes in idiopathic sudden sensorineural hearing loss (ISSNHL) is still challenging. We hypothesized that assessment of the semicircular canal (SCC) function via the video head impulse test (vHIT) might provide prognostic information. The medical records of patients diagnosed with ISSNHL from January 2015 to December 2018 were retrospectively reviewed. The prognostic values of the vHIT and other previously known factors in predicting hearing recovery were analyzed using a logistic regression model. A total of 148 patients with normal contra-lesional hearing were analyzed. Fifty-seven patients exhibited low gain (<0.7) on the vHIT in at least one SCC, more than the number of patients complaining of dizziness. Multivariable analysis revealed that non-recovery of normal hearing was associated with older age (OR 1.040), worse canal paresis on the caloric test (OR 1.023), worse initial hearing thresholds (OR 1.045) and abnormal vHIT result in the posterior SCC (OR 3.670). Low vHIT gain in the posterior SCC had specificity of 94.4% and positive predictive value of 85.7% in predicting non-recovery of normal hearing. In conclusion, abnormal vHIT gain in the posterior SCC appears to be a specific prognostic factor for incomplete hearing recovery in ISSNHL.