This study aimed to identify the physical factors contributing to the perception of the hardness of objects tapped using white canes, which are commonly used by the visually impaired for autonomous walking. First, physical factors such as vibration tapping sounds and reaction forces were measured during the indirect tapping of rubber sheets with different hardness using a white cane. Second, we determined the relationship between the subjective hardness perceived by the visually impaired individuals and the physical factors through multivariate analysis. In addition, we estimated the contribution of each factor. The result indicates that the white cane vibrated at similar frequencies during tapping even when the hardness of objects changed. In contrast, the tapping sound varied widely with the variation of the hardness of the rubber sheet. In addition, the peak values of the reaction forces significantly changed depending on the hardness. Based on the results of the multivariate analysis, the contributions of tapping sounds and reaction forces to hardness perception were remarkable. In contrast, the contributions of vibrations could not be found using our analytic method. These findings will help in the design and evaluation of support equipment for the visually impaired.