Speckle pattern, which is inherent in coherence imaging, influences significantly axial and transversal resolution of Optical Coherence Tomography (OCT) instruments. The well known speckle removal techniques are either sensitive to sample motion, require sophisticated and expensive sample tracking systems, or involve sophisticated numerical procedures. As a result, their applicability to in vivo real-time imaging is limited. In this work, we propose to average multiple A-scans collected in a fully controlled way to reduce the speckle contrast. This procedure involves non-coherent averaging of OCT A-scans acquired from adjacent locations on the sample. The technique exploits scanning protocol with fast beam deflection in the direction perpendicular to lateral dimension of the cross-sectional image. Such scanning protocol reduces the time interval between A-scans to be averaged to the repetition time of the acquisition system. Consequently, the averaging algorithm is immune to bulk motion of an investigated sample, does not require any sophisticated data processing to align cross-sectional images, and allows for precise control of lateral shift of the scanning beam on the object. The technique is tested with standard Spectral OCT system with an extra resonant scanner used for rapid beam deflection in the lateral direction. Ultrahigh speed CMOS camera serves as a detector and acquires 200,000 spectra per second. A dedicated A-scan generation algorithm allows for real-time display of images with reduced speckle contrast at 6 frames/second. This technique is applied to in vivo imaging of anterior and posterior segments of the human eye and human skin.