Infrared optical wireless communication system can achieve ultrahigh capacity and high privacy data transmission. However, for using narrow infrared laser beam as carrier to transmit signal, the high-speed data transmission can only be achieved by point-to-point connection. With the rapid number increasement of consumer electronic devices, such connection method puts a heavy burden on the number of transmitters. Thus, the transmitting end with the point-to-multipoint capability or multi-user accessibility is required. In this paper, we present a multi-user accessible indoor infrared optical wireless communication system employing passive diffractive optics based on a virtually imaged phased array (VIPA). Multiple beams can be generated in a point-to-multipoint scheme by using VIPA-based beam-steering antenna (BSA). On the other hand, by tuning wavelength of laser source, fast 2D steering of multiple beams with the same steering trajectory is supported, which can be used for user ends with changing locations. In the experiment, 5 beams are generated by utilizing only one transmitter. All five beams can realize 12.5 Gb/s on-off-keying (OOK) data rate transmission. Free-space optical wireless transmission at 3.6-m communication distance is demonstrated for system performance verification and evaluation. a total 3.44°×7.9° scanning field of view of five beams is achieved.