We successfully demonstrated an intelligent adaptive beam alignment scheme using a reinforcement learning (RL) algorithm integrated with an 8 × 8 photonic array antenna operating in the 40 GHz millimeter wave (MMW) band. In our proposed scheme, the three key elements of RL: state, action, and reward, are represented as the phase values in the photonic array antenna, phase changes with specified steps, and an obtained error vector magnitude (EVM) value, respectively. Furthermore, thanks to the Q-table, the RL agent can effectively choose the most suitable action based on its prior experiences. As a result, the proposed scheme autonomously achieves the best EVM performance by determining the optimal phase. In this Letter, we verify the capability of the proposed scheme in single- and multiple-user scenarios and experimentally demonstrate the performance of beam alignment to the user's location optimized by the RL algorithm. The achieved results always meet the signal quality requirement specified by the 3rd Generation Partnership Project (3GPP) criterion for 64-QAM orthogonal frequency division multiplexing (OFDM).