Fiber-wireless integration has been widely studied as a key technology to support radio access networks in sixth-generation wireless communication, empowered by artificial intelligence. In this study, we propose and demonstrate a deep-learning-based end-to-end (E2E) multi-user communication framework for a fiber-mmWave (MMW) integrated system, where artificial neural networks (ANN) are trained and optimized as transmitters, ANN-based channel models (ACM), and receivers. By connecting the computation graphs of multiple transmitters and receivers, we jointly optimize the transmission of multiple users in the E2E framework to support multi-user access in one fiber-MMW channel. To ensure that the framework matches the fiber-MMW channel, we employ a two-step transfer learning technique to train the ACM. In a 46.2 Gbit/s 10-km fiber-MMW transmission experiment, compared with the single-carrier QAM, the E2E framework achieves over 3.5 dB receiver sensitivity gain in the single-user case and 1.5 dB gain in the three-user case under the 7% hard-decision forward error correction threshold.