A two-dimensional signal constellation scheme for binary uniform memoryless source transmission in optical fiber channels is studied in this paper. In geometric shaping (GS), optimization algorithms are usually used to change the overall position of constellation points while maintaining the probability of constellation points unchanged. Different optimization functions are used to allocate the position of constellation symbols, thereby improving constellation performance. A 16 quadrature amplitude modulation (QAM) optical signal generation scheme based on weighted optimal Euclidean distance is proposed in this paper. In order to obtain the best constellation diagram and increase the shaping gain, the weighted optimal Euclidean distance that can minimize the bit error rate (BER) over multiple iterative optimizations is used as the objective function. On the one hand, the proposed 16QAM optical signal generation scheme based on weighted optimal Euclidean distance always outperforms the uniform square 16QAM and the uniform circle 16QAM schemes in the back to back (BTB) transmission. On the other hand, after analyzing the simulation demonstration in a 50GBaud coherent optical communication system over 3000 km, results demonstrate that the optical signal to noise ratio (OSNR) performance of this system is better than that of the uniform square 16QAM and the uniform circle 16QAM, which is improved by 0.52 dB and 0.85 dB, respectively. In addition, the proposed 16QAM system increases the transmission distance by 989 km and 741 km, respectively, compared to the other two systems. The performance confirms that the proposed novel 16QAM scheme, to the best of our knowledge, can effectively improve the reliability and transmission distance. Therefore, the proposed scheme has a certain development prospect in the future long-distance transmission of high-speed optical fiber communication.