To improve the performance of fee-space optical communication systems, this paper analyzes the performance of a relay-aided hybrid fee-space optical (FSO)/radio frequency (RF) cooperation system based on a selective combination and decoding forward transmission scheme. In this system, the FSO sub-link experienced Málaga turbulence with pointing errors and the RF sub-link suffered Nakagami-m fading. Firstly, the probability density function (PDF) and cumulative distribution function (CDF) of the end-to-end output signal-to-noise ratio (SNR) of the relay-aided hybrid FSO/RF system are derived. Then, using the extended generalized bivariate Meijer's G-function (EGBMGF) and the approximate analytical formula of the generalized Gauss-Laguerre integral, mathematical expressions of the end-to-end average bit error rate (ABER) and outage probability of the relay-aided hybrid FSO/RF system with different subcarrier intensity modulation and different detection schemes are derived. Through a simulation analysis of the system, the results show that compared with the other three modulation technologies, the hybrid FSO/RF direct link and relay-aided hybrid FSO/RF system with coherent binary phase shift keying (CBPSK) modulation have the best bit error performance. Compared with direct detection, the hybrid direct link and relay-aided hybrid system with coherent detection can significantly improve the communication performance. Increasing the RF fading parameter m can further improve the bit error and outage performance of the hybrid direct link and relay-aided hybrid system; the hybrid direct link can significantly mitigate the degradation of communication performance in the FSO system caused by pointing errors, and the relay-aided hybrid system can further improve the communication performance; under weak turbulence conditions, the impact of pointing errors on the performance of the relay-aided hybrid system can even be ignored. The greater the total number of paths in the relay-aided hybrid system, the better the communication performance of the system; however, the more hops, the worse the performance of the system. The outage probability of the hybrid direct link and relay-aided hybrid system are very sensitive to the decision threshold, and the larger the decision threshold, the worse the outage performance. The transmission distance of different hybrid direct links has little impact on the performance of hybrid direct links and relay-aided hybrid systems. Improving the signal-to-noise ratio of RF sub-links significantly improves the performance of hybrid direct links and relay-aided hybrid systems.
Keywords: Nakagami fading; atmospheric turbulence; pointing errors; relay-aided hybrid FSO/RF system.