We analyze surface plasmon resonance-based fiber-optic sensor for sensing of small concentrations of hydrogen gas in the visible region of the electromagnetic spectrum. One of the two probes considered has multilayers of zinc oxide (ZnO) and palladium (Pd) while the other has layer of their composite over a silver coated unclad core of the fiber. The analysis is carried out for different volume fractions of palladium nanoparticles dispersed in zinc oxide host material in the nanocomposite layer. For the analysis, a Maxwell-Garnett model is adopted for calculating the dielectric function of a ZnO:Pd nanocomposite having nanoparticles of dimensions smaller than the wavelength of radiation used. The effects of the volume fraction of the nanoparticles in the nanocomposite and the thickness of the nanocomposite layer on the figure of merit of the sensor have been studied. The film thickness of the layer and the volume fraction of nanoparticles in the ZnO:Pd nanocomposite layer have been optimized to achieve the maximum value of the figure of merit of the sensor. It has been found that the figure of merit of the sensing probe coated with ZnO:Pd nanocomposite is more than twofold of the sensing probe coated with multilayers of Pd and ZnO over a silver coated unclad core of the fiber; hence, the sensor with a nanocomposite layer works better than that with multilayers of zinc oxide and palladium. The sensor can be used for online monitoring and remote sensing of hydrogen gas.