We report high-sensitivity and high-temperature resolution experimental data for the temperature dependence of the optical birefringence of a nonpolar monolayer smectogen 4-butyloxyphenyl-4'-decyloxybenzoate (10[over ¯].O.4[over ¯]) liquid crystal by using a rotating-analyzer technique. The birefringence data cover nematic and smectic-A phases of the 10[over ¯].O.4[over ¯] compound. The birefringence data are used to probe the temperature behavior of the nematic order parameter S(T) in the vicinity of both the nematic-isotropic (N-I) and the nematic-smectic-A (N-SmA) transitions. For the N-I transition, from the data sufficiently far away from the smectic-A phase, the average value of the critical exponent β describing the limiting behavior of S(T) is found to be 0.2507±0.0010, which is in accordance with the so-called tricritical hypothesis, which predicts β=0.25 and excludes higher theoretical values. The critical behavior of S(T) at the N-I transition is discussed in detail by comparing our results with the latest reports in the literature and we conclude that by comparing with the previously reported results, the isotropic internal field assumption by the Vuks-Chandrasekhar-Madhusudana model is adequate to extract the critical behavior of S(T) from the optical birefringence data. We observe that there is no discontinuous behavior in the optical birefringence, signaling the second-order nature of the N-SmA transition. The effect of the coupling between the nematic and smectic-A order parameters on the optical birefringence near the N-SmA transition is also discussed. In a temperature range of about 4K above and below the N-SmA transition, the pretransitional evidence for the N-SmA coupling have been detected. From the analysis of the optical birefringence data above and below the N-SmA transition by means of various fitting expressions we test the validity of the scaling relation λ=1-α between the critical exponent λ describing the limiting behavior of the nematic order parameter and the specific heat capacity exponent α. We then show that the temperature derivative of the nematic order parameter S(T) near T(NA) exhibits the same power-law divergence as the specific heat capacity with an effective critical exponent of 0.2303±0.0035.