Approximate natural orbitals are investigated as a way to improve a Monte Carlo configuration interaction (MCCI) calculation. We introduce a way to approximate the natural orbitals in MCCI and test these and approximate natural orbitals from Møller-Plesset perturbation theory and quadratic configuration interaction with single and double substitutions in MCCI calculations of single-point energies. The efficiency and accuracy of approximate natural orbitals in MCCI potential curve calculations for the double hydrogen dissociation of water, the dissociation of carbon monoxide, and the dissociation of the nitrogen molecule are then considered in comparison with standard MCCI when using full configuration interaction as a benchmark. We also use the method to produce a potential curve for water in an aug-cc-pVTZ basis. A new way to quantify the accuracy of a potential curve is put forward that takes into account all of the points and that the curve can be shifted by a constant. We adapt a second-order perturbation scheme to work with MCCI (MCCIPT2) and improve the efficiency of the removal of duplicate states in the method. MCCIPT2 is tested in the calculation of a potential curve for the dissociation of nitrogen using both Slater determinants and configuration state functions.