Mobile phones offer an excellent low-cost alternative for Virtual Reality. However, the hardware constraints of these devices restrict the displayable visual complexity of graphics. Image-based rendering techniques arise as an alternative to solve this problem, but usually, the support of collisions and irregular surfaces (i.e., any surface that is not flat or even) represents a challenge. In this work, we present a technique suitable for both virtual and real-world environments that handle collisions and irregular surfaces for an image-based rendering technique in low-cost virtual reality. We also conducted a user evaluation for finding the distance between images that presents a realistic and natural experience by maximizing the perceived virtual presence and minimizing the cybersickness effects. The results prove the benefits of our technique for both virtual and real-world environments.