In vivo direct drug targeting aims at delivering drug molecules loaded on microrobots to the diseased site using the shortest possible physiological routes, which potentially improves targeting efficiency and reduces systemic toxicity. It is thus essential to consider realistic in-body limitations for direct drug targeting applications. Here, we present a novel controller for microrobot maneuver by considering four key in vivo constraints: non-Euclidean structure of capillaries, irreversibility of blood flow, invisibility of microvasculature, and inaccuracy of microrobot tracking. We use the taxicab geometry of capillaries as the a priori knowledge for steering and tracking a microrobot in lattice-like vessels. Furthermore, we introduce a minimax repulsive boundary function to prevent the microrobot from getting too close to the boundaries imposed by the direction of blood flow. We also propose a novel Kalman filtering algorithm to reduce tracking error, while avoiding possible obstacles such as vessel walls without knowing their actual locations. The proposed control method consists of four modules, namely a model predictive control module for tumor targeting, a Kalman filtering module for microrobot tracking, a blind obstacle detection module, and a vessel structure estimation module. The interplay of these four modules offers successful maneuver and tracking of the microrobot while avoiding obstacles in a blind manner by utilizing the taxicab geometry of blood vessels. We present a comprehensive in silico simulation study to verify our designed controller.