The ongoing search for fast Li-ion conducting solid electrolytes has driven the deployment surge on density functional theory (DFT) computation and materials informatics for exploring novel chemistries before actual experimental testing. Existing structure prototypes can now be readily evaluated beforehand not only to map out trends on target properties or for candidate composition selection but also for gaining insights on structure-property relationships. Recently, the tavorite structure has been determined to be capable of a fast Li ion insertion rate for battery cathode applications. Taking this inspiration, we surveyed the LiMTO4F tavorite system (M(3+)-T(5+) and M(2+)-T(6+) pairs; M is nontransition metals) for solid electrolyte use, identifying promising compositions with enormously low Li migration energy (ME) and understanding how structure parameters affect or modulate ME. We employed a combination of DFT computation, variable interaction analysis, graph theory, and a neural network for building a crystal structure-based ME prediction model. Candidate compositions that were predicted include LiGaPO4F (0.25 eV), LiGdPO4F (0.30 eV), LiDyPO4F (0.30 eV), LiMgSO4F (0.21 eV), and LiMgSeO4F (0.11 eV). With chemical substitutions at M and T sites, competing effects among Li pathway bottleneck size, polyanion covalency, and local lattice distortion were determined to be crucial for controlling ME. A way to predict ME for multiple structure types within the neural network framework was also explored.