Lithium niobate (LN) is the workhorse for modern optoelectronics industry and nonlinear optics. High quality (Q) factor LN microresonators are promising candidates for applications in optical communications, quantum photonics, and sensing. However, the phase-matching requirement of traditional evanescent coupling methods poses significant challenges to achieve high coupling efficiencies of the pump and signal light simultaneously, ultimately limiting the practical usefulness of these high Q factor LN resonators. Here, for the first time, to the best of our knowledge, we demonstrate deformed chaotic LN microcavities that feature directional emission patterns and high Q factors simultaneously. The chaotic LN microdisks are created using conventional semiconductor fabrication processes, with measured Q factors exceeding 106 in the telecommunication band. We show that our devices can be free-space-coupled with high efficiency by leveraging directional emission from the asymmetric cavity. Using this broadband approach, we demonstrate a 58-fold enhancement of free-space collection efficiency of a second harmonic generation signal, compared with a circular microdisk.