The purpose of this research was to investigate the influence of the glyoxylic acid (GA) modification of hydroxyapatite (HAP) nanofibers on their dispersion in bisphenol A glycidyl methacrylate (BisGMA)/triethylene glycol dimethacrylate (TEGDMA) dental composites and also to investigate the mechanical properties, water absorption and water solubility of the resulting dental resins and composites. Scanning/transmission electron microscopy images showed that microsized HAP nanofiber bundles could be effectively broken down into individual HAP nanofibers with an average length of ∼15 µm after the surface modification process. Fourier transform infrared spectroscopy, x-ray photoelectron spectroscopy and thermal gravimetric analysis characterization confirmed that GA was chemically grafted on the HAP nanofiber surface, hypothetically by reacting with the amine group on the HAP nanofiber surface. The enhanced dispersion of HAP nanofibers in the dental matrix led to increased biaxial flexural strength (BFS) compared with the corresponding dental resins and composites filled with untreated HAP nanofibers. In addition, impregnation of small mass fractions of the GA-modified HAP nanofibers into the BisGMA/TEGDMA dental resins (5 wt%, 10 wt%) or composites (2 wt%, 3 wt%) could also substantially improve the BFS in comparison with the controls (pure resins or dental composites filled with silica particles alone). Larger mass fractions could not increase the mechanical property further or even degraded the BFS values. Water behavior testing results indicated that the addition of the GA-modified HAP nanofibers resulted in higher water absorption and water solubility values, which are not preferred for clinical application. In summary, well-dispersed HAP nanofibers and their dental composites with enhanced mechanical properties have been successfully fabricated, but the water absorption and water solubility of such dental composites need to be further improved.