To investigate feasible treatment planning parameters, we aimed to evaluate the dosimetric and radiobiological impact of the dose calculation algorithm and grid size in the volumetric modulated arc therapy (VMAT) plan for prostate cancer. Twenty patients were selected, and the treatment plans were initially generated with anisotropic analytical algorithm (AAA) and recalculated with Acuros XB (AXB) algorithm. Various dose grids were used for AXB (1, 2, and 3 mm) and AAA (1, 3, and 5 mm) plan. Dosimetric parameters such as homogeneity index (HI) and conformity index (CI), and radiobiological parameters such as tumor control probability (TCP) and normal tissue complication probability (NTCP) were calculated. Significant differences were observed in the planning target volume (PTV) coverage between both algorithms, and the V95%, HI, and CI of AAA were significantly affected by grid (p < 0.01). On 1 mm grid, the mean rectal dose difference between both algorithms was 2.87% of the prescription dose (p < 0.01), which was the highest among the critical organs. The TCP and NTCP of the AAA were higher than those of AXB (p < 0.01). Compared to AXB with 1 mm grid, the 2 mm grid showed comparable dose calculation accuracy with short calculation time. This study found that the PTV and rectum show significant differences according to dose calculation algorithm and grid. Considering the dose calculation performance for heterogeneous area, we recommend AXB with 2 mm grid for improving treatment efficiency of prostate VMAT.