Despite the global efforts to mitigate the ongoing COVID-19 pandemic, the disease transmission and the effective controls still remain uncertain as the outcome of the epidemic varies from place to place. In this regard, the province-wise data from Nepal provides a unique opportunity to study the effective control strategies. This is because (a) some provinces of Nepal share an open-border with India, resulting in a significantly high inflow of COVID-19 cases from India; (b) despite the inflow of a considerable number of cases, the local spread was quite controlled until mid-June of 2020, presumably due to control policies implemented; and (c) the relaxation of policies caused a rapid surge of the COVID-19 cases, providing a multi-phasic trend of disease dynamics. In this study, we used this unique data set to explore the inter-provincial disparities of the important indicators, such as epidemic trend, epidemic growth rate, and reproduction numbers. Furthermore, we extended our analysis to identify prevention and control policies that are effective in altering these indicators. Our analysis identified a noticeable inter-province variation in the epidemic trend (3 per day to 104 per day linear increase during third surge period), the median daily growth rate (1 to 4% per day exponential growth), the basic reproduction number (0.71 to 1.21), and the effective reproduction number (maximum values ranging from 1.20 to 2.86). Importantly, results from our modeling show that the type and number of control strategies that are effective in altering the indicators vary among provinces, underscoring the need for province-focused strategies along with the national-level strategy in order to ensure the control of a local spread.