Ordinary differential equations (ODEs) provide a powerful formalism to model molecular networks mechanistically. However, inferring the model structure, given a set of time course measurements and a large number of alternative molecular mechanisms, is a challenging and open research question. Existing search heuristics are designed only for finding a single best model configuration and cannot account for the uncertainty in selecting the network components. In this study, we present a novel Markov chain Monte Carlo approach for performing Bayesian model structure inference over ODE models. We formulate a Metropolis algorithm that explores the model space efficiently and is suitable for obtaining probabilistic inferences about the network structure. The method and its special parallelization possibilities are demonstrated using simulated data. Furthermore, we apply the method to a time course RNA sequencing data set to infer the structure of the transiently evolving core regulatory network that steers the T helper 17 (Th17) cell differentiation. Our results are in agreement with the earlier finding that the Th17 lineage-specific differentiation program evolves in three sequential phases. Further, the analysis provides us with probabilistic predictions on the molecular interactions that are active in different phases of Th17 cell differentiation.