Causal discovery from observational data is an important but challenging task in many scientific fields. A recent line of work formulates the structure learning problem as a continuous constrained optimization task using an algebraic characterization of directed acyclic graphs (DAGs) and the least-square loss function. Though the least-square loss function is well justified under the standard Gaussian noise assumption, it is limited if the assumption does not hold. In this work, we theoretically show that the violation of the Gaussian noise assumption will hinder the causal direction identification, making the causal orientation fully determined by the causal strength as well as the variances of noises in the linear case and by the strong non-Gaussian noises in the nonlinear case. Consequently, we propose a more general entropy-based loss that is theoretically consistent with the likelihood score under any noise distribution. We run extensive empirical evaluations on both synthetic data and real-world data to validate the effectiveness of the proposed method and show that our method achieves the best in structure Hamming distance, false discovery rate (FDR), and true-positive rate (TPR) matrices.