Domestic species provides a powerful model for examining genetic mechanisms in the evolution of yield traits. The domestic silkworm (Bombyx mori) is an important livestock species in sericulture. While the mechanisms controlling cocoon yield are largely unknown. Here, using B. mori and its wild relative B. mandarina as intercross parents, 100 BC1 individuals were sequenced by restriction site-associated DNA sequencing (RAD-Seq). The linkage map contained 9,632 markers was constructed. We performed high-resolution quantitative trait locus (QTL) mapping for four cocoon yield traits. A total of 11 QTLs were identified, including one yield-enhancing QTL from wild silkworm. By integrating population genomics and transcriptomic analysis with QTLs, some favourable genes were revealed, including 14 domestication-related genes and 71 differentially expressed genes (DEGs) in the fifth-instar larval silk gland transcriptome between B. mori and B. mandarina. The relationships between the expression of two important candidate genes (KWMTBOMO04917 and KWMTBOMO12906) and cocoon yield were supported by quantitative real-time PCR (qPCR). Our results provide some new insights into the molecular mechanisms of complex yield traits in silkworm. The combined method might be an efficient approach for identifying putative causal genes in domestic livestock and wild relatives.