To investigate sepal/petal/lip formation in Oncidium Gower Ramsey, three paleoAPETALA3 genes, O. Gower Ramsey MADS box gene5 (OMADS5; clade 1), OMADS3 (clade 2), and OMADS9 (clade 3), and one PISTILLATA gene, OMADS8, were characterized. The OMADS8 and OMADS3 mRNAs were expressed in all four floral organs as well as in vegetative leaves. The OMADS9 mRNA was only strongly detected in petals and lips. The mRNA for OMADS5 was only strongly detected in sepals and petals and was significantly down-regulated in lip-like petals and lip-like sepals of peloric mutant flowers. This result revealed a possible negative role for OMADS5 in regulating lip formation. Yeast two-hybrid analysis indicated that OMADS5 formed homodimers and heterodimers with OMADS3 and OMADS9. OMADS8 only formed heterodimers with OMADS3, whereas OMADS3 and OMADS9 formed homodimers and heterodimers with each other. We proposed that sepal/petal/lip formation needs the presence of OMADS3/8 and/or OMADS9. The determination of the final organ identity for the sepal/petal/lip likely depended on the presence or absence of OMADS5. The presence of OMADS5 caused short sepal/petal formation. When OMADS5 was absent, cells could proliferate, resulting in the possible formation of large lips and the conversion of the sepal/petal into lips in peloric mutants. Further analysis indicated that only ectopic expression of OMADS8 but not OMADS5/9 caused the conversion of the sepal into an expanded petal-like structure in transgenic Arabidopsis (Arabidopsis thaliana) plants.