Object-based simultaneous localization and mapping (SLAM) is a more natural and robust way for agents to interact with their surrounding environment. However, it introduces a problem of semantic objects association. Correct object association is the key factor to achieve a successful object SLAM system because object association and SLAM are inherently coupled and have not been well tackled yet. A novel formulation of the object association problem based on a hierarchical Dirichlet process (HDP) is proposed. Through the HDP, we can hierarchically associate the grouped object measurements. This can improve the object association accuracy and computation efficiency. Thanks to the novel formulation, the proposed method is also able to correct failure object associations according to its sampling inference algorithm. Furthermore, we introduce object poses to the processing of pose optimization. The object association and pose optimization are then solved in a tightly coupled way, by which both aspects can promote each other. The proposed method is evaluated on indoor and outdoor datasets and the experimental results show a very impressive improvement with respect to the traditional SLAM.