Codebook-based feature encodings are a standard framework for image recognition issues. A codebook is usually constructed by clusterings, such as the k-means and the Gaussian Mixture Model (GMM). A codebook size is an important factor to decide the trade-off between recognition performance and computational complexity and a traditional framework has the disadvantage to image recognition issues when a large codebook; the number of unique clusters becomes smaller than a designated codebook size because some clusters converge to close positions. This paper focusses on the disadvantage from a perspective of the distribution of prior probabilities and presents a clustering framework including two objectives that are alternated to the k-means and the GMM. Our approach is first evaluated with synthetic clustering datasets to analyze a difference to traditional clustering. In the experiment section, although our approach alternated to the k-means generates similar results to the k-means results, our approach is able to finely tune clusters for our objective. Our approach alternated to the GMM significantly improves our objective and constructs intuitively appropriate clusters, especially for huge and complicatedly distributed samples. In the experiment on image recognition issues, two state-of-the-art encodings, the Fisher Vector (FV) using the GMM and the Vector of Locally Aggregated Descriptors (VLAD) using the k-means, are evaluated with two publicly available image datasets, the Birds and the Butterflies. For the results of the VLAD with our approach, the recognition performances tend to be worse compared to the original VLAD results. On the other hand, the FV using our approach is able to improve the performance, especially in a larger codebook size.