Rigorously evaluating acupuncture and other complementary and alternative medicine (CAM) therapies presents researches with many challenges. The failure to satisfactorily address these challenges has resulted in poorly designed studies, which yield findings that are difficult to interpret. Despite the publication of more than 10 randomized trials evaluating acupuncture as a treatment for chronic low back pain, the efficacy and effectiveness of acupuncture for this common problem remain unclear. We discuss the rationale for and design of a five-arm randomized controlled pilot clinical trial that addresses the major methodological shortcomings of previous studies (e.g., poorly justified treatment and control groups and lack of masking) and that lays the groundwork for a full scale trial evaluating acupuncture as a treatment for chronic low back pain. Although development and pilot testing of these design features required substantial time and resources, we believe that this investment in rigorous groundwork is essential to ensure that full-scale trials evaluating CAM treatments produce valid and interpretable results.