In terahertz (THz) and far-infrared (FIR) spectroscopic measurements, weak absorption spectral features due to small quantities of test sample can be masked by undesirable etalon fringe artifacts caused by multiple reflections within a pellet or a rigid sample holder. A double-layered nitrocellulose (NC) membrane structure is proposed in this paper as an alternative holder for small quantities of either dry or wet pure (no added polyethylene powder) samples with significantly reduced etalon artifacts. Utilizing a THz time-domain spectroscopy system and a synchrotron source, we demonstrate the performance of the NC structure across the THz/FIR spectrum, benchmarking against pellets holding similarly small quantities of α-lactose powder either with or without different grades of polyethylene powder. With only pure samples to consider, scattering can be mitigated effectively in NC-derived spectra to reduce their baselines.