The device-independent (DI) quantum key distribution (QKD) protocol requires minimal assumptions about the devices and its security relies on the violation of Bell inequalities, making it hard to realize in real world. Semi-device-independent (SDI) QKD protocol confines quantum state within finite dimensional Hilbert space, thus easier for implementation with existing experimental technology. In this paper, we propose a practical SDI prepare-and-measure BB84 protocol. By introducing min entropy for security proof, we obtain a security bound under the practical condition with finite resources. Numerical simulations imply the finite-key effect can not be ignored in the forthcoming SDI QKD experiment.