We propose a quantum secure group communication protocol for the purpose of sharing the same message among multiple authorized users. Our protocol can remove the need for key management that is needed for the quantum network built on quantum key distribution. Comparing with the secure quantum network based on BB84, we show our protocol is more efficient and securer. Particularly, in the security analysis, we introduce a new way of attack, i.e., the counterfactual quantum attack, which can steal information by "invisible" photons. This invisible photon can reveal a single-photon detector in the photon path without triggering the detector. Moreover, the photon can identify phase operations applied to itself, thereby stealing information. To defeat this counterfactual quantum attack, we propose a quantum multi-user authorization system. It allows us to precisely control the communication time so that the attack can not be completed in time.