Cryptography protects privacy and confidentiality. So, it is necessary to guarantee that the ciphers used are secure and cryptanalysis-resistant. In this paper, a new state recovery attack against the RC4 stream cipher is revealed. A plaintext attack is used in which the attacker has both the plaintext and the ciphertext, so they can calculate the keystream and reveal the cipher's internal state. To increase the quality of answers to practical and recent real-world global optimization difficulties, researchers are increasingly combining two or more variations. PSO and EO are combined in a hybrid PSOEO in an uncertain environment. We may also convert this method to its binary form to cryptanalyze the internal state of the RC4 cipher. When solving the cryptanalysis issue with HBPSOEO, we discover that it is more accurate and quicker than utilizing both PSO and EO independently. Experiments reveal that our proposed fitness function, in combination with HBPSOEO, requires checking 104 possible internal states; however, brute force attacks require checking 2128 states.
Keywords: Particle Swarm Optimization; cryptanalysis; fitness function; hybrid binary optimization; known plaintext attack; stream cipher.