We show that a set of optical memories can act as a configurable linear optical network operating on frequency-multiplexed optical states. Our protocol is applicable to any quantum memories that employ off-resonant Raman transitions to store optical information in atomic spins. In addition to the configurability, the protocol also offers favorable scaling with an increasing number of modes where N memories can be configured to implement arbitrary N-mode unitary operations during storage and readout. We demonstrate the versatility of this protocol by showing an example where cascaded memories are used to implement a conditional cz gate.