Non-linear interactions in coherent gases are not only at the origin of bright and dark solitons and superfluids; they also give rise to phenomena such as multistability, which hold great promise for the development of advanced photonic and spintronic devices. In particular, spinor multistability in strongly coupled semiconductor microcavities shows that the spin of hundreds of exciton-polaritons can be coherently controlled, opening the route to spin-optronic devices such as ultrafast spin memories, gates or even neuronal communication schemes. Here we demonstrate that switching between the stable spin states of a driven polariton gas can be controlled by ultrafast optical pulses. Although such a long-lived spin memory necessarily relies on strong and anisotropic spinor interactions within the coherent polariton gas, we also highlight the crucial role of non-linear losses and formation of a non-radiative particle reservoir for ultrafast spin switching.