Gravar-mail: Importance of matrix inelastic deformations in the initial response of magnetic elastomers