product rollback and recovery procedures