cirrus: avoid crash if driver fails to load

If we haven't inited the mm code, don't try and tear it down.

Signed-off-by: Dave Airlie <airlied@redhat.com>