This reverts commit 6939a5ac. Daniel Vetter supplied a set of fixes for all the module unload bugs he could trigger on his machines, so let the fun recommence!