test: improve code in test-console-instance

* use common.mustCall to validate functions executions
* use common.fail to check test fail
* improve error validations
* remove unnecessary assertions
* use arrow functions

PR-URL: https://github.com/nodejs/node/pull/10813


Reviewed-By: James M Snell <jasnell@gmail.com>
Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>