test: Added more validations to setEncoding

* Added more strict validations to properly test setEncoding
* Changed all var usage to const or let
* Changed assert.equal to assert.strictEqual

PR-URL: https://github.com/nodejs/node/pull/9997


Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: Gibson Fahnestock <gibfahn@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>