Fix a possible infinite loop on unary - that's not a part of a literal.

The behaviour did differ based on contents of unrelated memory...