At this time, you can remove color only with an undo operation.
But the default color is just white with a dark grey ambiant color. If you choose a grey color, you should obtain the same effect.

In case a search also leads someone else here looking for an answer to 'How do I reset the wall colour back to default?'. My work around is to click the 'Texture' button (but don't set any texture) and then 'OK'.