It should work as you have tried.
What operating system are you using?
Could it be an issue with your mouse?

Another thing you could try is highlighting the wall or other item and hit the delete key on your keyboard.