I believe this prejudice existed even before the availability of contact lenses and has likely intensified since they became available.

People love the sunnies because people can't see if you are stoned or drunk by your eyes, people cant see where you are looking (great on a nude beach), and if people can't make direct contact with your peepers they don't know if you're lying to them. It's all about the eyes, windows to your soul, everyone wants to protect theirs.