Most social circles would consider ivories and different shades of cream to be de riguer, not the faux pas you've been led to believe.
In fact, this season sees designers mixing whimsical ivories and neutrals with rustic colors to create chic and timeless looks. Look into Bottega Veneta for more details, but they pretty much some it up.
Look, if you want to wear white, go with a more muted shade of it. Unless you're a bride, then wear whatever the hell you want. :P
Lots of Love,