Actually it was Groucho who uttered the phrase to which you refer. However a little research shows that it was apparently originally from the "Forsyte Saga" novels about edwardian era upper-middle-class british life. You learn something new every day.

http://en.wikiquote.org/wiki/Groucho_Marx