Meaning: Mildly unwell; not in one's usual health or state of mind.

Origin

Since at least the 17th century 'sorts' has been the name of the letters used by typographers. This usage is referred to in Notes on a Century of Typography at the University Press Oxford 1693–1794 and is nicely defined in Joseph Moxon's Mechanick Exercises, or the Doctrine of Handy-works - Printing, 1683:

"The Letters... in every Box of the Case are... called Sorts in Printers and Founders Language; Thus a is a Sort, b is a Sort."

For sets of type blocks to be 'out of sorts' would clearly be unwelcome to a typesetter. That terminology could be the source of the phrase and the notion is certainly a tempting one. We need to be cautious with that attribution however as the above citation is pre-dated by one from The proverbs, epigrams, and miscellanies of John Heywood, 1562, which makes no explicit mention of typesetting:

Fit (adj): disordered, out of sorts

That unusual definition of 'fit' is, oddly, almost the opposite of our current usage of the word as an adjective.

It may well turn out that pre-1562 citations that refer to typesetting will be found; after all, Gutenberg invented the printing press in around 1440. Until then, I'm sure that many people will opt to believe that 'out of sorts' derives from typesetting. All I can do is present you with the evidence as I find it and let you come to your own conclusions.

http://www.straightdope.com