Born in Washington, Connecticut, Whittlesey moved with his parents in early youth to Salisbury, Connecticut. He attended the common schools at Danbury, and studied law there. He was admitted to the bar of Fairfield County and practiced in Danbury and Fairfield County. He also practiced in New Milford, Connecticut, in 1805. He moved to Canfield, Ohio, in 1806, where he practiced law and taught school. He served as prosecuting attorney of Mahoning County. He served as military and private secretary to Gen. William Henry Harrison and as brigade major in the Army of the Northwest in the War of 1812. He served as member of the Ohio House of Representatives in 1820 and 1821.

Whittlesey was elected to the Eighteenth through Twenty-second Congresses, elected as an Anti-Masonic candidate to the Twenty-third Congress, and elected as a Whig to the Twenty-fourth and Twenty-fifth Congresses and served from March 4, 1823, to July 9, 1838, when he resigned. He was one of the founders of the Whig Party. He served as chairman of the Committee on Claims (Twenty-first through Twenty-fifth Congresses). He was Sixth Auditor of the Treasury from March 18, 1841, until December 18, 1843,^[1] when he resigned and resumed the practice of law in Canfield. He was appointed general agent of the Washington Monument Association in 1847. He was appointed by President Zachary Taylor as First Comptroller of the Treasury and served from May 31, 1849, to March 26, 1857, when he was removed by President James Buchanan.^[2] He was reappointed by President Abraham Lincoln April 10, 1861, and served until his death in Washington, D.C., January 7, 1863. He was interred in the Canfield Village Cemetery, Canfield, Ohio.

He was an uncle of William Augustus Whittlesey and Charles Whittlesey, and a cousin of Frederick Whittlesey and Thomas Tucker Whittlesey.

• United States Congress. "Elisha Whittlesey (id: W000431)". Biographical Directory of the United States Congress. Retrieved on 2009-5-18
• Wilson, J. G.; Fiske, J., eds. (1889). "Whittlesey, Elisha" . Appletons' Cyclopædia of American Biography. New York: D. Appleton.

  This article incorporates public domain material from the Biographical Directory of the United States Congress