Born in Gettysburg in the Province of Pennsylvania to the former Maria Reed and her husband (also) James Stephenson, the family moved to Martinsburg, Berkeley County, Virginia (now West Virginia) by 1790. His elder brother William Stephenson would move to Knoxville, Tennessee, as would their mother and sisters after their father's death in 1804. His younger brother Benjamin Stephenson (1769-1822) would move to Kentucky, then the Illinois Territory, where he operated a general store and also served as a sheriff and militia leader before becoming a Congressman for the new state.

On May 17, 1792, Stephenson married Ann Cunningham in Berkeley County, but records remain only of their youngest children, who remained in the area: George Benjamin Stephenson (a physician, who inherited the farm and moved to Washington D.C. during or after the American Civil War) and his sister Anne Stephenson Beckham (her husband, Harpers Ferry mayor Fontaine Beckham, died in John Brown's raid, and his death as well as the treason conviction led to the resulting executions).

Young James Stephenson volunteered to serve in Virginia's militia, and he led a company under General Arthur St. Clair in his expedition against Native Americans in the Ohio River Valley. He survived the ignominious St. Clair's Defeat in 1791, and, three years later, helped lead forces sent to quell the Whiskey Rebellion. At some point, Stephenson rose to the rank of Major, as well as Brigade inspector.

Stephenson served as a local magistrate and was a farmer and slaveholder. He owned two slaves in 1810,^[2] six slaves in 1820 (half of them children),^[3] and four slaves in the 1830 census (all adults, one male and one female older than 55 years).^[4] Voters elected Stephenson as one of their delegates (a part-time position) in the Virginia House of Delegates in 1800–1803, and again in 1806 and 1807.

Between those terms, Stephenson was elected as a Federalist to the Eighth Congress (March 4, 1803 – March 3, 1805) with 53.57% of the vote, defeating Democratic-Republican Osburn Sprigg.

Stephenson was again elected as Federalist to the Eleventh Congress (March 4, 1809 – March 3, 1811) with 57.59% of the vote, defeating Democratic-Republican John Marrow.

Stephenson was elected as a Federalist to the Seventeenth Congress to fill the vacancy caused by the death of Thomas Van Swearingen.

He was reelected as a Crawford Federalist to the Eighteenth Congress and served from October 28, 1822, to March 3, 1825.

Stephenson died in Harpers Ferry (now West Virginia) on August 27, 1833. He is buried at Edge Hill Cemetery in Charles Town; a memorial was also erected to commemorate his widow, who died on January 1, 1865.

  This article incorporates public domain material from the Biographical Directory of the United States Congress