Joseph Gray was born in Susquehanna Township, Cambria County, Pennsylvania. He graduated from Eastman College at Poughkeepsie, New York, in 1905. He served as a private in Company H, Fifth Infantry, United States Army, from 1900 to 1902 and in the United States Army Signal Corps in 1902 and 1903. After his service in the military, he studied law, was admitted to the bar in 1910 and commenced practice in Ebensburg, Pennsylvania. He was a member of the Pennsylvania State House of Representatives in 1913 and 1914. He served as president of the board of health from 1916 to 1920, and became a motion-picture exhibitor at Spangler, Pennsylvania, in 1920. He was school director of Spangler from 1930 to 1934 and a councilman from 1939 to 1943.

Gray was elected as a Democrat to the 74th and 75th Congresses. He was an unsuccessful candidate for re-election in 1938 and 1940. He resumed the practice of law and also published The Conservative weekly newspaper. He died in Spangler, Pennsylvania, and is buried in Holy Cross Cemetery.

• United States Congress. "Joseph Anthony Gray (id: G000399)". Biographical Directory of the United States Congress.
• The Political Graveyard