Frazier was born in Chattanooga, Tennessee. His father was James B. Frazier, who served as Governor of Tennessee from 1903 to 1905 and as a United States senator from Tennessee from 1905 to 1911. His mother was Louise Douglas (Keith) Frazier. He was educated in the public schools and Baylor Preparatory School in Chattanooga, Tennessee. He attended the University of Virginia at Charlottesville and graduated from Chattanooga College of Law in 1914. Admitted to the bar in the same year, he began his practice of law in Chattanooga.

During the First World War, Frazier volunteered for service in the United States Army on April 21, 1917, and was discharged as a major in March 1919. Frazier was appointed United States attorney for the eastern district of Tennessee on September 25, 1933, and served until his resignation on April 12, 1948.^[1] He married Elizabeth Hope on March 30, 1939, and they had one daughter, Elizabeth Hope Frazier.^[2]

Elected as a Democrat to the United States House of Representatives from Tennessee, Frazier served from January 3, 1949 to January 3, 1963.^[3] He was an unsuccessful candidate for renomination in 1962 to the Eighty-eighth Congress, and resumed the practice of law in Chattanooga.

He was a signatory to the 1956 Southern Manifesto that opposed the desegregation of public schools ordered by the Supreme Court in Brown v. Board of Education.

Frazier died in Chattanooga, Hamilton County, Tennessee, on October 30, 1978 (age 88 years, 129 days). He is interred at Forest Hills Cemetery, Chattanooga, Tennessee.^[4]

• United States Congress. "James B. Frazier Jr. (id: F000353)". Biographical Directory of the United States Congress.
• James B. Frazier Jr. at Find a Grave
• Frazier & Frazier Papers Archived 2016-08-10 at the Wayback Machine, University of Tennessee, Knoxville Libraries