Ashford was born in Los Angeles, California. On February 20, 1951 he became an umpire in the Southwestern International League, the first black professional umpire. He went on to umpire in the Pacific Coast League for many years before being hired by the American League in 1961, and on April 11966 he became the first black major league umpire. He was an outfield umpire for the 1967 All-Star game, and worked in the 1970 World Series. He remained an umpire until mandatory retirement upon turning 56 in 1970.
He brought a new style to being an umpire. He dressed impeccably, wore jewelry, including flashy cuff links, and exaggerated his calls with gestures. While some observers believed that his race prevented him from working in the majors earlier than he did, others maintained that his flashy style actually delayed his major league debut due to general disdain for umpires to draw attention to themselves.
Ashford died of a heart attack at age 65 in Venice, a suburb of Los Angeles.