McCoy is one of the most important harmonica players in country music history. Born in Oak Hill (Fayette County), he moved to Florida as a child but stayed connected to his home. He moved to Nashville in 1959 and played on numerous recordings, including his own instrumental albums.

McCoy also played drums, trumpet, and bass, and performed with rock bands. He recorded with stars such as Roy Orbison and Bob Dylan and was the musical director for the TV show Hee Haw.

He was inducted into the West Virginia Music Hall of Fame in 2008, the Country Music Hall of Fame in 2009, and the Grand Ole Opry in 2022.