Yes, you had to take the trail across the street. We had the same question when we did it but luckily went across the street.
We almost gave up, as the summit trail was farther than we expected and there was some confusion following the map in Holmes’ guidebook (something about a stile indicated where there was none, or vice versa). We pressed on and bagged VA as our 5th in 24 hours. When we realized we weren’t going to make Black Mountain until about hour 25, we crashed at a hotel and tagged it the next morning.