They don’t hurt you or say anything unkind or steal from you, right? This doesn’t feel like harassment really, but it sounds like they were up to something. In general I would say trust your gut, if you see people ahead of you who look shady, avoid them and walk the other way. I’ve had bad experiences where I didn’t do that and should have. If someone speaks to you, you are usually best to just keep going. In this case I think you did the right thing. An assertive “can I help you,” is maybe the only thing you could have said. But really, best not to engage.