If the only place you're using the visitor is within the cpp file, then you just put it where it's being used. If you're going to be using the same visitor in multiple sources, it has to be in an hpp - unless you want to #include a cpp file, but that's not done very often.

As for the rest... it looks okay, but I only really gave a cursory glance. Property maps are (almost) always passed by value, so you don't need to worry about keeping references to them.