There can be several reasons. I know a few spots where I rarely get a fish, but when I do, the average size is considerably larger. The dream of the fish of a lifetime, caught on that very special spot, is generally what gets me there, but there's something else too... These places seem to have a common denominator - they're spectacular in their own, Danish, low key way. Places where I can usually fish alone, and one particular place haunts me and pulls me back every now and then, only because I one summer night some 20 years ago lost a gigantic sea trout or salmon (on light spinning tackle BTW :-). I've been back maybe 30 times since then - and did catch a few fish (one of them a decent 6-7 pounder), but never saw or heard anything of the mythical size of that beast I lost a long time ago.
Another place I visit quite a lot draws me becuase it's just a beautiful place, and also where I learned about saltwater fishing a long time ago. The fishing's not particularly good, and never yields more than one or two fish (except that summer night maybe 10 years ago, where I got 3 and the smallest was a 6-pound beauty). The picture is part of the view from this place.
There's a stretch on River Varde that I really enjoy fishing, mainly because it's challening fishing through a forest, and if you can't speycast a fast sinking line well, you're basically f....d. There are maybe only 5 decent spots where you can fish 25-50 meters of rivers down through the forest on a stretch which is maybe 5km long. But the challenging fishing is fun, and rewarding when you get a salmon or sea trout, and furthermore, the magnificent little king fisher is common there.
There's a small lake I sometimes go to. I've never caught anything but really small pike, but a friend landed a 13kg+ pike there a few years ago (on dead bait), and I'm relatively sure it's still there - and I'd really like to catch it. Lots of wasted hours so far, but I'll give it a few more days next spring.
There are also places I visit, beacuse the fishing's easy, and beacuse I know the chances of fish are very good - sometimes I just a need to boost my self confidence :-). Like a small gravel-pit-trout-lake, where I just know there are fish, and I just know that getting them is a matter of changing tactics 2-3 times, and then you hit them. The learning curve is steep in places like this, once you accept that when you're not getting them on intermediates and buzzers within an hour, it's time to do something else. And once you get to where you jusy know that it's either this or that tactic that'll nail them, it's time to start doing something elase to see if other tactics also might work, and learn more.
Have a great weekend!