I'm a total noob to the visual novel scene, but it looks like there are too many good ones to count. I played the Phoenix/Layton crossover game on the 3DS but that's about it.
I have only played 3 visual novels. Umineko no Naku Koro ni is my favorite one. It also involves lots of fun characters...that may die at any point...multiple times.
I've read the first Chapter/Episode of Umineko, through I'm aware of the direction it takes later on. Totally my kind of story and I think it's awesome so far. I'm excited about reading more of it. It's just...SOOOO LOOOONG. I haven't been able to find the time to get through it at a consistent pace. Its soundtrack is ridiculously good though, to the point where I've put it on my phone to use as standalone listening music. Apparently they're going to rerelease it in English on Steam with updated assets. I'll be all over it once that happens.
Just a heads up about Umineko: its story is independent of Higurashi no Naku Koro Ni (and has a completely different feel), but it spoils some aspects of Higurashi. If you can, get through that story first (either through VN, manga, or anime) because you might end up liking Higurashi a lot.
Speaking of mystery stories, if I had a PSVita I would be all over the Danganronpa series, but alas I do not have one (as far as handhelds go, my 3DS is already starting to collect dust). I instead chose to read through a Let's Play of the first game. Really fun stuff. I'm in the middle of watching a Let's Play of Danganronpa 2. I'm liking the second game even more. I feel the individual mysteries in the second game strike a much better balance between challenge and fairness.
They're making a 3rd Danganronpa on PSVita and PS4. Hrm, a PSVita is looking more tempting now.
Once I get through those games, I want to check out Muv-Luv+extra+alternative. Oddly enough, Muv-Luv:Alternative is the core game and the stuff before it is like prologue. I think a good way to descibe M-L:A would be to say it's what you might get if you crossed Attack on Titan, Gundam, and maybe Topgun. In fact, I think M-L:A was a major source of inspiration for Attack on Titan. I'm a fan of mechas in general, as well as AoT, so I'm really hoping I'll enjoy this series.