The blue hell is the blue void beneath the game world in GTA. You're not supposed to be able to enter the blue hell, but due some bugs it is possible to reach it (both on purpose and by accident). Traveling through the blue hell makes it possible to acces certain areas in te game you're not supposed to reach, like Ghost Town or the research section of Area 69. The blue hell should not be confused with the hidden interiors universe.
How does it work?
Picture the world of a GTA as a large rectanguluar area, floating in a void. Under the surface of the game area isn't solid ground, but a large empty space. This is what we call the blue hell. You're not supposed to be able to enter this void. However, due to some bugs like unsolid surfaces, it it possible to 'fall' through the ground, into the blue hell. You'll keep falling untill the game realises you're to low and you shouldn't be there, and spawns you back on the surface. If you can prevent going to low (for example, by using a helicopter or jetpack, or walking on the Porter Tunnel), you can use the blue hell to get to places that are usually out of reach.
De name 'blue hell' is derived from the blue color of the hell in GTA III (The blue hells of the other GTA's aren't neccesarely blue). The easiest way of entering the blue hell in GTA III is by entering the subway in Shoreside Vale through the first door of the train, and exitting it before the train leaves.
It is also possible to fall into the blue hell above the entrance to the Porter Tunnel in Portland. This allowes you to acces Staunton Island and Shoreside Vale before the Callahan Bridge is repaired.
GTA Vice City
Helicopters make it fairly easy to explore the blue hell of GTA Vice City, although there aren't many interesting locations in the blue hell of this GTA. You'll find Howlin' Pete's Bike Emporium, an unused interior opposite the Greasy Chopper, and a out-of-place printing machine used in a cutscene of the Printing Studios.
GTA San Andreas
The blue hell of GTA San Andreas is very easy to reach and explore due to the jetpack. There are several interesting interiors, including the underground research section of Area 69. The blue hell and the normal game world are called 'Heaven K'.