Question about Gaggia New Classic

you mean the metal waste water hose (at least that's what it's called in the manual), not the steam/hot water lance with rubber protection. If so, this one held and has held since I bought the machine, I didn't have to do anything special. Just plug it in and off it goes. Looks like something is defective on your machine.

It has to hold. No problem for me. Push it in with a little pressure. Overnight and when not in use for a longer period of time, I remove it and the water tank (for drying).

That's right, just plug it in. If it doesn't hold, something is defective.