Yes, it can be difficult. The way is there, I had to test it myself to make sure it was possible. You have to explore every place you can. I think at one point at one of the bottom of the ladders there is a place in the shadows to the side that is very small to fit through that is the right way. Eventually you'll head down a very deep shaft to the bottom, and somewhere down there is a door that leads to a long hall. Should be easy from there.