Secret entrances are often inconvenient. For most day to day activities there will be an easier route to use. Once the coolness of the secret entrance wears off people will use whatever is most convenient. Since this entrance is designed to be hidden and it wouldn't be much of a secret if everyone knew about it, all it takes is a small number of people to forget about the path they never use for it to become abandoned.
