Those cavemen could have co-existed with the civilization that built the structures, or simply found the structures useful without knowing the builders. Or, the builders of the structures could have experienced a collapse of their civilization and were forced to start from scratch. Without an accurate method to date the structures we many never know the truth, but it is more than obvious that they weren't built by the people who ultimately ended up in possession of them.