Edinburgh Castle, sitting on top of Castle Rock, can claim to be one of the oldest inhabited places in the city. Although the oldest parts of the Castle date 'only' to around 1100, there's evidence of humans living on the hill at least as early as the Iron Age.
The castle itself claims to be one of the most-attacked places in the world, with records of at least 26 sieges and many more attacks over the years. A few of those sieges ended in defeats that destroyed large parts of the building, but the St Margaret chapel, from the early 1100s survives as the city's oldest building. Since the 1600s, it's been a military site, with royal residences moved down the street to Holyrood. Today, it's mostly home to museums.
Although it's the most-visited paid attraction in Scotland, and although 79% of visitors to the city are said to visit it, we didn't!