The Battle of Waterloo raged all day, but in most sectors, there were lulls in the fighting. Only at Hougoumont was the struggle more or less continuous from 11.30 in the morning until 7.30 at night.
To read the full article, see Issue 6 of Military times
Your map isn’t accurate – for one that is, you should take a look at the one on Project Hougoumont’s website at: http://www.projecthougoumont.com/location_gb.html
It gives you an example of what it looked like in 1815 and what it looks like today.