I have seen a stack load of Wordpress blogs that are truly slow!
Two things I normally notice is that the startup page loads up a zillion of blogs, before a link to the next page comes along.
Which in itself is probably fine, but the problem is often that people upload photos straight from a camera of like 5 megapixels, which is several MB in size.
It seems the conversion in Wordpress through some image handler isn't really generating a new correct resized picture to view, it only downsizes it, despite keeping all the Megabytes in the picture.
Something that is easy to notice if you right click the photo and save it, then re-open it in a viewer.
This might of course not be your problem, but I have seen it several (many) times before.
If someone remembers, in the early days of Joomla 1.0 (from around 2.5 years back), immediately after the quantity of photos in the database exceeded 10,000 items, suddenly everything took a serious long time!
This turned out to be a bug (well, a weakspot) in the system, and since not many at that time used Joomla like this for huge galleries, it was hardly ever recognised.
So, logically, it sounds like you are perhaps experiencing something associated with huge amounts of posts, perhaps with photo assignments.
This is just a theory, cause as Daniel says, Wordpress is a pretty robust system.
So, I would start by looking in on the photo database size...