Only 50 IP addresses? That's not much at all. We have customers with thousands. So the number of IP's isn't the issue I don't think.
Are any other functions slow on your site? How long does the page take to refresh when you publish a post?
Regards,
Mark.