We ran into a really weird problem at work and were totally unable to find anyone whose solution was the same as ours, so I thought I’d share.

The scenario:

Client has six Apache web servers behind a Kemp load balancer. The database servers and file servers don’t really matter. LB is set in Round Robin, persistence is enabled, almost everything works except for specific AJAX calls. In our case, it was features in WordPress’s admin area. When trying to add a file to a podcast plugin or dragging a new widget to the sidebar, they would just fail without giving an error. Firebug just showed… nothing. We recently added CloudFlare to fight a DDOS attack (more on that another time) and once we did that, Firebug showed a 502 error from the proxy service.

My co-workers and I tried everything we could find. Nobody seemed to have this issue.

The solution was disabling compression on the Kemp’s virtual service for the site. If you have a Kemp, it’s in Add/Modify virtual services and in the Advanced area. Little checkbox. We discovered this by duplicating the main virtual service, pulling one of the real service out of it, and NATing a unique public IP to it, thereby giving us a direct connection to one isolated server. We set our Hosts file to go straight to this new IP, bypassing CloudFlare’s proxy, bypassing the main virtual service for the live site. Once we had this setup, we were able to tweak LB settings with impunity, eventually landing on this. The change was immediate — click, refresh page, test, worked.

We aren’t quite sure of what to make of this. I emailed Kemp just now to see if they could give us a technical perspective. I guess that it would be good to look at all the headers and the request in a broken virtual service and then compare it to a working one? My assumption is that compression is changing the request in such a way that WordPress (well, jQuery, since that’s what it’s using in this case) doesn’t know how to respond? Regardless, we are thrilled to have it fixed and I hope this helps someone out.