In the project I am working ( cannot disclose what it is for now ), I had to do a lot of web service calls over http. The server is written in erlang as a gen_server which spawn the process for each request.
Apache is used as the user facing http server. We have written a content handle (apache module) which acts as a c-node by connecting to a local or remote (configured) erlang VM. gen_server in erlang VM registered as a named process (let's say foo). Each incoming user request is sent to the registered process and response from the registered process is sent back to the user (XML or JSON).
Process spawned by gen_server make one or more web service call to the back-end servers. We were using erlang's http client. Apparently it is hard to configure as there is not much documentation available to maintain a persistent connection to the back-end (web service) servers. Thus there is a lot of socket churning as web service calls are http connections and were getting opened and closed. Location of the back-end servers are not necessarily near to our servers (physically). That means there is not only communication overhead but also http connection overhead. If ping time between our server and remote server is 25ms, each http connection will take 75ms (3-way hand shake). Having a persistent connection will solve this problem.
We came across ibrowse module. This has a simple configuration to maintain persistent connection as well as configuration for maximum number of connection and maximum number of http requests that can be pipelined on each connection. This improved our latency by 60% at 400qps and there is plenty of room to grow till 1000qps (on single box).
I can also revert back to erlang's default http client provided I can set the profile for httpc while starting inets. I did not find a decent document to do so. If anyone know about it, let me know.
2 comments:
It isn't clear from your post whether you think the performance limitation is in ibrowse, your code or the whole setup? What is the CPU usage at 400qps?
Have you tried increasing the number of concurrent connections?
Actually ibrowse solved the problem by creating more persistent connection. Default http client is not able to maintain consistent persistent connections to the back-end servers.
CPU usage at 400qps is ~30%.
Post a Comment