Why Erlang scares managers?

Managing is all about control. Managing is about knowing (almost) everything about what one is controlling. If the team is using something manager don't understand or willing to understand, manager is loosing control. Again it is all about control.

99% of managers are comfortable with technologies like PHP, Java etc. Languages with new ideas like Erlang, LISP are far fetched for many managers. They don't want to learn about these new technologies or new ideas. Having these new technologies in the team force them to come out of the comfort zone and take the control away from them. This is not a good news for them. One argument I have heard every time I make a case for why Erlang (is better in-terms of hardware utilization and lines of code) is that "It is hard to hire Erlang developers than PHP/Java developers'. As I see competent developers want to learn new things than putting themselves in the same-old-same-old world. Competent developers are capable of learning new technologies and new ideas.

Off course, there is a initial learning curve. It is always there anywhere you go. Developers/Engineers need to learn the new way of doing things anywhere they go. Some companies are willing to give more time to developers to get familiar with the internal technologies and some don't. But it is a absolute necessity. Everyone does not follow the same development methods or procedures. In a big organization, this changes from group to group.

To stay or to leave....

There comes a time when one has to stop doing the things one was doing for along time. It is hard to do that... Sometime it feels it is impossible.. Some people lack courage to do that, some people lack motivation and for some people situation don't let them..

Once the decision is taken, is it possible to revert that decision.. Yes.. I think it is possible to that. Question is what it the cost of reverting the decision. I don't think it is much..

Dog is not Cat

While talking to my friend today, something struck me regarding the notion of object oriented programming, hierarchy of classes and how these concepts are over used.

Objected oriented design is heavily used for thousands of years. For example, constructing a home. If you think about it, home is built in a very modular way (aka object oriented way). Every part of home is a object [door, window, wall, roof etc.]. They can be put together in many ways to construct a home. The final shape of home may change, but the the basic building blocks remain the same.

Now to the point of using this in computer programming. I think object oriented programming is misunderstood in the software industry. People seems to confuse reusability of the code with object orientation. For example, take the hierarchy of animals. Animal is a base class, cat is derived from class Animal. Now let's say I want to declare a new class Dog. Since Dog class was not considered while implementing Cat class, common functionality of Animal (like see, listen etc.) are implemented in Cat class. Now when one want to design Dog class, since Cat already has the functionalities required by Dog, derive Dog from Cat. This is so wrong. Logically when you derive one class from another, you are implying "is-a" relation. That means, by deriving Dog from Cat [just because you have functionality required by Dog is implemented in Cat], you are implying that Dog is a Cat.

Cases like these, it is better to sit back and think a little. May be code re-factoring will help here. One can move the functionalities that are required by both Dog and Cat [may be Elephant, Tiger etc.] can be moved into the Animal class and implement those functionalities in terms of some abstract concepts like eyes, legs, ears etc.

Now enough of messing your brain.. Go back to work and start thinking about the designs you have done earlier and how you could have done it better [not that what you did earlier was wrong].

Hacking search suggestions

Opensearch specification has extension for search suggestions. Major search providers have their own search suggestion entry points. After looking doing a search of my own, I found the entry points for the 3 major search providers

Bing - http://api.search.live.com/osjson.aspx?query={Search Term}
Google - http://suggestqueries.google.com/complete/search?q={Search Term}&client=firefox
Yahoo - http://ff.search.yahoo.com/gossip?output=fxjson&command={searchTerms}

For Google entry point, removing client parameter will also provide the number of results in the response. This format is not as per the Opensearch standards.

Who is more open? Google or Yahoo or Bing?

While looking at the search result page HTML from Google, Yahoo and Bing, I discovered that Google does not add auto discovery to its search result page. Where as Yahoo and Bing does.

That makes me wonder why Yahoo and Bing does not get as much credit as Google for adopting the open technologies/standards?

http request pipeline in Erlang

I tried to use Erlang's http module for high concurrent requests. It was not performing well due to pipelining and persistent connection issues. This seems to be solved in R13 version. I figured out how to use the http profiles to do selective pipelining/persistent connections to one server but not for others [if application is sending requests to multiple hosts].

First step in the process is to create a new http profile. It can be done in 2 ways. First one is to run a stand along http connection manager (httpc_manager).

{ok, Pid} = inets:start( httpc, [{profile, other}] ).

As per the documentation, this is not desirable as all benefits of OTP framework is lost.

Dynamically started services will not be handled by application takeover and failover behavior when inets is run as a distributed application. Nor will they be automatically restarted when the inets application is restarted, but as long as the inets application is up and running they will be supervised and may be soft code upgraded. Services started as stand_alone, e.i. the service is not started as part of the inets application, will lose all OTP application benefits such as soft upgrade. The "stand_alone-service" will be linked to the process that started it. In most cases some of the supervision functionality will still be in place and in some sense the calling process has now become the top supervisor

2nd method is to run it as a part of inets application via configuration file

Have a config file with the following content (say inets.config)

[{inets, 
[{services,[{httpc,[{profile, server1}]},
            {httpc, [{profile, server2}]}]}]
}].

Run the erlang shell as

erl -config inets.config

This will start 3 http profiles [server1, server2 and default].

Now the question is how to use the newly created profiles. Let's say the application is using 2 web services hosted at foo1.example.com and foo2.example.com. Web service hosted at foo1.example.com is hosted on a web server which can support lot of persistent connections [keep alive connections]. Web service hosted foo2.example.com is hosted on a normal web server which is not optimized for large number of persistent connectinons.

In the application set the profile for server1 for the connections to foo1.example.com. This can be done by changing the http options listed here.

http:set_options([{max_sessions, 20}, {pipeline_timeout, 20000}], server1).

NOTE:It is required to set the pipeline timeout in order to enable http pipelining.

Profile can be specified during the request time.

http:request( "http://foo1.example.com/v1/get_info/dudefrommangalore", server1).

There is no interface provided by httpc_manager or inets to get the info on the number of sessions open to a server. But good news is that the session information is kept in the ets table. One can query the ets table to get the list of persistent connections.

ets:tab2list(httpc_manager_server1_session_db).

Output is something like

{tcp_session,{{"fo11.example.com",80},
               <0.103.0>},
              false,http,#Port<0.1032>,...}

<0.103.0> is a Pid of httpc_handler gen server process. It is possible to get the status of this process via standard OTP sys module.

sys:get_status(erlang:list_to_pid("<0.103.0>")).

It is also possible to get all the pipelined requests on each persistent connections. For that it is necessary to get the pid of the httpc_manager via inets:services_info(). This call will return the pid of the httpc_manager.

[{httpc,<0.52.0>,[{profile,server1}]},
 {httpc,<0.53.0>,[{profile,server2}]},
 {httpc,<0.41.0>,[{profile,default}]}]

From the pid, get the status of httpc_manager gen server process.

sys:get_status(erlang:list_to_pid( "<0.52.0>")).

ets table name is in bold here.

15> sys:get_status(erlang:list_to_pid("<0.52.0>")).
{status,<0.52.0>,
        {module,gen_server},
        [[{'$ancestors',[httpc_profile_sup,httpc_sup,inets_sup,
                         <0.36.0>]},
          {'$initial_call',{httpc_manager,init,1}}],
         running,<0.40.0>,[],
         [httpc_manager_server1,
          {state,[],24596,
                 {undefined,28693},
                 httpc_manager_server1_session_db,httpc_manager_server1,
                 {options,{undefined,[]},
                          0,2,5,120000,2,disabled,false,inet,default,...}},
          httpc_manager,infinity]]}

Get the content of the ets table to get the pipelined connection

ets:tab2list(24596).

Application can tune the http options to utilize the network bandwidth better, get the most of the machine and network.

Erlang process mailbox performance

I came across this performance issue in Erlang while doing the pattern matching against the mailbox [a.k.a. selective message processing]. Here is the orignal code:

-module (perf).

-export( [start/0] ).

start() ->
  S = erlang:now(),
  Pids = spawn_n(fun test/1, 10000, []),
  wait(Pids),
  E = erlang:now(),
  io:format( "Total time: ~p~n", [timer:now_diff(E, S)/1000] ).

spawn_n(_F, 0, Acc) -> Acc;
spawn_n(F, N, Acc) -> 
  Me = self(),
  Pid = spawn(fun() -> F(Me) end),
  spawn_n(F, N-1, [Pid|Acc]).

test(Pid) -> Pid ! {self(), ok}.

wait([]) -> ok;
wait([Pid|Pids]) -> 
    receive {Pid, ok} -> ok end,
    wait(Pids).

Run time for perf:start() was 1.3 seconds

Erlang (BEAM) emulator version 5.6.5 [source] [smp:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.6.5 (abort with ^G)
1> perf:start().
Total time: 1368.038
ok
2>

Now I changed wait(Pids) to wait(lists:reverse(Pids)). After this change, run time for perf:start() was 83 milliseconds.

1> perf:start().
Total time: 83.037
ok

15x improvement just by changing the way mailbox scan is done.

Little things like this are usually overlooked and the language is blamed for the performance issues.