Concurrent future...

Moving blog

2011-06-21T13:13:00.001-07:00

I have moved my blog from here to wordpress for no apparent reason. Using feedburner to make the subscription easier.

PHP author(s) doesn't understand functional programming

2010-10-14T12:38:00.000-07:00

PHP try to emulate functional programming by providing the functions like array_reduce. But what they lack is the knowledge of principals of functional programming.

I had a problem where there are multiple arrays and need to be merged. array_merge function works only on 2 arrays. Merging multiple arrays need a for loop OR use array_reduce function [I thought]. But it turns out that array_reduce function is not functional as I imagined. Even though the documentation say that the last parameter to this function is the initial value for the array fold function, actually it is not.

If the initial parameter is not an integer, PHP will default to zero. For the above problem I tried the following

array_reduce($my_arrays, "array_merge", array());

What I expected it is to work as

$initial = array();
foreach ( $my_arrays as $m ) { $initial = array_merge($initial, $m); }

What I got was
PHP Warning: array_merge(): Argument #1 is not an array in Command line code on line 1
PHP Warning: array_merge(): Argument #1 is not an array in Command line code on line 1

These kind of behavior will lead to serious problem in the system and is hard to debug.

State of the Interview process

2010-10-10T11:58:00.000-07:00

I have attended several interviews and also have interviewed several people. I have seen the difference between the people I interviewed with and the way I do interview.

In the interviews I have given for several job positions at companies I don't want to mention here, most of the time I felt that the person who is interviewing me is not at all achieving his goal of the interview. What that person is trying to do is to prove to me that he/she is smarter than me in all respect.

For me interview is a combination of 2 words "inter" and "view". That means exchanging views among people involved in the process. Thus what I expect is to have a debate on the subject in question. But mostly what happens is not a debate.

In one interview, I was given a problem to solve. Instead of listening to my solution and to have a debate on my solution, the Interviewer was playing some game on this iPhone. I was explaning the solution to him and that person was not even listening to me. Instead he was trying to be an smart ass, saying your solution is wrong. My question is how can a solution be wrong if it can solve the problem. The solution may be not efficient enough. Nevertheless solution is never wrong. Other person may have different solution in mind (which may or may not be the efficient one), but still the purpose of the interview is not achieved.

What I like to do when I am conducting an interview is not to prove that I am smarter than the interviewee (which I am :-)). What I am trying to gauge is the knowledge and the potential of the person opposite to me. I am a believer a person willing to learn and push his/her limit can do wonders. No one is born smart. Smartness is achieved by hard work and will to gain more knowledge. I always try to find a person who can fit into my immediate job description and is smart/hard working enough to achieve more in the near future.

At least in the technology field, what I find is that interview is a lost cause. Most people are I interviewed with are trying to prove that they are smarter than me or they are the smartest people in the world. They are not even try to find if I can fit into the current job and can do more in the future.

Another thing I try to achieve in the interview is the personality of the person I am interviewing. No matter how smart one is unless that person has the right attitude and the personality, he/she cannot fit into the team. It takes only one bad apple to disintegrate the whole basket of good apples. It is all about team work.

Without a good team dynamics, it is not possible to achieve the long term goals. I takes ne person with wrong attitude in the a big team to lead the project to failure. What happens is that this person will demotivate others in the team. It is the responsibility of the manager solve this problem and also the responsibility of the team members to raise the issue. Again it is the responsibility of the manager to see the signs and take actions at the early stage.

Having the social skills is equally important as the knowledge of the current field he/she is working in. Social skills refers to personality, how comfortable one can make others, how good of a team player one can be etc. etc.

Content vs Ads

2010-09-09T14:01:00.001-07:00

Looking at several websites known to provide content, I am wondering what is important for these websites.. Content or Ads.

There was a time when I used to get ads within the content, but now a days I am searching for content within ads. Page is full of ads and somewhere hidden is a small pre-text of content. I have to click on several ads (or real links) to figure out what is the actual content. Most of the time I end up clicking on ads which look like content (thanks to Google adsense ads buried in the content). Google makes it easy for publishers to format the ads so that it looks like the content. "ads by Google" is like a fine-print in the terms of service notice. It is barely visible.

Apart from them, even big names like CNN is also doing it but in a different way. They are displaying big banner ads at the top of the page which pushes the content below the first fold of the page. They make sure that you view the ads. Whether or not you click on the ads is not important, they are banner ads mostly used for brand awareness anyways.

Look at this page from CNN. Big BMW ad at the top of the page covering approximately 80% of the page.

Only title of the article is viewable on the first fold of the content. Hello... I know the title of the article from the website (in this case Yahoo! Finance). I am here to read the content of the article. You are already making money by the content distribution. Please stop throwing the ads at my face...

Spelling checker in Craigslist

2010-09-06T19:14:00.001-07:00

In-spite of being a market leader in listing business, I don't think craigslist understand anything about search. People do mistake while typing search term. So now a days it is absolutely necessary to have a spelling checker and correction service along with the search. It is missing in Craigslist. I found out today when I was searching for car. I typed "toyta car". Instead of giving no results and suggesting corrected spelling, it is threw just a no result page at my face.

Now a days it is not hard to build spelling correction in the system. It is not required to build you own spelling correction system. There are lot of APIs available. BOSS API is one of the prominent one which does provide spelling correction API.

http://boss.yahooapis.com/ysearch/spelling/v1/toyta+car?appid=

HTML Message Ajax Design Pattern

2010-08-27T17:31:00.000-07:00

Having more and more browsers supporting XMLHTTPRequest object to send the data to server and get the response from server without reloading the page, it is possible to refresh only a part of the page. There are various techniques to do this. One such technique is that the server send the XML back to the browser and the Javascript on the browser will convert the XML into HTML by creating the DOM nodes on the fly. This is useful but has some limitations.

* If the Javascript is turned off, there is no graceful degradation of application
* Since the Javascript is running on the browser (client side), changing the structure of the generated HTML is hard
* This is not feasible when HTML generated is complex

One can overcome these limitations by sending the HTML snippet in the response. Instead of sending the XML back to browser, generate the HTML on the server side itself. On the browser side, replace the content of the DOM node with the response HTML response from the server.

If the Javascript is turned off on the browser, user will see at least the content of the HTML on the browser instead of just the XML content. So there is a graceful degradation of service on the client side

If HTML generated need to be changed, application is not dependent on changing the Javascript and hoping that the client will reload the page and will get new Javascript. Even if the Javascript is changed, some web proxies may not pull the new Javascript, resulting in the browsers (clients) behind the proxies will continue to get the old Javascript and old HTML will get rendered.

There are various frameworks like Symfony, Django make it possible to generate the complex HTML possible with least effort. In case the request is not made via XMLHTTPRequest, it is possible to send the whole page (with header, footer etc.) in the response. If the request is made via XMLHTTPRequest object, send the response without the layout decoration (just the HTML snippet). Javascript running on client side will replace the content of the DOM node as required.

Take for example, there is an application where the page loaded has many graphics and only a part of the page changes when user take some action like filling a form for sending the email. It is absolutely unnecessary to generate the whole page again just for a simple action of sending an email. I will make an attempt to explain with the example of sending the feedback via a form embedded in the page. I will try to explain this with PHP Symfony framework and Javascript (AJAX).

The URL for the page is /new-page. This page has a feedback form in it. The action of this form is handled by /feedback. Server side code for /feedback expect a form having a text content, sends an email to the product manager and send same page as new-page but replacing the form with thank you note.

Below is a code for the new-page action.

Code for new-page response.

Now is the magic of jQuery Javascript library to submit the content to /feedback url and use the response content to replace the HTML form.

Server side code for handling the post from the browser to /feedback

HTML response from /feedback

In Symfony framework it is possible disable/enable the layout during run-time via setLayout() method of sfAction instance. Setting the parameter to false will disable the layout and send only the content of the action template back to the caller.

There you go.. A simple way to improve the usability of the site even without Javascript support.

Execute action within another action

2010-08-25T16:58:00.001-07:00

In the current project I am working on (using Symfony framework), I came across a situation where I had to call one action within another action and capture the content of 2nd action. This content is included within the result from 1st action. After doing some research found that it it possible to do so by calling getPresentationFor method of the controller object.

Within the action's execute method,

$content = $this->getController()->getPresentationFor($module, 'my2ndAction');
$this->content = $content;

File System Over SSH

2010-08-05T23:17:00.000-07:00

Some of the development I am doing require me to do the development on some remote machine. Since the development is done in Java, I need Eclipse environment. On the remote system I cannot run Eclipse as it is a server box and does not have display attached to it. I found an elegant solution for this. Thanx for Macfusion

I can mount remote machine's directory on my local machine and work as if it is a local directory. All the dirty work of communication is taken care. I don't need nfs mounting which is not secure. Macfusion let me mount the directory on remote machine over ssh

Why Erlang scares managers?

2010-07-11T12:21:00.000-07:00

Managing is all about control. Managing is about knowing (almost) everything about what one is controlling. If the team is using something manager don't understand or willing to understand, manager is loosing control. Again it is all about control.

99% of managers are comfortable with technologies like PHP, Java etc. Languages with new ideas like Erlang, LISP are far fetched for many managers. They don't want to learn about these new technologies or new ideas. Having these new technologies in the team force them to come out of the comfort zone and take the control away from them. This is not a good news for them. One argument I have heard every time I make a case for why Erlang (is better in-terms of hardware utilization and lines of code) is that "It is hard to hire Erlang developers than PHP/Java developers'. As I see competent developers want to learn new things than putting themselves in the same-old-same-old world. Competent developers are capable of learning new technologies and new ideas.

Off course, there is a initial learning curve. It is always there anywhere you go. Developers/Engineers need to learn the new way of doing things anywhere they go. Some companies are willing to give more time to developers to get familiar with the internal technologies and some don't. But it is a absolute necessity. Everyone does not follow the same development methods or procedures. In a big organization, this changes from group to group.

To stay or to leave....

2010-06-27T15:07:00.000-07:00

There comes a time when one has to stop doing the things one was doing for along time. It is hard to do that... Sometime it feels it is impossible.. Some people lack courage to do that, some people lack motivation and for some people situation don't let them..

Once the decision is taken, is it possible to revert that decision.. Yes.. I think it is possible to that. Question is what it the cost of reverting the decision. I don't think it is much..

Dog is not Cat

2010-04-20T15:14:00.000-07:00

While talking to my friend today, something struck me regarding the notion of object oriented programming, hierarchy of classes and how these concepts are over used.

Objected oriented design is heavily used for thousands of years. For example, constructing a home. If you think about it, home is built in a very modular way (aka object oriented way). Every part of home is a object [door, window, wall, roof etc.]. They can be put together in many ways to construct a home. The final shape of home may change, but the the basic building blocks remain the same.

Now to the point of using this in computer programming. I think object oriented programming is misunderstood in the software industry. People seems to confuse reusability of the code with object orientation. For example, take the hierarchy of animals. Animal is a base class, cat is derived from class Animal. Now let's say I want to declare a new class Dog. Since Dog class was not considered while implementing Cat class, common functionality of Animal (like see, listen etc.) are implemented in Cat class. Now when one want to design Dog class, since Cat already has the functionalities required by Dog, derive Dog from Cat. This is so wrong. Logically when you derive one class from another, you are implying "is-a" relation. That means, by deriving Dog from Cat [just because you have functionality required by Dog is implemented in Cat], you are implying that Dog is a Cat.

Cases like these, it is better to sit back and think a little. May be code re-factoring will help here. One can move the functionalities that are required by both Dog and Cat [may be Elephant, Tiger etc.] can be moved into the Animal class and implement those functionalities in terms of some abstract concepts like eyes, legs, ears etc.

Now enough of messing your brain.. Go back to work and start thinking about the designs you have done earlier and how you could have done it better [not that what you did earlier was wrong].

Hacking search suggestions

2010-01-30T12:33:00.000-08:00

Opensearch specification has extension for search suggestions. Major search providers have their own search suggestion entry points. After looking doing a search of my own, I found the entry points for the 3 major search providers

Bing - http://api.search.live.com/osjson.aspx?query={Search Term}
Google - http://suggestqueries.google.com/complete/search?q={Search Term}&client=firefox
Yahoo - http://ff.search.yahoo.com/gossip?output=fxjson&command={searchTerms}

For Google entry point, removing client parameter will also provide the number of results in the response. This format is not as per the Opensearch standards.

Who is more open? Google or Yahoo or Bing?

2010-01-30T12:01:00.000-08:00

While looking at the search result page HTML from Google, Yahoo and Bing, I discovered that Google does not add auto discovery to its search result page. Where as Yahoo and Bing does.

That makes me wonder why Yahoo and Bing does not get as much credit as Google for adopting the open technologies/standards?

http request pipeline in Erlang

2010-01-22T15:34:00.000-08:00

I tried to use Erlang's http module for high concurrent requests. It was not performing well due to pipelining and persistent connection issues. This seems to be solved in R13 version. I figured out how to use the http profiles to do selective pipelining/persistent connections to one server but not for others [if application is sending requests to multiple hosts].

First step in the process is to create a new http profile. It can be done in 2 ways. First one is to run a stand along http connection manager (httpc_manager).


{ok, Pid} = inets:start( httpc, [{profile, other}] ).

As per the documentation, this is not desirable as all benefits of OTP framework is lost.

Dynamically started services will not be handled by application takeover and failover behavior when inets is run as a distributed application. Nor will they be automatically restarted when the inets application is restarted, but as long as the inets application is up and running they will be supervised and may be soft code upgraded. Services started as stand_alone, e.i. the service is not started as part of the inets application, will lose all OTP application benefits such as soft upgrade. The "stand_alone-service" will be linked to the process that started it. In most cases some of the supervision functionality will still be in place and in some sense the calling process has now become the top supervisor

2nd method is to run it as a part of inets application via configuration file

Have a config file with the following content (say inets.config)


[{inets, 
[{services,[{httpc,[{profile, server1}]},
            {httpc, [{profile, server2}]}]}]
}].

Run the erlang shell as


erl -config inets.config

This will start 3 http profiles [server1, server2 and default].

Now the question is how to use the newly created profiles. Let's say the application is using 2 web services hosted at foo1.example.com and foo2.example.com. Web service hosted at foo1.example.com is hosted on a web server which can support lot of persistent connections [keep alive connections]. Web service hosted foo2.example.com is hosted on a normal web server which is not optimized for large number of persistent connectinons.

In the application set the profile for server1 for the connections to foo1.example.com. This can be done by changing the http options listed here.


http:set_options([{max_sessions, 20}, {pipeline_timeout, 20000}], server1).

NOTE:It is required to set the pipeline timeout in order to enable http pipelining.

Profile can be specified during the request time.


http:request( "http://foo1.example.com/v1/get_info/dudefrommangalore", server1).

There is no interface provided by httpc_manager or inets to get the info on the number of sessions open to a server. But good news is that the session information is kept in the ets table. One can query the ets table to get the list of persistent connections.


ets:tab2list(httpc_manager_server1_session_db).

Output is something like


{tcp_session,{{"fo11.example.com",80},
               <0.103.0>},
              false,http,#Port<0.1032>,...}

<0.103.0> is a Pid of httpc_handler gen server process. It is possible to get the status of this process via standard OTP sys module.


sys:get_status(erlang:list_to_pid("<0.103.0>")).

It is also possible to get all the pipelined requests on each persistent connections. For that it is necessary to get the pid of the httpc_manager via inets:services_info(). This call will return the pid of the httpc_manager.


[{httpc,<0.52.0>,[{profile,server1}]},
 {httpc,<0.53.0>,[{profile,server2}]},
 {httpc,<0.41.0>,[{profile,default}]}]

From the pid, get the status of httpc_manager gen server process.


sys:get_status(erlang:list_to_pid( "<0.52.0>")).

ets table name is in bold here.


15> sys:get_status(erlang:list_to_pid("<0.52.0>")).
{status,<0.52.0>,
        {module,gen_server},
        [[{'$ancestors',[httpc_profile_sup,httpc_sup,inets_sup,
                         <0.36.0>]},
          {'$initial_call',{httpc_manager,init,1}}],
         running,<0.40.0>,[],
         [httpc_manager_server1,
          {state,[],24596,
                 {undefined,28693},
                 httpc_manager_server1_session_db,httpc_manager_server1,
                 {options,{undefined,[]},
                          0,2,5,120000,2,disabled,false,inet,default,...}},
          httpc_manager,infinity]]}

Get the content of the ets table to get the pipelined connection


ets:tab2list(24596).

Application can tune the http options to utilize the network bandwidth better, get the most of the machine and network.

Erlang process mailbox performance

2010-01-19T16:22:00.000-08:00

I came across this performance issue in Erlang while doing the pattern matching against the mailbox [a.k.a. selective message processing]. Here is the orignal code:


-module (perf).

-export( [start/0] ).

start() ->
  S = erlang:now(),
  Pids = spawn_n(fun test/1, 10000, []),
  wait(Pids),
  E = erlang:now(),
  io:format( "Total time: ~p~n", [timer:now_diff(E, S)/1000] ).

spawn_n(_F, 0, Acc) -> Acc;
spawn_n(F, N, Acc) -> 
  Me = self(),
  Pid = spawn(fun() -> F(Me) end),
  spawn_n(F, N-1, [Pid|Acc]).

test(Pid) -> Pid ! {self(), ok}.

wait([]) -> ok;
wait([Pid|Pids]) -> 
    receive {Pid, ok} -> ok end,
    wait(Pids).

Run time for perf:start() was 1.3 seconds

Erlang (BEAM) emulator version 5.6.5 [source] [smp:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.6.5 (abort with ^G)
1> perf:start().
Total time: 1368.038
ok
2>

Now I changed wait(Pids) to wait(lists:reverse(Pids)). After this change, run time for perf:start() was 83 milliseconds.

1> perf:start().
Total time: 83.037
ok

15x improvement just by changing the way mailbox scan is done.

Little things like this are usually overlooked and the language is blamed for the performance issues.

Concurrency in Java - Part 2

2010-01-18T13:54:00.000-08:00

In the earlier post I covered the basic cached thread pool.

Another facility offered by Java's concurrency framework is to schedule a thread after certain time or at regular interval (like standard unix cron job). There are 2 ways to schedule the thread at regular interval. First one is to run a task at regular interval regardless of the previous job. Second one is to run a task and wait for the certain interval after the previous job is done.

Second one is helpful in situation like crawlers. It is necessary to download the pages with some politeness factor [wait for sometime before downloading a page from the same website].


ScheduledExecutorService executionService = Executors.newScheduledThreadPool(2);

Above code snippet create a pool of 2 threads. This service can schedule the threads at regular interval.

ScheduledExecutorService provide a method schedule


Runnable task = new Runnable() {
   public void run() {
      System.out.println( "I am responsible for downloading a page" );
      return;
   }
};
/* TimeUnit is defined within java.util.concurrent package */
Future future = executionService.schedule(task, 2000, TimeUnit.MILLISECONDS);

The above code schedule a task to run after 2 seconds [2000 milliseconds]. schedule method return a future object. This future object can be used to check the status of the task [isDone method] or to cancel the task [cancel method]. Read more about the future object and the methods available here.

ScheduledExecutorService also support repeated execution of a task via scheduleAtFixedRate and scheduleWithFixedDelay.

scheduleAtFixedRate method schedule the task at regular interval. This method does not check if the previously scheduled task is finshed or not.

scheduleWithFixedDelay method is similar to scheduleAtFixedRate except that this method wait for the previous execution to finish, wait for the fixed interval and then schedule the task again.

Concurrency in Java

2010-01-17T19:03:00.000-08:00

Java 5.x introduced the concurrency framework. It make the life of developer easier to run multiple threads. This framework also take care of thread caching this reducing the number of spawned threads in the system.

When the concepts of thread was introduced in the operating systems, it was considered light-weight processes. As the clock speed of the CPU is increasing dramatically and also number of CPU cores available for the programs are increasing, even this light-weight processes are deemed to costly to start. Thus introduced the concept of cached threads. Erlang solve this problem by introducing ultra-light-weight processes. Millions of such processes can be spawned within few seconds. This is not the case in kernel threads. Even when kernel threads are used, there is a cost of context switching to schedule those threads from wait state to run state.

I am new to Java concurrency framework. So I am taking baby steps to learn to use the classes available in Java 5.x. All the concurrency related classes are in java.util.concurrent package.

First step in start using these classes is to introduce Executors class. This class has several class methods to create thread pools.


import java.util.concurrent.Executors;
import java.util.concurrent.ExecutorService;

ExecutorService threadPool = Executors.newCachedThreadPool();

The above code create a cached thread pool. Behavior of this thread pool is documented here.

Creates a thread pool that creates new threads as needed, but will reuse previously constructed threads when they are available. These pools will typically improve the performance of programs that execute many short-lived asynchronous tasks. Calls to execute will reuse previously constructed threads if available. If no existing thread is available, a new thread will be created and added to the pool. Threads that have not been used for sixty seconds are terminated and removed from the cache. Thus, a pool that remains idle for long enough will not consume any resources. Note that pools with similar properties but different details (for example, timeout parameters) may be created using ThreadPoolExecutor constructors

A task can be submitted to the newly created thread pool for execution. A task must be an instance of Runnable interface. Submitting a task to the ExecutorService will return an instance of Future interface. This is a wrapper around the task submitted. This instance can be used to query the submitted task for completion, as well as to cancel the task.



Future task = threadPool.submit( new Runnable() {
       public void run() {
          System.out.println( "Hello world from within the thread pool" );
       }
  });

Finally wait for the task to be completed


  while ( !task.isDone() ) {
  }

Once the task is completed and thread pool is no-longer necessary, send shutdown message to the thread pool to terminate all the threads created.


  threadPool.shutdown();
  while (!thread.isTerminated()) {
  }

There it is. First Hello world code using the Concurrent Thread Pool in Java. As and when I learn new methods in this framework, I will write about that here.

Until then, happy thread pooling and utilizing all the cores on the system.

Update: Download the source from here

What do you mean by giving more than 100%?

2009-10-08T15:33:00.000-07:00

Here is a little something someone sent me that is _indisputable_ mathematical logic. (It also made me Laugh Out Loud.)

Remember, this is a strictly mathematical viewpoint. It goes like this:

What Makes 100%? What does it mean to give MORE than 100%? Ever wonder about those people who say they are giving more than 100%? We have all been to those meetings where someone wants you to give over 100%. How about achieving 103%? What makes up 100% in life?

Here's a little mathematical formula that might help you answer these questions:

If:
A B C D E F G H I J K L M N O P Q R S T U V W X Y Z

is represented as:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26.

Then:

H-A-R-D-W-O-R-K
8+1+18+4+23+15+18+11 = 98%*

and

K-N-O-W-L-E-D-G-E
11+14+15+23+12+5+4+7+5= 96%

But,*

A-T-T-I-T-U-D-E
1+20+20+9+20+21+4+5 = 100%

And,

B-U-L-L-S-H-I-T
2+21+12+12+19+8+9+20 = 103%

AND, look how far ass kissing will take you.

A-S-S-K-I-S-S-I-N-G
1+19+19+11+9+19+19+9+14+7 = 118%
So, one can conclude with _mathematical certainty_, that while* Hardwork* and *Knowledge *will get you close, and* Attitude *will get you there, its the *Bullshit* and *Ass kissing* that will put you over the top.

How to see the running processes in erlang shell

2009-09-29T17:25:00.000-07:00

Erlang provides a built-in utility called etop, that is same as standard top utility on Unix. Instead of monitoring processes on the operating system, etop show the current running/active Erlang processes. By default it starts a GUI window to show the processes running. Sometimes GUI is not desirable especially when the Erlang shell is running on production servers. Documentation does not show any way to start etop in text mode. Here is how to run it


etop:start([{output, text}]).

Detecting rejected file upload in PHP code

2009-09-29T16:14:00.000-07:00

PHP can be configured to reject the file uploads exceeding some limit via upload_max_filesize server configuration parameter (in php.ini). As per the documentation here, even though the file upload is rejected, an entry is made in $_FILES array with error code set to UPLOAD_ERR_INI_SIZE. This is documented here.

Practically speaking this is not the case. When the uploaded content size exceed upload_max_filesize, $_FILES array is empty. This array cannot be inspected to find the reason for the error. I found a hack to do this.

* For file uploads, $_SERVER['CONTENT_TYPE'] is always set to multipart/form-data followed by the multipart boundary string.
* $_SERVER['CONTENT_LENGTH'] will be set to the content-length in the http header.

Combination of above 2 can be used to detect the file upload exceeding upload_max_filesize.

if ( stripos( $_SERVER['CONTENT_TYPE'], 'multipart/form-data') === 0 && intval($_SERVER['CONTENT_LENGTH']) > 0 &&
count($_FILES) == 0) {
error_log( 'File upload rejected due to file size exceeding upload_max_filesize limit' );
}

Load balancing in Erlang using pg2 process groups

2009-09-26T13:45:00.000-07:00

While working with some project @ my current workplace, we came across a situation where we are using pg2 process group. Apparently pg2 does not load balance across processes registered in the pg2 group. It usually done in round robin way. So it may happen that even though many processes are registered under same group, but all messages will get queues in single process.

My co-worker and I came with an algorithm to solve this. It is describe here.

Clustering RabbitMQ servers for High Availability

2009-08-21T19:09:00.001-07:00

Clustering guide on rabbitmq website is a good start. But it does not seem to work out of the box.

Default rabbitmq-server script uses -sname (short node name) option on the erlang shell command line. This makes it impossible to setup the cluster as it is not possible to put a node into the cluster when short name is given. So first step is remove that line from the rabbitmq-server script

By default erlang shell generate a random cookie and save that in the .erlang.cookie file in the home directory. Since the cookie generated is random, unless the same cookie file copied onto the other nodes, it is not possible to setup the cluster. Better idea is to set the cookie on the command line itself. This can done via the environment variable RABBITMQ_SERVER_START_ARGS.

Also by default, rabbitmq-server do not set the full node name. One need to specify that on the command line argument. Better place is to set the environment variable RABBITMQ_SERVER_START_ARGS (same as the one used to set the cookie)

Erlang's DNS lookup is done via inet module. By default it uses /etc/hosts as the first one to resolve the hostname. Therefore in some environments (at least in my environment), short name does not resolve to long name when I use inet:gethostbyname(Shortname) function. I get short name back in hostent structure. Workaround for this is to force the use of native library for dns lookup. This can be done via inetrc file. By default Erlang uses $HOME/.inetrc file (if exists). Non-standard location can be specified via ERL_INETRC environment variable. Don't forget to set the full path name.

Content of the inetrc file will look like

{lookup, [native]}.

Other possible options are file,yp,dns.

It is good to know some other environment variables.

RABBITMQ_LOG_BASE -> log location (default /var/log/rabbitmq)
RABBITMQ_NODENAME -> node name to be used (default rabbit)
RABBITMQ_MNESIA_BASE -> location for mnesia database (default /var/lib/rabbitmq/mnesia)

If your machine has multiple network interfaces, by default RabbitMQ binds to all the network interfaces. In some cases it is not desirable. One can force it to bind to only one network interface by setting the environment varaible RABBITMQ_NODE_IP_ADDRESS to the ip address of the network interface.

Here is a summary of all environment variables required

RABBITMQ_SERVER_START_ARGS="-name rabbit@`hostname` -setcookie rabbit"
RABBITMQ_LOG_BASE=/home/baliga/rabbit
RABBITMQ_NODENAME=rabbit
RABBITMQ_MNESIA_BASE=/home/baliga/rabbit
ERL_INETRC=/home/baliga/inetrc

Now it is time to start rabbitmq server on all the nodes.

rabbitmq-server -detached

To setup the cluster rabbitmqctl script is used. By default this script too has -sname option setup on the command line. Delete this line from rabbitmqctl file. In order for this script to communicate with the node, long name and cookie required. Set this via RABBITMQ_CTL_ERL_ARGS environment variable

export RABBITMQ_CTL_ERL_ARGS="-name rabbitmqctl@`hostname` -setcookie rabbit"

Note that the cookie should be same as that of the rabbitmq-server instance.

Now you can run the rabbitmqctl to setup the cluster.

On each of the node, run the following command

1. rabbitmqctl stop_app
2. rabbitmqctl reset
3. rabbitmqctl cluster
4. Repeat step 3 for all nodes in the cluster except for itself.
4. rabbitmqctl start_app

Now the cluster is ready to be used.

Some questions are not answered here like how the clustering works when nodes are behind separate firewalls. What ports need to be opened in firewall for clustering to work. I am still doing the research on it. Will post the findings soon.

Is parallel computing/concurrency hard to get?

2009-08-13T19:14:00.001-07:00

Regarding this subject, first question I would like to ask is why one need concurrency? My one line answer would be "As we go into the future, number of cores available on even desktops will increase dramatically". That's why one need to think in terms of parallel operations instead of serial operations. Many libraries are already there to achieive this like MPM.

In my opinion, these libraries are hard to use and still need to deal with the underlying network architecture. But it is not necessary. It can be achieved without getting into the low level functions. Only thing is require is "changing the way we think of designing the system".

Functional is my answer to this problem. Just thinking functional is not enough. But one should get the real feel of it. One need to forget everything he/she knows and start thinking like a child. Start to learn things from the begining.

When I started thinking functional, even though I was thinking functional but that functional always turned out to be serial. Then I got introduced to this wonderful language called "Erlang". This really force one to think and do things not only in functional but do things in concurrency.

Nice photography!

2009-08-11T23:15:00.001-07:00

New and improved delicious released

2009-08-04T08:51:00.000-07:00

Today Yahoo! Delicious released a new cool features.

* Integration with Twitter. See the related tweets.
* Share interesting links not just with delicious users, but also with your friends via email.
* Search is ever more improved now. You can see the search results on timeline. Find the activity for your search terms over time (even though it is not done with Yahoo! Search)
* You can watch youtube and other videos inline now. No need to leave delicious search page.
* Flickr images are also displayed inline.

Congratulations to delicious team.

Module inherittance in Erlang

2009-08-03T12:21:00.000-07:00

Erlang support module inheritance via extends module property.

Here is the example code:

parent.erl
-------------

-module (parent).
-export( [fun1/0, fun2/0] ).

fun1() ->
io:format( "In parent::fun1/0~n" ).

fun2() ->
io:format( "In parent:fun2/0~n" ).

child.erl
----------

-module (child).
-extends(parent).
-export( [fun1/0] ).

fun1() ->
io:format( "In child:fun1/0~n" ).

Testing this:

erl

> c(parent), c(child).
{ok,child}.
> parent:fun1().
In parent:fun1/0
> parent:fun2().
In parent:fun2/0.
> child:fun1().
In child:fun1/0.
> child:fun2().
In parent:fun2/0

Even though fun2 is not defined/exported in child module, calling child:fun2/0 is a valid call as parent has exported fun2/0 function.

If you call module_info() on child, you won't see fun2 function in the exported function list. Erlang VM find the extended module via attributes property of the module.

Obama, what are you doing?

2009-07-10T07:42:00.001-07:00

SAX Xml parsing in Erlang - Part 1

2009-06-20T19:57:00.000-07:00

Erlang has a default XML parser called xmerl. Even there is a documetation associated with this module, it is not adequate enough to start writing the code. Especially SAX parsing is the least documented and least understood. There is not much literature (infact no documentation at all) for the SAX parsing. I had to do lot of code reading to figure out the SAX parsing module.

In this article I will make an attempt to explain the SAX parsing module.

SAX parsing is done via xmerl_eventp:stream/2 and xmerl_eventp:stream_sax/4.

I did not find xmerl_eventp:stream_sax/4 useful as user state is not maintained across all events. Also accumulate function cannot be overridden. So it is not useful when you do not need to accumulate all the elements (for large file parsing). I was trying to use this function to parse large file ( > 10MB ) and extract only some part of the whole file. Since it is not possible to override accumulate, this function was out of question. If one need to accumulate all the elements in the document, I would rather do xmerl_scan:file/1 or xmerl_scan:string/1 which will scan the file/string and generate in-memory document.

xmerl_eventp:stream/2 function take 2 arguments.
* File name
* List of options

For this post, 3 options are important. They are user_state, event_fun and acc_fun.

user_state is any term which can be accessed from event_fun and acc_fun. event_fun is called for every event during the SAX parsing. acc_fun is called for every element (after parsing end of element).

event_fun has a signature fun/2. First parameter is of type #xmerl_event. Second parameter is a global state. user_state from this global state can be accessed via xmerl_scan:user_state/1 function. User state can be modified via xmerl_scan:user_state/2 function. event_fun should return the new global state.

Example of event_fun:

event_function(#xmerl_event{event = Event, data = Data}, GlobalState) ->
UserState = xmerl_scan:user_state(GlobalState),
NewUserState = do_something(UserState),
NewGlobalState = xmerl_scan:user_state(NewUserState, GlobalState),
NewGlobalState.

Possible values for #xmerl_event.event are started and ended.

#xmerl_event.data is set to document at the beginning of document parsing and after all the elements are parsed.

event_function( #xmerl_event{event = started, data = document}, GlobalState) ->
% get the user state
% do something with the user state
% set the new user state in global state
% return global state

One may want to open a file or open a socket connection at the beginning of the document parsing. In subsequent event, one may want to write the content to the previously opened file or socket connection.

End of document parsing event can be used to close the previously opened file handle or socket connection.

acc_fun can be used to accumulate the content. acc_fun is of the form fun/3. First parameter is the current xml element. Second parameter is the accumulator. Third parameter is the global state.

This function should return a new tuple of 2 elements. First element is the new accumulator and 2nd element is a new global state (user state can be changed in the new global state as explained earlier).

user_state can be any Erlang term.

In the next article, I will discuss an implementation of SAX parsing to transform an XML into CSV.

What is the longest word in English?

2009-06-20T19:51:00.000-07:00

Longest english word is "Pneumonoultramicroscopicsilicovolcanocon". As per Wikipedia,

a factitious word alleged to mean 'a lung disease caused by the inhalation of very fine silica dust, causing inflammation in the lungs.

More here

Controlling a node from stop script

2009-04-06T11:41:00.000-07:00

There is no command line option on Erlang to terminate a node in a graceful manor and report the same. Here is a module I came up with.

-module (node_ctrl).

-export ([stop/1]).

stop([Node]) ->
io:format( "Stopping ~p: ", [Node] ),
case net_kernel:connect_node(Node) of
false -> io:format( "not reachable~n", [] );
true ->
net_kernel:monitor_nodes(true),
rpc:call(Node, init, stop, [] ),
receive
{nodedown, Node} -> io:format( "done~n", [])
after 20000 -> io:format( "refused to die~n", [])
end
end,
init:stop().

Usage:

erl -name foo@example.foo.com -setcookie mycookie -s node_ctrl stop "targetnode@example.foo.com"

Assumptions:
* foo@example.foo.com is started with mycookie

Erlang XML parser comparison

2009-04-03T16:40:00.000-07:00

I am looking into various XML parsing in Erlang. I found mainly 3 of them.

* Xmerl from Erlang distribution
* Erlsom
* Linked-in driver based on libexpat from ejabberd

I did some benchmarking for 3 parsers. Parsing was done on 42K sized XML.

xmerl took 124ms, erlsom took 28ms and linked-in driver based parser took 7ms.

libexpat based parser is the fastest. But it has some drawbacks.
* It cannot do callbacks for SAX parser as linked-in driver cannot do rpc call into the host VM. So it is not good for parsing huge XML data.
* With this one will loose the platform independence. Need to compile the linked-in driver for the platform one is working on.

Erlsom seems to be better than the default xmerl parser.
* It also generate a Erlang data structure (tuples and list).
* Provides continuation function callback when data not enough data is there for parsing.
* Convert the XML to erlang data structure as per the XS
* SAX based parsing
* Some limitations are listed here

Linked-in driver libexpat based parser is the fastest one. It is not flexible enough. It returns list of tuples where first element of tuple is an integer which indicate the begining of element, end of element and cdata/character content. Some parser is necessary to convert this to xmerl structure or any other desired structure.
* Since this does not provide callbacks, it is not desirable to parse huge files
* DOM generated need re-parsing
* Since it is libexpat based parser, it check for utf-8 validity etc.

I heard that next version of xmerl is going to be faster than what it is now. I have not gotten the latest xmerl parser yet. Once I have it, I will do the benchmarking and do another post on my findings.

UPDATE: xmerl also support validating XML against XSD via xmerl_xsd module.
{ok, Xml} = xmerl_scan:string(XmlString),
{ok, Schema} = xmerl_xsd:proces_schema(XsdFile),
xmerl_xsd:validate(Xml, Schema)

UPDATE: Tried xmerl_scan:file/1 with R13 release. It is a significant improvement in performance. Now it can do 44K file in 35-40ms. Still slower than erlsom and expat based parser.

RPC from Erlang Linked-in driver port

2009-03-31T15:09:00.001-07:00

Developing SAX based XML parser (using libexpat) as a linked-in driver for Erlang, I came across the requirement to do the callback within linked-in driver. I have done it earlier in the c-node via ei_rpc C API. This require linked-in driver to run as a c-node. I was looking for a way to avoid it.

Then I came across this function driver_send_term. This can be used to send a term to any PID within the local VM. It is easy to send the term to the same process which did port_cmd. The PID for that process can be obtained using driver_caller C API.

My requirement is to send it to some other PID than the calling process. There is no standard API to do so. But going through the source code for erlang, I figured the way to convert erlang_pid sent by the caller to the one usable within driver_send_term call.

ErlDrvTermData pid = ((ErlDrvTermData) ( ((callbackPid.serial << 15 | callbackPid.num)) << 4 | (0x0 << 2 | 0x3)) );

This works with R12 and R13 version of Erlang. This is not guaranteed to work in the future releases. But till then I am going to use it.

I wonder why Erlang did not provide the standard interface to convert erlang_pid to ErlDrvTermData to be used within driver_send_term.

Sharing binary data and reference counting

2009-03-30T18:26:00.000-07:00

This email thread discussion concludes that

constants in erlang code are stored in a constant pool memory instead of process memory. So there is no copying of this data in the process memory. This is more efficient when one knows that the data is not going to change.

Example for this kind of data is configuration. If configuration is known at the compile time, well and good. But what if the configuration is read from the file during run-time. This data will get copied into the process memory.

I got the suggestion to generate the code during run-time and load that code. Thus erlang VM will ensure that these configuration elements are in constant pool memory and the data is not copied into process memory.

I decided to give it a try. Generated the code and stored it in a file.

ConfigFetcher = list_to_atom("fetch_" ++ atom_to_list(Module) ++ "_config"),
FileName = code:priv_dir(Module) ++ "/" ++ atom_to_list(ConfigFetcher) ++ ".erl",
file:write_file(FileName, GeneratedCode).

Compiled it using

compile:file(FileName, [{outdir, code:lib_dir(Module, ebin)}]).

Load the compiled file,

code:load_file(ConfigFetcher)

Let's say that fetch is a function exported in the generated module. Using this fetch function for fetching the configuration for a given value, ran a small test to find the performance improvement.

The numbers I saw were mind boggling. Using old method I could get 11k fetches per second. With new method (code generation and loading), I got 500k fetcher per second.

AMPQ Client gotchas!

2009-03-29T17:32:00.000-07:00

There is a small change required in this article on Introducing The Erlang AMQP Client.

In this article, code for subscribing to an queue is

#'basic.consume_ok'{consumer_tag = ConsumerTag}
= amqp_channel:call(Channel, BasicConsume, self()),

This throws exception

Channel 1 is shutting down due to: {{badmatch,false},
[{rabbit_writer,assemble_frames,4},
{rabbit_writer,
internal_send_command_async,5},
{rabbit_writer,handle_message,2},
{rabbit_writer,mainloop,1}]}

Change it to amqp_channel:subscribe(Channel, BasicConsume, self())

NOTE: Download rabbitmq erlang client from here. The version from this page does not work.

TCP server in Erlang

2009-03-26T11:02:00.000-07:00

Came across this good article on writing a TCP server in Erlang. Author forgot to mention one thing here.

connect(Listen) ->
{ok, Socket} = gen_tcp:accept(Listen),
inet:setopts(Socket, ?TCP_OPTS),
% kick off another process to handle connections concurrently
spawn(fun() -> connect(Listen) end),
recv_loop(Socket),
gen_tcp:close(Socket).

In the above code snipper (from the above mentioned blog), it is to be noted that after getting the connection via gen_tcp:accept/1, a process is spawned to continue listen on the socket and the client is served from the same process unlike in any other language where a thread is spawned to serve the incoming client request and main thread continue to listen on the socket. This is very important to note as the incoming client socket has a ownership relationship with the process. If you spawn a process to serve the client, it won't work as the messages won't be delivered to the new process instead are delivered to the process in which gen_tcp:accept/1 was executed.

string:tokens/2 does not handle empty tokens

2009-03-24T22:43:00.000-07:00

string:tokens("A,,,,,", ",") returns ["A"] where as I expect it to return ["A", "", "", "", ""]. This can be solved using regexp:split function.

regexp:split("A,,,,,", ",") returns ["A", [], [], [], []]

Imperative language constructs in Erlang

2009-03-21T16:59:00.000-07:00

"How can I do for loop in Erlang?", "How can I do while loop in Erlang?"....

These are the common questions people ask when they get into the world of functional programming from imperative language(s). It is easy to switch to functional programming in Python, Ruby etc as they are the mixture of functional and imperative language constructs. Programming languages like Erlang (probably Haskell) do not provide explicit for, while constructs. Here are the equivalents way to implement some of the imperative language constructs.

For Loop

for_loop(N, Fun) ->
lists:foreach(Fun, lists:seq(1, N))

for_loop(0, Fun) -> ok;
for_loop(N, Fun) ->
Fun(N),
for_loop(N-1, Fun).

While loop

while_loop(Predicate, Fun, State) ->
while_loop(Predicate(State), Predicate, Fun, State).

while_loop(PredResult, Predicate, Fun, State) when PredResult == false -> ok;
while_loop(_PredResult, Predicate, Fun, State) ->
NewState = Fun(State),
whle_loop(PredResult(State), Predicate, Fun, NewState).

Auto increment in Mnesia database

2009-03-18T15:19:00.000-07:00

Mnesia provides some obscure API called dirty_increment_update/3. This can be used to generate unique ids. Here is an example.

-module (uniqueid).

-export( [test/0] ).

-record( unique_ids, {type, id} ).

test() ->
mnesia:start(),
mnesia:create_table( unique_ids, [{attributes, record_info(fields, unique_ids)}] ),
Id = mnesia:dirty_update_counter(unique_ids, record_type, 1),
io:format( "Id => ~p~n", [Id] ),
Id1 = mnesia:dirty_update_counter(unique_ids, record_type, 1),
io:format( "Id => ~p~n", [Id1] ),
Id2 = mnesia:dirty_update_counter(unique_ids, another_type, 1),
io:format( "Id => ~p~n", [Id2] ),

The output you will get is

Id => 1
Id => 2
Id => 1

A single table can be used to generate the unique ids for other tables. In this example, unique ids are generated for record_type and another_type.

Tuning TCP stack on Linux

2009-03-12T18:51:00.000-07:00

TCP stack on most of the Linux distributions are tuned for the desktop. Using Linux distribution on servers having high load require some tuning on TCP stack. Here is what I found.

Add the following lines into /etc/sysctl.conf

# tcp tuning
net.core.rmem_max=16777216
net.core.wmem_max=16777216
net.ipv4.tcp_wmem=4096 65536 16777216
net.ipv4.tcp_rmem=4095 87380 16777216
net.ipv4.tcp_no_metrics_save=1

Run

sudo sysctl -p

Restart your application(s).

This is effective especially when clients are on slow connection. These configuration change will change the tcp write and read memory size so that lot many bytes can be sent over to client in a single packet.

Gracefully terminating erlang VM

2009-03-04T10:52:00.001-08:00

Once started usually it is not required to stop erlang VM. We can stop the services running as well as erlang VM in a graceful manner with init:stop/0 function. Trick is to call this function on the target VM via remote shell.

For example, to stop the VM running as foo@example.foo.com with cookie foo

erl -name bar@example.foo.com -remsh foo@example.foo.com -setcookie foo -s init stop

The above command will not work as init:stop() is executed within bar@localhost VM, not in foo@localhost. Another way is to do spawn in the target.

erl -name bar@localhost -remsh foo@localhost -setcookie foo -s proc_lib spawn 'foo@example.foo.com' init stop "" -s init stop

The above command will try to execute proc_lib:spawn( 'foo@example.foo.com', init, stop, []). There is a small caveat here. -s option pass all the paramter to proc_lib:spawn/4 as a list, not as a individual parameter.

proc_lib:spawn( ['foo@example.foo.com', init, stop, [] ] ).

Since there is no such function defined, it will not work.

Only other solution is to create a module with one exported function which will spawn a function init:stop/0 on a remote shell.

-module (ctrl).

-export([stop/1]).

stop(Node) ->
proc_lib:spawn( hd(Node), init, stop, [] ).

Note that ctrl:stop/1 function gets list as an argument and it taking only the head element from it.

erl -name bar@example.foo.com -remsh foo@example.foo.com -setcookie foo -s ctrl stop 'foo@example.foo.com' -s init stop

The above command line execute ctrl:stop(['foo@example.foo.com']), init:stop()

UPDATE: This can be done from command line without using any other module. All you have to do is to use -eval option on erl command line.

erl -name bar@example.foo.com -setcookie foo -eval "rpc:call('foo@example.foo.com', init, stop, []), init:stop()."

ibrowse module for erlang

2009-02-28T18:52:00.000-08:00

In the project I am working ( cannot disclose what it is for now ), I had to do a lot of web service calls over http. The server is written in erlang as a gen_server which spawn the process for each request.

Apache is used as the user facing http server. We have written a content handle (apache module) which acts as a c-node by connecting to a local or remote (configured) erlang VM. gen_server in erlang VM registered as a named process (let's say foo). Each incoming user request is sent to the registered process and response from the registered process is sent back to the user (XML or JSON).

Process spawned by gen_server make one or more web service call to the back-end servers. We were using erlang's http client. Apparently it is hard to configure as there is not much documentation available to maintain a persistent connection to the back-end (web service) servers. Thus there is a lot of socket churning as web service calls are http connections and were getting opened and closed. Location of the back-end servers are not necessarily near to our servers (physically). That means there is not only communication overhead but also http connection overhead. If ping time between our server and remote server is 25ms, each http connection will take 75ms (3-way hand shake). Having a persistent connection will solve this problem.

We came across ibrowse module. This has a simple configuration to maintain persistent connection as well as configuration for maximum number of connection and maximum number of http requests that can be pipelined on each connection. This improved our latency by 60% at 400qps and there is plenty of room to grow till 1000qps (on single box).

I can also revert back to erlang's default http client provided I can set the profile for httpc while starting inets. I did not find a decent document to do so. If anyone know about it, let me know.

iolost vs list in Eralng

2009-02-18T11:25:00.001-08:00

I started using Erlang few months back. Erlang has very little documentation (in most cases) and no documentation (in some cases).

In Erlang strings are implemented in terms of list. Unlike C, string is a linked list. So every character in a string occupy 2 machine words, one for the character and another for the next pointer. On a 32-bit machine, each byte translates to 8 bytes and on 64-bit box 16 bytes. This is an issue when you are dealing with transporting string over the network.

This becomes an issue when transporting data between erlang VM and c-node. If string is transported as internal representation, erlang will transport 8x number of bytes for a plain string. It is better to transport string as a binary (use erlang:list_to_binary/1 function). This will reduce the network overhead by 8x (on 32-bit machine).

If a string is constructed in VM and is transported to C-node, it is easier to construct a iolist and use iolist_to_binary at the end to transport it as a binary. This will reduce the overhead of string concats and also save a lot on the garbage collection.

For example,

construct_string() ->
[[<<"this is">>, <<" a">>], [" very long", <<" string.">>], <<" This string will go to c-node">>].

transport_to_code() ->
iolist_to_binary(construct_string()).

I saw huge improvement in memory as well as CPU utilization by using binaries instead of strings. Lesson learned is that do not use strings unless it is absolutely required. If binaries will do the work, use it.

Pattern matching for string and list in erlang

2008-12-21T10:18:00.001-08:00

Earlier I wrote a post on how to distinguish string from list. Steve and Justinpointed that my approach will not work for a list of anythingg other than string.

Here is what I am trying to solve. I have a main configuratin file and client specific configuration files. Client specific configuration file can override or add more values from main configuration file. If the value in main configuration file is a string, I want the client configuration to override the value of main configuration. If the value in main configuration is a list, I want to append the list from the client configuration to the main configuration.

For example, main configuration file is

[{blacklist, ["foo.com", "aaa.com"]},
{source_host, "http://www.bar.com"}].

and client configuration file is

[{blacklist, [ "bar.com" ]},
{source_host, "http://www.something.com"}].

After merging above 2 configuration, I want the final configuration to look like,

[{blacklist, [ "foo.com", "aaa.com", "bar.com" ]},
{source_host, "http://www.something.com"}].

The code reading the configuration files and merging them does not know the first element of the tuple ahead of time. The code is generic to read a file of list of tuples and merge the values to generate the final configuration.

Since Erlang implement string as a list, using is_list guard for a function will not work. If anybody has solved this problem earlier, please comment on this blog.

Differentiate string from regular list in Erlang

2008-12-13T16:13:00.000-08:00

Erlang is a great language to work with. I need to concentrate on the business logic rather than nifty details about the distributed computing like communication between nodes, handle node failures, restart apps etc. All these details are already taken care by the language itself. Yes.. distributed computing and concurrent programming is a integrated part of the language.

Erlang implement string in terms of list. So it is tricky to distinguish regular string from the list. For example,


foo(C) when is_string(C) ->
   % do something with string;
foo(C) when is_list(C) ->
   % do something with list.

First of all there is no function called "is_string" in Erlang. (I just made it up). But the point here is that how to distinguish regular string from the list. I came up with this solution.


foo(C) when lists:flatten(C) == C ->
   % do something with string;
foo(C) when is_list(C) ->
   % do something with list.

Somewhat inefficient because of call to lists:flatten/1 call. But for small strings, it is not an issue at all.

Shame on you Mr. Manmohan Singh....

2008-12-06T10:51:00.001-08:00

Here is a comment from Condoleezza Rice:

US Secretary of State Condoleezza Rice has told Pakistan that there is "irrefutable evidence" of involvement of elements in this country in the Mumbai terror attacks and it had no option but to act urgently "otherwise, the US will act."

Dispute with Pakistan is of India's own. Should Indian government wait for USA to protect it inspite of having one of the biggest and powerful army in the world? Shame on you the respected prime minister of Manmohan Singh. I expected such a statement to come from Indian government as soon as India had enough proof to implicate "Elements within Pakistan" for the attack on Mumbai on 26/11. Why should Indian government present the proof to US government and wait for them to act on behalf of us? Isn't that shameful?

I am not supporting war with Pakistan as it is bad for the people of India as well as Pakistan. The terroists activity in Mumbai has nothing to do with the people of Pakistan. It is some groups freely operating within Pakistan is responsible for it. Punishing the whole country for the act of few people is not a wise idea. But India should have acted against the group immediately. Do what US did in Afghanistan. Ask the government to hand over the people responsible. If they don't, get in and take them out yourself. Don't wait for some other country to do that for you.

Shame.. Shame.. Shame...

A Letter to the Prime Minsiter of India

2008-12-02T16:23:00.001-08:00

Today I received this email from one of my colleague. It really captured my attention. Here is the letter.

LETTER TO PRIME MINISTER

Dear Mr. Prime minister

I am a typical mouse from Mumbai. In the local train compartment which has capacity of 100 persons, I travel with 500 more mouse. Mouse at least squeak but we don't even do that.

Today I heard your speech. In which you said 'NO BODY WOULD BE SPARED'. I would like to remind you that fourteen years has passed since serial bomb blast in Mumbai took place. Dawood was the main conspirator. Till today he is not caught. All our bolywood actors, our builders, our Gutka king meets him but your Government can not catch him. Reason is simple; all your ministers are hand in glove with him. If any attempt is made to catch him everybody will be exposed. Your statement 'NOBODY WOULD BE SPARED' is nothing but a cruel joke on this unfortunate people of India.

Enough is enough. As such after seeing terrorist attack carried out by about a dozen young boys I realize that if same thing continues days are not away when terrorist will attack by air, destroy our nuclear reactor and there will be one more Hiroshima.

We the people are left with only one mantra. Womb to Bomb to Tomb. You promised Mumbaikar Shanghai what you have given us is alianwala Baug.

Today only your home minister resigned. What took you so long to kick out this joker? Only reason was that he was loyal to Gandhi family. Loyalty to Gandhi family is more important than blood of innocent people, isn't it?

I am born and bought up in Mumbai for last fifty eight years. Believe me corruption in Maharashtra is worse than that in Bihar. Look at all the politician, Sharad Pawar, Chagan Bhujbal, Narayan Rane, Bal Thackray, Gopinath Munde, Raj Thackray, Vilasrao Deshmukh all are rolling in money. Vilasrao Deshmukh is one of the worst Chief minister I have seen. His only business is to increase the FSI every other day, make money and send it to Delhi so Congress can fight next election. Now the clown has found new way and will increase FSI for fisherman so they can build concrete house right on sea shore. Next time terrorist can comfortably live in those house, enjoy the beauty of sea and then attack the Mumbai at their will.

Recently I had to purchase house in Mumbai. I met about two dozen builders. Everybody wanted about 30% in black. A common person like me knows this and with all your intelligent agency & CBI you and your finance minister are not aware of it. Where all the black money goes? To the underworld isn't it? Our politicians take help of these goondas to vacate people by force. I myself was victim of it. If you have time please come to me, I will tell you everything.

If this has been land of fools, idiots then I would not have ever cared to write you this letter. Just see the tragedy, on one side we are reaching moon, people are so intelligent and on other side you politician has converted nectar into deadly poison. I am everything Hindu, Muslim, Christian, Schedule caste, OBC, Muslim OBC, Christian Schedule caste, Creamy Schedule caste only what I am not is INDIAN. You politician have raped every part of mother India by your policy of divide and rule.

Take example of former president Abdul Kalam. Such a intelligent person, such a fine human being. You politician didn't even spare him. Your party along with opposition joined the hands, because politician feels they are supreme and there is no place for good person.

Dear Mr Prime minister you are one of the most intelligent person, most learned person. Just wake up, be a real SARDAR. First and foremost expose all selfish politician. Ask Swiss bank to give name of all Indian account holder. Give reins of CBI to independent agency. Let them find wolf among us. There will be political upheaval but that will better than dance of death which we are witnessing every day. Just give us ambient where we can work honestly and without fear. Let there be rule of law. Everything else
will be taken care of.

Choice is yours Mr. Prime Minister. Do you want to be lead by one person or you want to lead the nation of 100 Crore people?

What should India do about Mumbai attacks?

2008-11-29T11:12:00.000-08:00

During his preliminary interrogation, Qasad believed to have reportedly told police, "I have done right and I have no regrets." He has also admitted to being a member of the Lashkar.

This what Qasad said who was captured by NSG in Mumbai. Should India listen to this and keep quite? I say NO... Make him regret. Whatever means necessary.

If I was given control, I would make him regret by making his loved one loose their life in front of his eyes and set an example for the future. Make people think about the consequences if at all they decide to do something like what happened in Mumbai.

For those who say I am cold, I would say bring it on.

Who is responsible for Mumbai attack?

2008-11-28T23:33:00.000-08:00

US expert on South Asia dubs Mumbai attacks as India's "domestic issue"

Here is what he said:

This is a domestic issue. India's domestic problems and long tensions between local communities were at the root of the rise of terrorism in the country

How come when the attack is in US, it is global terrorism and when the attack is on other countries (ofcourse other than the so called friends of US) it is domestic issue.

This guy failed to check that the attackers were not from India, but everyone of them were from Pakistan. More over, Pakistan reverted its decision to send the head of ISI to India to share the information on this terrorist attack.

Mr/Ms. Christine Fair, get your facts right before making any statements.

Read more about it here.

Wah bhai wah!

Parallel assignment

2008-11-14T23:33:00.000-08:00

Today I was working on some problem (off course in Erlang). What I am trying to achieve was to execute a function in parallel and assign result of function to a variable.


Func = fun(Arg) ->
          Parent = self(),
          Pid = spawn( fun() -> timer:sleep(1000), Parent ! {self(), Arg + 2} end ),
          receive {Pid, Val} -> Val
       end,
RetVal = Func(),
SomeOtherValue = 10,
% rest of the code
io:format( "~p~n", [RetVal] ),
% more code here

In the above code, calling Func() block the execution until receive loop within Func is done receiving the data from the spawned process. SomeOtherValue is assigned only after the execution of Func.

What I want is not that. I want the execution to continue even if the execution of Func is not finished. I want to block only when I am trying to use RetVal variable.

Currently I am acheiving this by waiting for the response from the spawned process only when I need the data from the process, i.e., same as blocking when the variable is accessed.


Func = fun(Arg) ->
          Parent = self(),
          Pid = spawn( fun() -> timer:sleep(1000), Parent ! {self(), Arg + 2} end ),
       end,
Pid = Func(),
SomeOtherValue = 10,
% rest of the code
RetVal = receive {self(), Val} -> Val end,
io:format( "~p~n", [RetVal] ),
% more code here

Even though it is a trivial code change, I would like if the parallel assignment is allowed and block when I try to use it (if necessary).

Erlang and Garbage collection

2008-11-13T07:00:00.000-08:00

In my new project, we did the whole coding in erlang. Yaws is our front-end. When running a load test, we observed that memory usage goes high and ultimately crash the erlang virtual machine. We did a thorough analysis of our code to find any possible memory leak (not possible as erlang uses garbage collection) and also studied the code for any possible recursion instead of tail recursion. We did not find any. It was pretty puzzling. Then I did find one strange thing. This is all about erlang garbage collection works.

In turns out that Erlang's garbage collection works per process. Each process has its own heap and stack. If a process is a long running one and does lot of things for each call, memory gets accumulated and will get released very slowly. When Erlang virtual machine is busy doing things, garbage collection thread gets lower priority and may never get called. Even if it is called, it may not do the whole memory garbage collection. Thus, memory keeps on growing.

In our code, we were running a gen server. This gen server was doing lot of work in the same process. We were running several hundreds of these gen_servers (monitored by a supervisor). Child specs for the supervisor was created dynamically depending on the initial configuration. In nutshell, it was doin the following


-module(foo_server)
-behaviour(gen_server)

% Not outlining all methods exported here, only the required ones

execute(Params) ->
   gen_server:call({run, Params}).

handle_call({run, Params}, _From, State) ->
  some_module:execute(Params). % some_module:execute does most of the work here.

Note that here some_module:execute/1 runs in the same process as gen_server. Idea here was that gen_server is monitored by supervisor and we get free process monitoring. But since same process gets called over and over again, memory accumulates. Running load test was increasing memory usage.

Now Erlang process comes to rescue. Because of functional aspect, it got pretty easy to change the code to run some_module:execute/1 in a different process and return the result to the caller via process message passing. I did the following changes


-module(foo_server)

execute(Params) ->
  MySelf = self(),
  spawn( fun() -> Result = some_module:execute(Params), MySelf ! {MySelf, Result} end),
  receive 
      {MySelf, R} -> R;
      after 1000 -> {error, timeout}
  end.

Eureka! Running the same load test did not result in increased memory usage. Memory was pretty stable even after running the load test for more than an hour.

URL Shortcut in Firefox

2008-11-10T11:49:00.000-08:00

Firefox has this amazingly useful but hidden feature of adding shortcut for a bookmark. This is not just restricted to search based urls. Here is how to do this.

1. Bookmark your favorite URL
2. Go to Bookmarks -> Organize Bookmarks... menu
3. Choose the bookmark from (1)
4. In the bottom panel, click on "More"
5. Enter shortcut word or letter in "Keyword" text box
6. Close "Organize Bookmarks" window
7. In the address bar, type the word or letter from (5)
8. Viola!
9. Enjoy the time saved in taking the mouse pointer to Bookmarks menu and selecting the bookmark :-)

Quatum of Solace...

2008-11-08T12:16:00.001-08:00

Java is like writing story.

2008-11-03T19:17:00.000-08:00

Here is a code to get md5 signature for a text


public static String MD5(String text) 
        throws NoSuchAlgorithmException, UnsupportedEncodingException  {
                MessageDigest md;
                md = MessageDigest.getInstance("MD5");                
                byte[] md5hash = new byte[32];   
                md.update(text.getBytes("iso-8859-1"), 0, text.length());
                md5hash = md.digest();
                return convertToHex(md5hash);
}

private static String convertToHex(byte[] data) {
        StringBuffer buf = new StringBuffer();
        for (int i = 0; i < data.length; i++) {
                int halfbyte = (data[i] >>> 4) & 0x0F;
                int two_halfs = 0;
                do {
                        if ((0 <= halfbyte) && (halfbyte <= 9))
                        buf.append((char) ('0' + halfbyte));
                    else
                        buf.append((char) ('a' + (halfbyte - 10)));
                        halfbyte = data[i] & 0x0F;
                } while(two_halfs++ < 1);
        }
        return buf.toString();
}

Here is code in perl


use Digest::MD5 qw(md5_hex);
my $digest = md5_hex($text);

This is 20+ lines vs. 3 lines. Java seems to make everything over-object-oriented. Why do I need to getInstance(), update and then call digest() to just get the md5 of the string?

Viewing PDF from within Firefox on Mac

2008-10-16T18:18:00.001-07:00

I was searching for the firefox plugin to view pdf files within the browser (embedded). I was not able to find the anything useful via google search. But Yahoo! search got me the result, that too first one...

Here is the page

Hurray.. At last I can read PDFs within Firefox without downloading it on to my desktop. No more folder clutter..

How to setup grid using Erlang?

2008-10-07T19:07:00.000-07:00

Erlang is a functional programming language with the built-in support for concurrency and distributed computing. Nodes running erlang virtual machine can communicate with each other without any additional coding.

Erlang use light weight process model to achieve the distributed computing. Erlang processes are lighter than linux thread. One erlang node can run more than 10k (sometime more than 100k) processes on a single node.

Erlang process is started using spawn or spawn_link function. A process can be started on the current vm (virtual machine) or on any other vm in the grid by just adding one additional parameter to these functions.

In order for 2 nodes to communicate with each other, both nodes should have the same cookie. In a grid, all erlang nodes should have the same secret (set via -setcookie command line option or using erlang:set_cookie/2 function)

Erlang nodes follow the following naming convention.

name@hostname where name can be any arbitrary name

In this example, I am assuming that I have 2 hosts namely node1.example.com and node2.example.com

Run the erlang vm on each node with short name

On node1.example.com,

erl -sname foo -setcookie mysecret

On node2.example.com,

erl -sname bar -setcookie mysecret

On node1.example.com, ping node2.

net_adm:ping('bar@node2.example.com').
pong
nodes().
['bar@node2.example.com']

Why Microsoft just don't get it?

2008-10-01T09:17:00.001-07:00

Today Microsoft's Live Search announced SearchPerks program. Users are given point when they use Live Search. These points can be re-deemed for X-Box games, t-shirts etc. But the catch is you have to use Internet Explorer 6.0 or higher.

To earn value for your searches and join the SearchPerks! promotion you must have Internet Explorer 6.0 or higher to participate

Microsoft just don't get that Internet is a open space and users should be allowed to use whatever browser they are using.

I can do everything in C or in Java..

2008-09-29T16:13:00.000-07:00

Here is the philosophy that should be used by programmers

As a programmer you should spend your time expressing your ideas, not jumping through hoops to accommodate a language. If you spend the majority of your time doing the latter, the language you're working in probably isn't very good.

From here

No shadow building

2008-09-29T10:14:00.000-07:00

Le Project Triangle is one of those buildings that make us think that we may actually drive flying cars one day. To be completed by 2014 in the Porte de Versailles area in Paris, its most impressive feature is that, according to the architects, it won't cast shadows on adjacent buildings. The trick is the orientation and it's shape: While it looks like a massive pyramid from one side, the other side shows that it really is an ultra-thin triangle resembling a shark's fin

More here

Threads are at last in C++ standard

2008-09-26T12:13:00.000-07:00

It's been decades since I am doing thread programming in C++. It was so painful to use POSIX thread APIs. It is like writing a same novel many times. Not anymore... C++0x standard is out there now. Yahoooooo.. Thread is included in the standard now. Read more about it here

Illustration of Boost shared_ptr

2008-09-26T10:32:00.000-07:00

Illustration of Boost Shared Pointer - Upload a Document to Scribd