Today Yahoo! Delicious released a new cool features.
* Integration with Twitter. See the related tweets.
* Share interesting links not just with delicious users, but also with your friends via email.
* Search is ever more improved now. You can see the search results on timeline. Find the activity for your search terms over time (even though it is not done with Yahoo! Search)
* You can watch youtube and other videos inline now. No need to leave delicious search page.
* Flickr images are also displayed inline.
Congratulations to delicious team.
Tuesday, August 4, 2009
Monday, August 3, 2009
Module inherittance in Erlang
Erlang support module inheritance via extends module property.
Here is the example code:
parent.erl
-------------
-module (parent).
-export( [fun1/0, fun2/0] ).
fun1() ->
io:format( "In parent::fun1/0~n" ).
fun2() ->
io:format( "In parent:fun2/0~n" ).
child.erl
----------
-module (child).
-extends(parent).
-export( [fun1/0] ).
fun1() ->
io:format( "In child:fun1/0~n" ).
Testing this:
erl
> c(parent), c(child).
{ok,child}.
> parent:fun1().
In parent:fun1/0
> parent:fun2().
In parent:fun2/0.
> child:fun1().
In child:fun1/0.
> child:fun2().
In parent:fun2/0
Even though fun2 is not defined/exported in child module, calling child:fun2/0 is a valid call as parent has exported fun2/0 function.
If you call module_info() on child, you won't see fun2 function in the exported function list. Erlang VM find the extended module via attributes property of the module.
Here is the example code:
parent.erl
-------------
-module (parent).
-export( [fun1/0, fun2/0] ).
fun1() ->
io:format( "In parent::fun1/0~n" ).
fun2() ->
io:format( "In parent:fun2/0~n" ).
child.erl
----------
-module (child).
-extends(parent).
-export( [fun1/0] ).
fun1() ->
io:format( "In child:fun1/0~n" ).
Testing this:
erl
> c(parent), c(child).
{ok,child}.
> parent:fun1().
In parent:fun1/0
> parent:fun2().
In parent:fun2/0.
> child:fun1().
In child:fun1/0.
> child:fun2().
In parent:fun2/0
Even though fun2 is not defined/exported in child module, calling child:fun2/0 is a valid call as parent has exported fun2/0 function.
If you call module_info() on child, you won't see fun2 function in the exported function list. Erlang VM find the extended module via attributes property of the module.
Friday, July 10, 2009
Saturday, June 20, 2009
SAX Xml parsing in Erlang - Part 1
Erlang has a default XML parser called xmerl. Even there is a documetation associated with this module, it is not adequate enough to start writing the code. Especially SAX parsing is the least documented and least understood. There is not much literature (infact no documentation at all) for the SAX parsing. I had to do lot of code reading to figure out the SAX parsing module.
In this article I will make an attempt to explain the SAX parsing module.
SAX parsing is done via xmerl_eventp:stream/2 and xmerl_eventp:stream_sax/4.
I did not find xmerl_eventp:stream_sax/4 useful as user state is not maintained across all events. Also accumulate function cannot be overridden. So it is not useful when you do not need to accumulate all the elements (for large file parsing). I was trying to use this function to parse large file ( > 10MB ) and extract only some part of the whole file. Since it is not possible to override accumulate, this function was out of question. If one need to accumulate all the elements in the document, I would rather do xmerl_scan:file/1 or xmerl_scan:string/1 which will scan the file/string and generate in-memory document.
xmerl_eventp:stream/2 function take 2 arguments.
* File name
* List of options
For this post, 3 options are important. They are user_state, event_fun and acc_fun.
user_state is any term which can be accessed from event_fun and acc_fun. event_fun is called for every event during the SAX parsing. acc_fun is called for every element (after parsing end of element).
event_fun has a signature fun/2. First parameter is of type #xmerl_event. Second parameter is a global state. user_state from this global state can be accessed via xmerl_scan:user_state/1 function. User state can be modified via xmerl_scan:user_state/2 function. event_fun should return the new global state.
Example of event_fun:
event_function(#xmerl_event{event = Event, data = Data}, GlobalState) ->
UserState = xmerl_scan:user_state(GlobalState),
NewUserState = do_something(UserState),
NewGlobalState = xmerl_scan:user_state(NewUserState, GlobalState),
NewGlobalState.
Possible values for #xmerl_event.event are started and ended.
#xmerl_event.data is set to document at the beginning of document parsing and after all the elements are parsed.
event_function( #xmerl_event{event = started, data = document}, GlobalState) ->
% get the user state
% do something with the user state
% set the new user state in global state
% return global state
One may want to open a file or open a socket connection at the beginning of the document parsing. In subsequent event, one may want to write the content to the previously opened file or socket connection.
End of document parsing event can be used to close the previously opened file handle or socket connection.
acc_fun can be used to accumulate the content. acc_fun is of the form fun/3. First parameter is the current xml element. Second parameter is the accumulator. Third parameter is the global state.
This function should return a new tuple of 2 elements. First element is the new accumulator and 2nd element is a new global state (user state can be changed in the new global state as explained earlier).
user_state can be any Erlang term.
In the next article, I will discuss an implementation of SAX parsing to transform an XML into CSV.
In this article I will make an attempt to explain the SAX parsing module.
SAX parsing is done via xmerl_eventp:stream/2 and xmerl_eventp:stream_sax/4.
I did not find xmerl_eventp:stream_sax/4 useful as user state is not maintained across all events. Also accumulate function cannot be overridden. So it is not useful when you do not need to accumulate all the elements (for large file parsing). I was trying to use this function to parse large file ( > 10MB ) and extract only some part of the whole file. Since it is not possible to override accumulate, this function was out of question. If one need to accumulate all the elements in the document, I would rather do xmerl_scan:file/1 or xmerl_scan:string/1 which will scan the file/string and generate in-memory document.
xmerl_eventp:stream/2 function take 2 arguments.
* File name
* List of options
For this post, 3 options are important. They are user_state, event_fun and acc_fun.
user_state is any term which can be accessed from event_fun and acc_fun. event_fun is called for every event during the SAX parsing. acc_fun is called for every element (after parsing end of element).
event_fun has a signature fun/2. First parameter is of type #xmerl_event. Second parameter is a global state. user_state from this global state can be accessed via xmerl_scan:user_state/1 function. User state can be modified via xmerl_scan:user_state/2 function. event_fun should return the new global state.
Example of event_fun:
event_function(#xmerl_event{event = Event, data = Data}, GlobalState) ->
UserState = xmerl_scan:user_state(GlobalState),
NewUserState = do_something(UserState),
NewGlobalState = xmerl_scan:user_state(NewUserState, GlobalState),
NewGlobalState.
Possible values for #xmerl_event.event are started and ended.
#xmerl_event.data is set to document at the beginning of document parsing and after all the elements are parsed.
event_function( #xmerl_event{event = started, data = document}, GlobalState) ->
% get the user state
% do something with the user state
% set the new user state in global state
% return global state
One may want to open a file or open a socket connection at the beginning of the document parsing. In subsequent event, one may want to write the content to the previously opened file or socket connection.
End of document parsing event can be used to close the previously opened file handle or socket connection.
acc_fun can be used to accumulate the content. acc_fun is of the form fun/3. First parameter is the current xml element. Second parameter is the accumulator. Third parameter is the global state.
This function should return a new tuple of 2 elements. First element is the new accumulator and 2nd element is a new global state (user state can be changed in the new global state as explained earlier).
user_state can be any Erlang term.
In the next article, I will discuss an implementation of SAX parsing to transform an XML into CSV.
What is the longest word in English?
Longest english word is "Pneumonoultramicroscopicsilicovolcanocon". As per Wikipedia,
More here
a factitious word alleged to mean 'a lung disease caused by the inhalation of very fine silica dust, causing inflammation in the lungs.
More here
Monday, April 6, 2009
Controlling a node from stop script
There is no command line option on Erlang to terminate a node in a graceful manor and report the same. Here is a module I came up with.
-module (node_ctrl).
-export ([stop/1]).
stop([Node]) ->
io:format( "Stopping ~p: ", [Node] ),
case net_kernel:connect_node(Node) of
false -> io:format( "not reachable~n", [] );
true ->
net_kernel:monitor_nodes(true),
rpc:call(Node, init, stop, [] ),
receive
{nodedown, Node} -> io:format( "done~n", [])
after 20000 -> io:format( "refused to die~n", [])
end
end,
init:stop().
Usage:
erl -name foo@example.foo.com -setcookie mycookie -s node_ctrl stop "targetnode@example.foo.com"
Assumptions:
* foo@example.foo.com is started with mycookie
-module (node_ctrl).
-export ([stop/1]).
stop([Node]) ->
io:format( "Stopping ~p: ", [Node] ),
case net_kernel:connect_node(Node) of
false -> io:format( "not reachable~n", [] );
true ->
net_kernel:monitor_nodes(true),
rpc:call(Node, init, stop, [] ),
receive
{nodedown, Node} -> io:format( "done~n", [])
after 20000 -> io:format( "refused to die~n", [])
end
end,
init:stop().
Usage:
erl -name foo@example.foo.com -setcookie mycookie -s node_ctrl stop "targetnode@example.foo.com"
Assumptions:
* foo@example.foo.com is started with mycookie
Friday, April 3, 2009
Erlang XML parser comparison
I am looking into various XML parsing in Erlang. I found mainly 3 of them.
* Xmerl from Erlang distribution
* Erlsom
* Linked-in driver based on libexpat from ejabberd
I did some benchmarking for 3 parsers. Parsing was done on 42K sized XML.
xmerl took 124ms, erlsom took 28ms and linked-in driver based parser took 7ms.
libexpat based parser is the fastest. But it has some drawbacks.
* It cannot do callbacks for SAX parser as linked-in driver cannot do rpc call into the host VM. So it is not good for parsing huge XML data.
* With this one will loose the platform independence. Need to compile the linked-in driver for the platform one is working on.
Erlsom seems to be better than the default xmerl parser.
* It also generate a Erlang data structure (tuples and list).
* Provides continuation function callback when data not enough data is there for parsing.
* Convert the XML to erlang data structure as per the XS
* SAX based parsing
* Some limitations are listed here
Linked-in driver libexpat based parser is the fastest one. It is not flexible enough. It returns list of tuples where first element of tuple is an integer which indicate the begining of element, end of element and cdata/character content. Some parser is necessary to convert this to xmerl structure or any other desired structure.
* Since this does not provide callbacks, it is not desirable to parse huge files
* DOM generated need re-parsing
* Since it is libexpat based parser, it check for utf-8 validity etc.
I heard that next version of xmerl is going to be faster than what it is now. I have not gotten the latest xmerl parser yet. Once I have it, I will do the benchmarking and do another post on my findings.
UPDATE: xmerl also support validating XML against XSD via xmerl_xsd module.
{ok, Xml} = xmerl_scan:string(XmlString),
{ok, Schema} = xmerl_xsd:proces_schema(XsdFile),
xmerl_xsd:validate(Xml, Schema)
UPDATE: Tried xmerl_scan:file/1 with R13 release. It is a significant improvement in performance. Now it can do 44K file in 35-40ms. Still slower than erlsom and expat based parser.
* Xmerl from Erlang distribution
* Erlsom
* Linked-in driver based on libexpat from ejabberd
I did some benchmarking for 3 parsers. Parsing was done on 42K sized XML.
xmerl took 124ms, erlsom took 28ms and linked-in driver based parser took 7ms.
libexpat based parser is the fastest. But it has some drawbacks.
* It cannot do callbacks for SAX parser as linked-in driver cannot do rpc call into the host VM. So it is not good for parsing huge XML data.
* With this one will loose the platform independence. Need to compile the linked-in driver for the platform one is working on.
Erlsom seems to be better than the default xmerl parser.
* It also generate a Erlang data structure (tuples and list).
* Provides continuation function callback when data not enough data is there for parsing.
* Convert the XML to erlang data structure as per the XS
* SAX based parsing
* Some limitations are listed here
Linked-in driver libexpat based parser is the fastest one. It is not flexible enough. It returns list of tuples where first element of tuple is an integer which indicate the begining of element, end of element and cdata/character content. Some parser is necessary to convert this to xmerl structure or any other desired structure.
* Since this does not provide callbacks, it is not desirable to parse huge files
* DOM generated need re-parsing
* Since it is libexpat based parser, it check for utf-8 validity etc.
I heard that next version of xmerl is going to be faster than what it is now. I have not gotten the latest xmerl parser yet. Once I have it, I will do the benchmarking and do another post on my findings.
UPDATE: xmerl also support validating XML against XSD via xmerl_xsd module.
{ok, Xml} = xmerl_scan:string(XmlString),
{ok, Schema} = xmerl_xsd:proces_schema(XsdFile),
xmerl_xsd:validate(Xml, Schema)
UPDATE: Tried xmerl_scan:file/1 with R13 release. It is a significant improvement in performance. Now it can do 44K file in 35-40ms. Still slower than erlsom and expat based parser.
Subscribe to:
Posts (Atom)