Controlling a node from stop script

There is no command line option on Erlang to terminate a node in a graceful manor and report the same. Here is a module I came up with.

-module (node_ctrl).

-export ([stop/1]).

stop([Node]) ->
io:format( "Stopping ~p: ", [Node] ),
case net_kernel:connect_node(Node) of
false -> io:format( "not reachable~n", [] );
true ->
net_kernel:monitor_nodes(true),
rpc:call(Node, init, stop, [] ),
receive
{nodedown, Node} -> io:format( "done~n", [])
after 20000 -> io:format( "refused to die~n", [])
end
end,
init:stop().


Usage:

erl -name foo@example.foo.com -setcookie mycookie -s node_ctrl stop "targetnode@example.foo.com"

Assumptions:
* foo@example.foo.com is started with mycookie

Erlang XML parser comparison

I am looking into various XML parsing in Erlang. I found mainly 3 of them.

* Xmerl from Erlang distribution
* Erlsom
* Linked-in driver based on libexpat from ejabberd

I did some benchmarking for 3 parsers. Parsing was done on 42K sized XML.

xmerl took 124ms, erlsom took 28ms and linked-in driver based parser took 7ms.

libexpat based parser is the fastest. But it has some drawbacks.
* It cannot do callbacks for SAX parser as linked-in driver cannot do rpc call into the host VM. So it is not good for parsing huge XML data.
* With this one will loose the platform independence. Need to compile the linked-in driver for the platform one is working on.

Erlsom seems to be better than the default xmerl parser.
* It also generate a Erlang data structure (tuples and list).
* Provides continuation function callback when data not enough data is there for parsing.
* Convert the XML to erlang data structure as per the XS
* SAX based parsing
* Some limitations are listed here

Linked-in driver libexpat based parser is the fastest one. It is not flexible enough. It returns list of tuples where first element of tuple is an integer which indicate the begining of element, end of element and cdata/character content. Some parser is necessary to convert this to xmerl structure or any other desired structure.
* Since this does not provide callbacks, it is not desirable to parse huge files
* DOM generated need re-parsing
* Since it is libexpat based parser, it check for utf-8 validity etc.

I heard that next version of xmerl is going to be faster than what it is now. I have not gotten the latest xmerl parser yet. Once I have it, I will do the benchmarking and do another post on my findings.

UPDATE: xmerl also support validating XML against XSD via xmerl_xsd module.
{ok, Xml} = xmerl_scan:string(XmlString),
{ok, Schema} = xmerl_xsd:proces_schema(XsdFile),
xmerl_xsd:validate(Xml, Schema)

UPDATE: Tried xmerl_scan:file/1 with R13 release. It is a significant improvement in performance. Now it can do 44K file in 35-40ms. Still slower than erlsom and expat based parser.

RPC from Erlang Linked-in driver port

Developing SAX based XML parser (using libexpat) as a linked-in driver for Erlang, I came across the requirement to do the callback within linked-in driver. I have done it earlier in the c-node via ei_rpc C API. This require linked-in driver to run as a c-node. I was looking for a way to avoid it.

Then I came across this function driver_send_term. This can be used to send a term to any PID within the local VM. It is easy to send the term to the same process which did port_cmd. The PID for that process can be obtained using driver_caller C API.

My requirement is to send it to some other PID than the calling process. There is no standard API to do so. But going through the source code for erlang, I figured the way to convert erlang_pid sent by the caller to the one usable within driver_send_term call.

ErlDrvTermData pid = ((ErlDrvTermData) ( ((callbackPid.serial << 15 | callbackPid.num)) << 4 | (0x0 << 2 | 0x3)) );

This works with R12 and R13 version of Erlang. This is not guaranteed to work in the future releases. But till then I am going to use it.

I wonder why Erlang did not provide the standard interface to convert erlang_pid to ErlDrvTermData to be used within driver_send_term.

Sharing binary data and reference counting

This email thread discussion concludes that

constants in erlang code are stored in a constant pool memory instead of process memory. So there is no copying of this data in the process memory. This is more efficient when one knows that the data is not going to change.

Example for this kind of data is configuration. If configuration is known at the compile time, well and good. But what if the configuration is read from the file during run-time. This data will get copied into the process memory.

I got the suggestion to generate the code during run-time and load that code. Thus erlang VM will ensure that these configuration elements are in constant pool memory and the data is not copied into process memory.

I decided to give it a try. Generated the code and stored it in a file.

ConfigFetcher = list_to_atom("fetch_" ++ atom_to_list(Module) ++ "_config"),
FileName = code:priv_dir(Module) ++ "/" ++ atom_to_list(ConfigFetcher) ++ ".erl",
file:write_file(FileName, GeneratedCode).

Compiled it using

compile:file(FileName, [{outdir, code:lib_dir(Module, ebin)}]).

Load the compiled file,

code:load_file(ConfigFetcher)

Let's say that fetch is a function exported in the generated module. Using this fetch function for fetching the configuration for a given value, ran a small test to find the performance improvement.

The numbers I saw were mind boggling. Using old method I could get 11k fetches per second. With new method (code generation and loading), I got 500k fetcher per second.

AMPQ Client gotchas!

There is a small change required in this article on Introducing The Erlang AMQP Client.

In this article, code for subscribing to an queue is

#'basic.consume_ok'{consumer_tag = ConsumerTag}
= amqp_channel:call(Channel, BasicConsume, self()),

This throws exception

Channel 1 is shutting down due to: {{badmatch,false},
[{rabbit_writer,assemble_frames,4},
{rabbit_writer,
internal_send_command_async,5},
{rabbit_writer,handle_message,2},
{rabbit_writer,mainloop,1}]}


Change it to amqp_channel:subscribe(Channel, BasicConsume, self())

NOTE: Download rabbitmq erlang client from here. The version from this page does not work.

TCP server in Erlang

Came across this good article on writing a TCP server in Erlang. Author forgot to mention one thing here.

connect(Listen) ->
{ok, Socket} = gen_tcp:accept(Listen),
inet:setopts(Socket, ?TCP_OPTS),
% kick off another process to handle connections concurrently
spawn(fun() -> connect(Listen) end),
recv_loop(Socket),
gen_tcp:close(Socket).

In the above code snipper (from the above mentioned blog), it is to be noted that after getting the connection via gen_tcp:accept/1, a process is spawned to continue listen on the socket and the client is served from the same process unlike in any other language where a thread is spawned to serve the incoming client request and main thread continue to listen on the socket. This is very important to note as the incoming client socket has a ownership relationship with the process. If you spawn a process to serve the client, it won't work as the messages won't be delivered to the new process instead are delivered to the process in which gen_tcp:accept/1 was executed.

string:tokens/2 does not handle empty tokens

string:tokens("A,,,,,", ",") returns ["A"] where as I expect it to return ["A", "", "", "", ""]. This can be solved using regexp:split function.

regexp:split("A,,,,,", ",") returns ["A", [], [], [], []]