http request pipeline in Erlang

I tried to use Erlang's http module for high concurrent requests. It was not performing well due to pipelining and persistent connection issues. This seems to be solved in R13 version. I figured out how to use the http profiles to do selective pipelining/persistent connections to one server but not for others [if application is sending requests to multiple hosts].

First step in the process is to create a new http profile. It can be done in 2 ways. First one is to run a stand along http connection manager (httpc_manager).

{ok, Pid} = inets:start( httpc, [{profile, other}] ).

As per the documentation, this is not desirable as all benefits of OTP framework is lost.

Dynamically started services will not be handled by application takeover and failover behavior when inets is run as a distributed application. Nor will they be automatically restarted when the inets application is restarted, but as long as the inets application is up and running they will be supervised and may be soft code upgraded. Services started as stand_alone, e.i. the service is not started as part of the inets application, will lose all OTP application benefits such as soft upgrade. The "stand_alone-service" will be linked to the process that started it. In most cases some of the supervision functionality will still be in place and in some sense the calling process has now become the top supervisor

2nd method is to run it as a part of inets application via configuration file

Have a config file with the following content (say inets.config)

[{inets, 
[{services,[{httpc,[{profile, server1}]},
            {httpc, [{profile, server2}]}]}]
}].

Run the erlang shell as

erl -config inets.config

This will start 3 http profiles [server1, server2 and default].

Now the question is how to use the newly created profiles. Let's say the application is using 2 web services hosted at foo1.example.com and foo2.example.com. Web service hosted at foo1.example.com is hosted on a web server which can support lot of persistent connections [keep alive connections]. Web service hosted foo2.example.com is hosted on a normal web server which is not optimized for large number of persistent connectinons.

In the application set the profile for server1 for the connections to foo1.example.com. This can be done by changing the http options listed here.

http:set_options([{max_sessions, 20}, {pipeline_timeout, 20000}], server1).

NOTE:It is required to set the pipeline timeout in order to enable http pipelining.

Profile can be specified during the request time.

http:request( "http://foo1.example.com/v1/get_info/dudefrommangalore", server1).

There is no interface provided by httpc_manager or inets to get the info on the number of sessions open to a server. But good news is that the session information is kept in the ets table. One can query the ets table to get the list of persistent connections.

ets:tab2list(httpc_manager_server1_session_db).

Output is something like

{tcp_session,{{"fo11.example.com",80},
               <0.103.0>},
              false,http,#Port<0.1032>,...}

<0.103.0> is a Pid of httpc_handler gen server process. It is possible to get the status of this process via standard OTP sys module.

sys:get_status(erlang:list_to_pid("<0.103.0>")).

It is also possible to get all the pipelined requests on each persistent connections. For that it is necessary to get the pid of the httpc_manager via inets:services_info(). This call will return the pid of the httpc_manager.

[{httpc,<0.52.0>,[{profile,server1}]},
 {httpc,<0.53.0>,[{profile,server2}]},
 {httpc,<0.41.0>,[{profile,default}]}]

From the pid, get the status of httpc_manager gen server process.

sys:get_status(erlang:list_to_pid( "<0.52.0>")).

ets table name is in bold here.

15> sys:get_status(erlang:list_to_pid("<0.52.0>")).
{status,<0.52.0>,
        {module,gen_server},
        [[{'$ancestors',[httpc_profile_sup,httpc_sup,inets_sup,
                         <0.36.0>]},
          {'$initial_call',{httpc_manager,init,1}}],
         running,<0.40.0>,[],
         [httpc_manager_server1,
          {state,[],24596,
                 {undefined,28693},
                 httpc_manager_server1_session_db,httpc_manager_server1,
                 {options,{undefined,[]},
                          0,2,5,120000,2,disabled,false,inet,default,...}},
          httpc_manager,infinity]]}

Get the content of the ets table to get the pipelined connection

ets:tab2list(24596).

Application can tune the http options to utilize the network bandwidth better, get the most of the machine and network.

Erlang process mailbox performance

I came across this performance issue in Erlang while doing the pattern matching against the mailbox [a.k.a. selective message processing]. Here is the orignal code:

-module (perf).

-export( [start/0] ).

start() ->
  S = erlang:now(),
  Pids = spawn_n(fun test/1, 10000, []),
  wait(Pids),
  E = erlang:now(),
  io:format( "Total time: ~p~n", [timer:now_diff(E, S)/1000] ).

spawn_n(_F, 0, Acc) -> Acc;
spawn_n(F, N, Acc) -> 
  Me = self(),
  Pid = spawn(fun() -> F(Me) end),
  spawn_n(F, N-1, [Pid|Acc]).

test(Pid) -> Pid ! {self(), ok}.

wait([]) -> ok;
wait([Pid|Pids]) -> 
    receive {Pid, ok} -> ok end,
    wait(Pids).

Run time for perf:start() was 1.3 seconds

Erlang (BEAM) emulator version 5.6.5 [source] [smp:2] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.6.5 (abort with ^G)
1> perf:start().
Total time: 1368.038
ok
2>

Now I changed wait(Pids) to wait(lists:reverse(Pids)). After this change, run time for perf:start() was 83 milliseconds.

1> perf:start().
Total time: 83.037
ok

15x improvement just by changing the way mailbox scan is done.

Little things like this are usually overlooked and the language is blamed for the performance issues.

How to see the running processes in erlang shell

Erlang provides a built-in utility called etop, that is same as standard top utility on Unix. Instead of monitoring processes on the operating system, etop show the current running/active Erlang processes. By default it starts a GUI window to show the processes running. Sometimes GUI is not desirable especially when the Erlang shell is running on production servers. Documentation does not show any way to start etop in text mode. Here is how to run it

etop:start([{output, text}]).

Load balancing in Erlang using pg2 process groups

While working with some project @ my current workplace, we came across a situation where we are using pg2 process group. Apparently pg2 does not load balance across processes registered in the pg2 group. It usually done in round robin way. So it may happen that even though many processes are registered under same group, but all messages will get queues in single process.

My co-worker and I came with an algorithm to solve this. It is describe here.

Clustering RabbitMQ servers for High Availability

Clustering guide on rabbitmq website is a good start. But it does not seem to work out of the box.

Default rabbitmq-server script uses -sname (short node name) option on the erlang shell command line. This makes it impossible to setup the cluster as it is not possible to put a node into the cluster when short name is given. So first step is remove that line from the rabbitmq-server script

By default erlang shell generate a random cookie and save that in the .erlang.cookie file in the home directory. Since the cookie generated is random, unless the same cookie file copied onto the other nodes, it is not possible to setup the cluster. Better idea is to set the cookie on the command line itself. This can done via the environment variable RABBITMQ_SERVER_START_ARGS.

Also by default, rabbitmq-server do not set the full node name. One need to specify that on the command line argument. Better place is to set the environment variable RABBITMQ_SERVER_START_ARGS (same as the one used to set the cookie)

Erlang's DNS lookup is done via inet module. By default it uses /etc/hosts as the first one to resolve the hostname. Therefore in some environments (at least in my environment), short name does not resolve to long name when I use inet:gethostbyname(Shortname) function. I get short name back in hostent structure. Workaround for this is to force the use of native library for dns lookup. This can be done via inetrc file. By default Erlang uses $HOME/.inetrc file (if exists). Non-standard location can be specified via ERL_INETRC environment variable. Don't forget to set the full path name.

Content of the inetrc file will look like

{lookup, [native]}.

Other possible options are file,yp,dns.

It is good to know some other environment variables.

RABBITMQ_LOG_BASE -> log location (default /var/log/rabbitmq)
RABBITMQ_NODENAME -> node name to be used (default rabbit)
RABBITMQ_MNESIA_BASE -> location for mnesia database (default /var/lib/rabbitmq/mnesia)

If your machine has multiple network interfaces, by default RabbitMQ binds to all the network interfaces. In some cases it is not desirable. One can force it to bind to only one network interface by setting the environment varaible RABBITMQ_NODE_IP_ADDRESS to the ip address of the network interface.

Here is a summary of all environment variables required

RABBITMQ_SERVER_START_ARGS="-name rabbit@`hostname` -setcookie rabbit"
RABBITMQ_LOG_BASE=/home/baliga/rabbit
RABBITMQ_NODENAME=rabbit
RABBITMQ_MNESIA_BASE=/home/baliga/rabbit
ERL_INETRC=/home/baliga/inetrc

Now it is time to start rabbitmq server on all the nodes.

rabbitmq-server -detached

To setup the cluster rabbitmqctl script is used. By default this script too has -sname option setup on the command line. Delete this line from rabbitmqctl file. In order for this script to communicate with the node, long name and cookie required. Set this via RABBITMQ_CTL_ERL_ARGS environment variable

export RABBITMQ_CTL_ERL_ARGS="-name rabbitmqctl@`hostname` -setcookie rabbit"

Note that the cookie should be same as that of the rabbitmq-server instance.

Now you can run the rabbitmqctl to setup the cluster.

On each of the node, run the following command

1. rabbitmqctl stop_app
2. rabbitmqctl reset
3. rabbitmqctl cluster
4. Repeat step 3 for all nodes in the cluster except for itself.
4. rabbitmqctl start_app

Now the cluster is ready to be used.

Some questions are not answered here like how the clustering works when nodes are behind separate firewalls. What ports need to be opened in firewall for clustering to work. I am still doing the research on it. Will post the findings soon.

SAX Xml parsing in Erlang - Part 1

Erlang has a default XML parser called xmerl. Even there is a documetation associated with this module, it is not adequate enough to start writing the code. Especially SAX parsing is the least documented and least understood. There is not much literature (infact no documentation at all) for the SAX parsing. I had to do lot of code reading to figure out the SAX parsing module.

In this article I will make an attempt to explain the SAX parsing module.

SAX parsing is done via xmerl_eventp:stream/2 and xmerl_eventp:stream_sax/4.

I did not find xmerl_eventp:stream_sax/4 useful as user state is not maintained across all events. Also accumulate function cannot be overridden. So it is not useful when you do not need to accumulate all the elements (for large file parsing). I was trying to use this function to parse large file ( > 10MB ) and extract only some part of the whole file. Since it is not possible to override accumulate, this function was out of question. If one need to accumulate all the elements in the document, I would rather do xmerl_scan:file/1 or xmerl_scan:string/1 which will scan the file/string and generate in-memory document.

xmerl_eventp:stream/2 function take 2 arguments.
* File name
* List of options

For this post, 3 options are important. They are user_state, event_fun and acc_fun.

user_state is any term which can be accessed from event_fun and acc_fun. event_fun is called for every event during the SAX parsing. acc_fun is called for every element (after parsing end of element).

event_fun has a signature fun/2. First parameter is of type #xmerl_event. Second parameter is a global state. user_state from this global state can be accessed via xmerl_scan:user_state/1 function. User state can be modified via xmerl_scan:user_state/2 function. event_fun should return the new global state.

Example of event_fun:

event_function(#xmerl_event{event = Event, data = Data}, GlobalState) ->
UserState = xmerl_scan:user_state(GlobalState),
NewUserState = do_something(UserState),
NewGlobalState = xmerl_scan:user_state(NewUserState, GlobalState),
NewGlobalState.

Possible values for #xmerl_event.event are started and ended.

#xmerl_event.data is set to document at the beginning of document parsing and after all the elements are parsed.

event_function( #xmerl_event{event = started, data = document}, GlobalState) ->
% get the user state
% do something with the user state
% set the new user state in global state
% return global state

One may want to open a file or open a socket connection at the beginning of the document parsing. In subsequent event, one may want to write the content to the previously opened file or socket connection.

End of document parsing event can be used to close the previously opened file handle or socket connection.

acc_fun can be used to accumulate the content. acc_fun is of the form fun/3. First parameter is the current xml element. Second parameter is the accumulator. Third parameter is the global state.

This function should return a new tuple of 2 elements. First element is the new accumulator and 2nd element is a new global state (user state can be changed in the new global state as explained earlier).

user_state can be any Erlang term.

In the next article, I will discuss an implementation of SAX parsing to transform an XML into CSV.

Controlling a node from stop script

There is no command line option on Erlang to terminate a node in a graceful manor and report the same. Here is a module I came up with.

-module (node_ctrl).

-export ([stop/1]).

stop([Node]) ->
io:format( "Stopping ~p: ", [Node] ),
case net_kernel:connect_node(Node) of
false -> io:format( "not reachable~n", [] );
true ->
net_kernel:monitor_nodes(true),
rpc:call(Node, init, stop, [] ),
receive
{nodedown, Node} -> io:format( "done~n", [])
after 20000 -> io:format( "refused to die~n", [])
end
end,
init:stop().


Usage:

erl -name foo@example.foo.com -setcookie mycookie -s node_ctrl stop "targetnode@example.foo.com"

Assumptions:
* foo@example.foo.com is started with mycookie

Erlang XML parser comparison

I am looking into various XML parsing in Erlang. I found mainly 3 of them.

* Xmerl from Erlang distribution
* Erlsom
* Linked-in driver based on libexpat from ejabberd

I did some benchmarking for 3 parsers. Parsing was done on 42K sized XML.

xmerl took 124ms, erlsom took 28ms and linked-in driver based parser took 7ms.

libexpat based parser is the fastest. But it has some drawbacks.
* It cannot do callbacks for SAX parser as linked-in driver cannot do rpc call into the host VM. So it is not good for parsing huge XML data.
* With this one will loose the platform independence. Need to compile the linked-in driver for the platform one is working on.

Erlsom seems to be better than the default xmerl parser.
* It also generate a Erlang data structure (tuples and list).
* Provides continuation function callback when data not enough data is there for parsing.
* Convert the XML to erlang data structure as per the XS
* SAX based parsing
* Some limitations are listed here

Linked-in driver libexpat based parser is the fastest one. It is not flexible enough. It returns list of tuples where first element of tuple is an integer which indicate the begining of element, end of element and cdata/character content. Some parser is necessary to convert this to xmerl structure or any other desired structure.
* Since this does not provide callbacks, it is not desirable to parse huge files
* DOM generated need re-parsing
* Since it is libexpat based parser, it check for utf-8 validity etc.

I heard that next version of xmerl is going to be faster than what it is now. I have not gotten the latest xmerl parser yet. Once I have it, I will do the benchmarking and do another post on my findings.

UPDATE: xmerl also support validating XML against XSD via xmerl_xsd module.
{ok, Xml} = xmerl_scan:string(XmlString),
{ok, Schema} = xmerl_xsd:proces_schema(XsdFile),
xmerl_xsd:validate(Xml, Schema)

UPDATE: Tried xmerl_scan:file/1 with R13 release. It is a significant improvement in performance. Now it can do 44K file in 35-40ms. Still slower than erlsom and expat based parser.

RPC from Erlang Linked-in driver port

Developing SAX based XML parser (using libexpat) as a linked-in driver for Erlang, I came across the requirement to do the callback within linked-in driver. I have done it earlier in the c-node via ei_rpc C API. This require linked-in driver to run as a c-node. I was looking for a way to avoid it.

Then I came across this function driver_send_term. This can be used to send a term to any PID within the local VM. It is easy to send the term to the same process which did port_cmd. The PID for that process can be obtained using driver_caller C API.

My requirement is to send it to some other PID than the calling process. There is no standard API to do so. But going through the source code for erlang, I figured the way to convert erlang_pid sent by the caller to the one usable within driver_send_term call.

ErlDrvTermData pid = ((ErlDrvTermData) ( ((callbackPid.serial << 15 | callbackPid.num)) << 4 | (0x0 << 2 | 0x3)) );

This works with R12 and R13 version of Erlang. This is not guaranteed to work in the future releases. But till then I am going to use it.

I wonder why Erlang did not provide the standard interface to convert erlang_pid to ErlDrvTermData to be used within driver_send_term.

Sharing binary data and reference counting

This email thread discussion concludes that

constants in erlang code are stored in a constant pool memory instead of process memory. So there is no copying of this data in the process memory. This is more efficient when one knows that the data is not going to change.

Example for this kind of data is configuration. If configuration is known at the compile time, well and good. But what if the configuration is read from the file during run-time. This data will get copied into the process memory.

I got the suggestion to generate the code during run-time and load that code. Thus erlang VM will ensure that these configuration elements are in constant pool memory and the data is not copied into process memory.

I decided to give it a try. Generated the code and stored it in a file.

ConfigFetcher = list_to_atom("fetch_" ++ atom_to_list(Module) ++ "_config"),
FileName = code:priv_dir(Module) ++ "/" ++ atom_to_list(ConfigFetcher) ++ ".erl",
file:write_file(FileName, GeneratedCode).

Compiled it using

compile:file(FileName, [{outdir, code:lib_dir(Module, ebin)}]).

Load the compiled file,

code:load_file(ConfigFetcher)

Let's say that fetch is a function exported in the generated module. Using this fetch function for fetching the configuration for a given value, ran a small test to find the performance improvement.

The numbers I saw were mind boggling. Using old method I could get 11k fetches per second. With new method (code generation and loading), I got 500k fetcher per second.

AMPQ Client gotchas!

There is a small change required in this article on Introducing The Erlang AMQP Client.

In this article, code for subscribing to an queue is

#'basic.consume_ok'{consumer_tag = ConsumerTag}
= amqp_channel:call(Channel, BasicConsume, self()),

This throws exception

Channel 1 is shutting down due to: {{badmatch,false},
[{rabbit_writer,assemble_frames,4},
{rabbit_writer,
internal_send_command_async,5},
{rabbit_writer,handle_message,2},
{rabbit_writer,mainloop,1}]}


Change it to amqp_channel:subscribe(Channel, BasicConsume, self())

NOTE: Download rabbitmq erlang client from here. The version from this page does not work.

TCP server in Erlang

Came across this good article on writing a TCP server in Erlang. Author forgot to mention one thing here.

connect(Listen) ->
{ok, Socket} = gen_tcp:accept(Listen),
inet:setopts(Socket, ?TCP_OPTS),
% kick off another process to handle connections concurrently
spawn(fun() -> connect(Listen) end),
recv_loop(Socket),
gen_tcp:close(Socket).

In the above code snipper (from the above mentioned blog), it is to be noted that after getting the connection via gen_tcp:accept/1, a process is spawned to continue listen on the socket and the client is served from the same process unlike in any other language where a thread is spawned to serve the incoming client request and main thread continue to listen on the socket. This is very important to note as the incoming client socket has a ownership relationship with the process. If you spawn a process to serve the client, it won't work as the messages won't be delivered to the new process instead are delivered to the process in which gen_tcp:accept/1 was executed.

string:tokens/2 does not handle empty tokens

string:tokens("A,,,,,", ",") returns ["A"] where as I expect it to return ["A", "", "", "", ""]. This can be solved using regexp:split function.

regexp:split("A,,,,,", ",") returns ["A", [], [], [], []]

Imperative language constructs in Erlang

"How can I do for loop in Erlang?", "How can I do while loop in Erlang?"....

These are the common questions people ask when they get into the world of functional programming from imperative language(s). It is easy to switch to functional programming in Python, Ruby etc as they are the mixture of functional and imperative language constructs. Programming languages like Erlang (probably Haskell) do not provide explicit for, while constructs. Here are the equivalents way to implement some of the imperative language constructs.

For Loop

for_loop(N, Fun) ->
lists:foreach(Fun, lists:seq(1, N))

for_loop(0, Fun) -> ok;
for_loop(N, Fun) ->
Fun(N),
for_loop(N-1, Fun).

While loop

while_loop(Predicate, Fun, State) ->
while_loop(Predicate(State), Predicate, Fun, State).

while_loop(PredResult, Predicate, Fun, State) when PredResult == false -> ok;
while_loop(_PredResult, Predicate, Fun, State) ->
NewState = Fun(State),
whle_loop(PredResult(State), Predicate, Fun, NewState).

Auto increment in Mnesia database

Mnesia provides some obscure API called dirty_increment_update/3. This can be used to generate unique ids. Here is an example.

-module (uniqueid).

-export( [test/0] ).

-record( unique_ids, {type, id} ).

test() ->
mnesia:start(),
mnesia:create_table( unique_ids, [{attributes, record_info(fields, unique_ids)}] ),
Id = mnesia:dirty_update_counter(unique_ids, record_type, 1),
io:format( "Id => ~p~n", [Id] ),
Id1 = mnesia:dirty_update_counter(unique_ids, record_type, 1),
io:format( "Id => ~p~n", [Id1] ),
Id2 = mnesia:dirty_update_counter(unique_ids, another_type, 1),
io:format( "Id => ~p~n", [Id2] ),


The output you will get is

Id => 1
Id => 2
Id => 1

A single table can be used to generate the unique ids for other tables. In this example, unique ids are generated for record_type and another_type.

Gracefully terminating erlang VM

Once started usually it is not required to stop erlang VM. We can stop the services running as well as erlang VM in a graceful manner with init:stop/0 function. Trick is to call this function on the target VM via remote shell.

For example, to stop the VM running as foo@example.foo.com with cookie foo

erl -name bar@example.foo.com -remsh foo@example.foo.com -setcookie foo -s init stop

The above command will not work as init:stop() is executed within bar@localhost VM, not in foo@localhost. Another way is to do spawn in the target.

erl -name bar@localhost -remsh foo@localhost -setcookie foo -s proc_lib spawn 'foo@example.foo.com' init stop "" -s init stop

The above command will try to execute proc_lib:spawn( 'foo@example.foo.com', init, stop, []). There is a small caveat here. -s option pass all the paramter to proc_lib:spawn/4 as a list, not as a individual parameter.

proc_lib:spawn( ['foo@example.foo.com', init, stop, [] ] ).

Since there is no such function defined, it will not work.

Only other solution is to create a module with one exported function which will spawn a function init:stop/0 on a remote shell.

-module (ctrl).

-export([stop/1]).

stop(Node) ->
proc_lib:spawn( hd(Node), init, stop, [] ).

Note that ctrl:stop/1 function gets list as an argument and it taking only the head element from it.

erl -name bar@example.foo.com -remsh foo@example.foo.com -setcookie foo -s ctrl stop 'foo@example.foo.com' -s init stop

The above command line execute ctrl:stop(['foo@example.foo.com']), init:stop()

UPDATE: This can be done from command line without using any other module. All you have to do is to use -eval option on erl command line.

erl -name bar@example.foo.com -setcookie foo -eval "rpc:call('foo@example.foo.com', init, stop, []), init:stop()."

ibrowse module for erlang

In the project I am working ( cannot disclose what it is for now ), I had to do a lot of web service calls over http. The server is written in erlang as a gen_server which spawn the process for each request.

Apache is used as the user facing http server. We have written a content handle (apache module) which acts as a c-node by connecting to a local or remote (configured) erlang VM. gen_server in erlang VM registered as a named process (let's say foo). Each incoming user request is sent to the registered process and response from the registered process is sent back to the user (XML or JSON).

Process spawned by gen_server make one or more web service call to the back-end servers. We were using erlang's http client. Apparently it is hard to configure as there is not much documentation available to maintain a persistent connection to the back-end (web service) servers. Thus there is a lot of socket churning as web service calls are http connections and were getting opened and closed. Location of the back-end servers are not necessarily near to our servers (physically). That means there is not only communication overhead but also http connection overhead. If ping time between our server and remote server is 25ms, each http connection will take 75ms (3-way hand shake). Having a persistent connection will solve this problem.

We came across ibrowse module. This has a simple configuration to maintain persistent connection as well as configuration for maximum number of connection and maximum number of http requests that can be pipelined on each connection. This improved our latency by 60% at 400qps and there is plenty of room to grow till 1000qps (on single box).

I can also revert back to erlang's default http client provided I can set the profile for httpc while starting inets. I did not find a decent document to do so. If anyone know about it, let me know.

iolost vs list in Eralng

I started using Erlang few months back. Erlang has very little documentation (in most cases) and no documentation (in some cases).

In Erlang strings are implemented in terms of list. Unlike C, string is a linked list. So every character in a string occupy 2 machine words, one for the character and another for the next pointer. On a 32-bit machine, each byte translates to 8 bytes and on 64-bit box 16 bytes. This is an issue when you are dealing with transporting string over the network.

This becomes an issue when transporting data between erlang VM and c-node. If string is transported as internal representation, erlang will transport 8x number of bytes for a plain string. It is better to transport string as a binary (use erlang:list_to_binary/1 function). This will reduce the network overhead by 8x (on 32-bit machine).

If a string is constructed in VM and is transported to C-node, it is easier to construct a iolist and use iolist_to_binary at the end to transport it as a binary. This will reduce the overhead of string concats and also save a lot on the garbage collection.

For example,

construct_string() ->
[[<<"this is">>, <<" a">>], [" very long", <<" string.">>], <<" This string will go to c-node">>].

transport_to_code() ->
iolist_to_binary(construct_string()).

I saw huge improvement in memory as well as CPU utilization by using binaries instead of strings. Lesson learned is that do not use strings unless it is absolutely required. If binaries will do the work, use it.

Pattern matching for string and list in erlang

Earlier I wrote a post on how to distinguish string from list. Steve and Justinpointed that my approach will not work for a list of anythingg other than string.

Here is what I am trying to solve. I have a main configuratin file and client specific configuration files. Client specific configuration file can override or add more values from main configuration file. If the value in main configuration file is a string, I want the client configuration to override the value of main configuration. If the value in main configuration is a list, I want to append the list from the client configuration to the main configuration.

For example, main configuration file is

[{blacklist, ["foo.com", "aaa.com"]},
{source_host, "http://www.bar.com"}].

and client configuration file is

[{blacklist, [ "bar.com" ]},
{source_host, "http://www.something.com"}].

After merging above 2 configuration, I want the final configuration to look like,

[{blacklist, [ "foo.com", "aaa.com", "bar.com" ]},
{source_host, "http://www.something.com"}].

The code reading the configuration files and merging them does not know the first element of the tuple ahead of time. The code is generic to read a file of list of tuples and merge the values to generate the final configuration.

Since Erlang implement string as a list, using is_list guard for a function will not work. If anybody has solved this problem earlier, please comment on this blog.

Parallel assignment

Today I was working on some problem (off course in Erlang). What I am trying to achieve was to execute a function in parallel and assign result of function to a variable.

Func = fun(Arg) ->
          Parent = self(),
          Pid = spawn( fun() -> timer:sleep(1000), Parent ! {self(), Arg + 2} end ),
          receive {Pid, Val} -> Val
       end,
RetVal = Func(),
SomeOtherValue = 10,
% rest of the code
io:format( "~p~n", [RetVal] ),
% more code here

In the above code, calling Func() block the execution until receive loop within Func is done receiving the data from the spawned process. SomeOtherValue is assigned only after the execution of Func.

What I want is not that. I want the execution to continue even if the execution of Func is not finished. I want to block only when I am trying to use RetVal variable.

Currently I am acheiving this by waiting for the response from the spawned process only when I need the data from the process, i.e., same as blocking when the variable is accessed.

Func = fun(Arg) ->
          Parent = self(),
          Pid = spawn( fun() -> timer:sleep(1000), Parent ! {self(), Arg + 2} end ),
       end,
Pid = Func(),
SomeOtherValue = 10,
% rest of the code
RetVal = receive {self(), Val} -> Val end,
io:format( "~p~n", [RetVal] ),
% more code here

Even though it is a trivial code change, I would like if the parallel assignment is allowed and block when I try to use it (if necessary).