Giapnguyen's Blog

I’m busy to deploy Network/System monitoring system and IDS for my company, so I stop work on the project for a while…
Install erlang for Ubuntu
sudo apt-get install build-essential subversion
sudo apt-get install java-gcj-compat java-gcj-compat-dev
sudo apt-get install unixodbc-dev unixodbc-bin unixodbc
sudo apt-get install openssl libssl-dev
sudo apt-get install libsctp-dev libncurses5 libncurses5-dev

wget http://erlang.org/download/otp_src_R13B01.tar.gz
tar -xzvf otp_src_R13B01.tar.gz
cd otp_src_R13B01
./configure \
–enable-threads –enable-smp-support –enable-kernel-poll \
–enable-hipe –enable-sctp –prefix=/opt/erlang
make
sudo make install
for file in erl erlc epmd run_erl to_erl dialyzer typer escript; do
sudo ln -s /opt/erlang/lib/erlang/bin/$file /usr/bin/$file;
done

After a week to struggle with jquery (js framework) and nitrogen, I able to write some small web 2 applications. But it is appear that a trading client is more complex than the system!!!

First the web application not suite for a high reliable trading platform – few of trading softwares written in web.

Second, I want implement a push-technology not a polling one (use in my company Vb.net software). So it double my work.

Main idea is a User agent process. When a user login, a User agent which global register with username created.  The  User agent bond with Session so if user logout or session timeout, it will be destroyed. Next time, user must login again to create a new User agent. So a User agent represent for a login user.

With global register name, there is a unique user agent per a user existed. If the same user login from other computer, the current user agent kicked out. Current user logout with error message – login from other computer(like yahoo email).

Pages interact with User agent by subscribe an event. User agent will push events to subscribe pages. Then the page push event to user browser via comet connection.

Other method to interact with user agent is job. Job is a block process, User agent will stop handle other event until the job not completed nor timeout (or crash). Job also stopped  if user agent destroyed.

I hated to write a web application; tons of javascript, html and css code – mostly for decorate a button or a form

Another thing is performance, web application is very very slow…

I found an awesome article about a guy push 1 million comet connections on a single mochiweb server. It is fucking amazing. A million comet connections mean 1 million concurrent active connections – it cost more resource than 1 million normal web requests which close connection after finish their job. There are 3 parts of the article – step by step solve the million connection problems. It’s interesting that most of tuning jobs are for Linux not erlang its self. Erlang born for huge numbers of concurrent processes.

http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-1/

Another awesome thing is Nitrogen framework http://nitrogenproject.com

Nitrogen is pure erlang web 2 framework. You can code a web application like eat a piece of cake , they have a sample of comet chat room just 57 loc . Nitrogen support jquery as client, yaws or mochiweb as server and long-poll comet technical.

Of cause nitrogen can’t handle 1 million comet as mochiweb. It is complex framework and not optimize yet. But it save my ass for coding web application and I can patch nitrogen later for performance. Nitrogen also contain some interesting piece of code. Sync and mirror module allow migrate an erlang application from computer to computer automatically. Great, I can use it to write a code management for my application. I can update a node and whole system can work without more work.

I draw a whole picture for my application.

After play around with Rabbitmq, I’m back to business logic. I finished contract engine module which receive match data from match engine module to generate contracts. Nice! next move is accounting module with debit/credit user accounts.
I also look for a client technology. Mochiweb/Erlycomet very promise. It likes lightstreamer in Net.
http://www.slideshare.net/moschel/comet-overview
http://cometdaily.com/2007/12/14/getting-started-with-comet-on-erlang/

http://cometdaily.com/2008/01/03/comet-gazing-memory-usage/

ErlyComet is a Comet server written in Erlang. Memory usage per connection starts at 43 KB, measured from the ErlyComet (development snapshot) demo application. That allows more than 20,000 connections per GB of RAM. Not included in this view is the in-memory database, which maps connection identification to process identification and might contain other shared information. Its size with the demo application is marginal, but with a complex application, its memory consumption can easily grow beyond the total memory consumed by the connection processes. How memory consumption increases when all connections are simultaneously exchanging messages needs to be determined with stress testing.

To reduce overall memory consumption, there are some Erlang specific tricks:

    * For text, binaries should be used whenever possible for the internal representation, instead of lists.

    * For the in-memory database, partitioned tables (across nodes) can be used instead of per-node replicated tables.

I choose RabbitMQ because

• It have big community (easy to find an answer).
• It able to persistent messages prevent loss from restart MQ server.
• It write in erlang (easy to patch RabbitMQ for custom needs).
• It have a erlang client.

First test – broker send an order via RabbitMQ to x_server.The x_server is a gen_server which wait to receive order from RabbitMQ. The server will call x_engine to process the order.

I’ve just finished a match orders engine module (erlang) and look around for a Message Queuing software to integrate.

My list is

- Zeromq http://www.zeromq.org/

- Rabbitmq http://www.rabbitmq.com

Discuss about Message Queue here:

http://wiki.secondlife.com/wiki/Message_Queue_Evaluation_Notes

Bank monitor margin to lock/release cash. If no more cash avaible , bank will warning user. If margin too low, bank will auto-close the trade to return loan.

Equity = Asset – Debt
Equity = Asset0 + Cash – Debt
Equity – Cash = Asset0  – Debt = Margin

%Margin = (Equity-Cash)/Asset0 (current locked margin)

%ExtendMargin = Equity/Asset0 (extensible margin)

Cash0 (free cash) = Equity – Asset0*Fix_Margin%

Bank monitor free cash if it =0, users have not enough cash to maintain their position.

Deadlock,  Inconsistent, Low performance

The platform require bank lock user money before accept/execute orders, but the bank require platform provide market price before it process orders -> easy to death lock, inconsistent. If we avoid from deadlock and inconsistent, we must accept low performance. The platform should not process any order  before bank finish its job.

Accounting of trading gold (ofcause at Vietnam only) very complex.
- You must freeze a margin before placing an order (pending orders).

- In long order, you borrow %money (based on %margin) + your own margin to buy gold. Of cause the gold locked, until you sell to close the order or deposit money to pay debt to withdraw it.

- In short order, you borrow gold from bank to sell for money, but the money + margin is locked just for buy gold back to close the order. Except you deposit gold to unlocked the money.

Trading accounting journal

1. Place an order buy 1 gold price 2000, margin 1%

2 Cancel the pending order

3. Execute the pending order buy 1 gold @ 2000

4. Sell close the order buy 1 gold @ 1990

5. Execute a pending order sell gold @ 2200

6. Buy close the order sell 1 gold @ 2210

7. Deposit money to pay debt to withdraw gold

8. Deposit gold to unlocked money

How bank monitor the risk?

Spawn 10,000 concurrent proccesses to write to mnesia. Erlang crash with a lot of “too many db tables”.

test_basic_3(N) ->
    {T,S} = timer:tc(?MODULE, test_basic_3_spawn,[N]),
    ?debugFmt("test_basic 3: ~p us; ~p",[T,S]).

test_basic_3_spawn(N) when N>0 ->
    spawn(
        fun()->
            Order = #order{
                price=random2(18000,23000),
                id=time2us(now()),
                type=sell,
                size=random2(1,100)},

            mnesia:activity(
                {transaction, infinity},
                fun() ->
                    m_write_pending_order(Order),
                    Order = m_read_pending_order(?SELL_DESK, Order#order.id)
                end)
        end
    ),
    test_basic_3_spawn(N-1);

test_basic_3_spawn(0) ->
    ok.

The problem is each mnesia transaction create a ets table to hold transaction state. 10k transactions will create 10k ets table. It is out of max ets tables -default ~ 1400, so erlang throw  “too many db tables” errors. To solve the problem, we can increase max of ets tables:
erl -env ERL_MAX_ETS_TABLES 100000

Run the test again, no more “too many db tables”