EUnit Testing gen_fsm – Part 1


This post will be TDD based and shows how the Tradepost program is developed and is tested.

First iteration – Start and Stop

I start writing the test! This gives me the opportunity to crystallize my thoughts about the program.

The module implementing the tests of the Tradepost

-module(tradepost_tests).
-include_lib("eunit/include/eunit.hrl").
% This is the main point of "entry" for my EUnit testing.
% A generator which forces setup and cleanup for each test in the testset
main_test_() ->
    {foreach,
     fun setup/0,
     fun cleanup/1,
     % Note that this must be a List of TestSet or Instantiator
     % (I have instantiators == functions generating tests)
     [
      % First Iteration
      fun started_properly/1,
     ]}.

% Setup and Cleanup
setup()      -> {ok,Pid} = tradepost:start_link(), Pid.
cleanup(Pid) -> tradepost:stop(Pid).

% Pure tests below
% ------------------------------------------------------------------------------
% Let's start simple, I want it to start and check that it is okay.
% I will use the introspective function for this
started_properly(Pid) ->
    fun() ->
            ?assertEqual(pending,tradepost:introspection_statename(Pid)),
            ?assertEqual([undefined,undefined,undefined,undefined,undefined],
                         tradepost:introspection_loopdata(Pid))
    end.

Compilation and running should fail as the gen_fsm module is barely minimal

zen:EUnitFSM zenon$ tree .
.
├── ebin
├── include
├── src
│   └── tradepost.erl
└── test
    └── tradepost_tests.erl

4 directories, 2 files
zen:EUnitFSM zenon$ 

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> eunit:test(tradepost).
tradepost_tests: started_properly...*failed*
::error:{assertEqual_failed,
          [{module,tradepost_tests},
           {line,33},
           {expression,"tradepost : introspection_statename ( Pid )"},
           {expected,pending},
           {value,ok}]}
  in function tradepost_tests:'-started_properly/1-fun-0-'/2
  in call from tradepost_tests:'-started_properly/1-fun-2-'/1

module 'tradepost_tests'
*** context cleanup failed ***
::error:undef
  in function tradepost:stop/1
    called as stop(<0.39.0>)

=======================================================
  Failed: 1.  Skipped: 0.  Passed: 0.
One or more tests were cancelled.
error
2>

Clearly the tradepost does not handle the introspective functions and stop, let’s write them first.

The interface

introspection_statename(TradePost) ->
    gen_fsm:sync_send_all_state_event(TradePost,which_statename).
introspection_loopdata(TradePost) ->
    gen_fsm:sync_send_all_state_event(TradePost,which_loopdata).
stop(Pid) -> gen_fsm:sync_send_all_state_event(Pid,stop).

And the handling

handle_sync_event(which_statename, _From, StateName, LoopData) ->
    {reply, StateName, StateName, LoopData};
handle_sync_event(which_loopdata, _From, StateName, LoopData) ->
    {reply,tl(tuple_to_list(LoopData)),StateName,LoopData};
handle_sync_event(stop,_From,_StateName,LoopData) ->
    {stop,normal,ok,LoopData}.

Running

zen:EUnitFSM zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> eunit:test(tradepost,[verbose]).
======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly...ok
    [done in 0.004 s]
  [done in 0.005 s]
=======================================================
  Test passed.
ok
2>


Wow, I’m glad we got that sorted out. Now,  as we have set the first nail in the mountain and hooked us to it, the climb begins in a series of cycles. Next up, the Seller API.

Second Iteration – Seller API

What the seller needs, is a way to insert an item, and a way to remove an item. Also to identify him/her self with a naive password approach. Once identified, the seller (and only the seller) should be able to add and retract items.

The main_test_() has now been expanded to

% This is the main point of "entry" for my EUnit testing.
% A generator which forces setup and cleanup for each test in the testset
main_test_() ->
    {foreach,
     fun setup/0,
     fun cleanup/1,
     % Note that this must be a List of TestSet or Instantiator
     % (I have instantiators)
     [
      % First Iteration
      fun started_properly/1,
      % Second Iteration
      fun identify_seller/1,
      fun insert_item/1,
      fun withdraw_item/1
     ]}.

And the definition of the tests themselves

% Now, we are adding the Seller API tests
identify_seller(Pid) ->
    fun() ->
            % From Pending, identify seller, then state should be pending
            % loopdata should now contain seller_password
            ?assertEqual(pending,tradepost:introspection_statename(Pid)),
            ?assertEqual(ok,tradepost:seller_identify(Pid,seller_password)),
            ?assertEqual(pending,tradepost:introspection_statename(Pid)),
            ?assertEqual([undefined,undefined,seller_password,undefined,
                       undefined],tradepost:introspection_loopdata(Pid))
    end.

insert_item(Pid) ->
    fun() ->
            % From pending and identified seller, insert item
            % state should now be item_received, loopdata should now contain itm
            tradepost:introspection_statename(Pid),
            tradepost:seller_identify(Pid,seller_password),
            ?assertEqual(ok,tradepost:seller_insertitem(Pid,playstation,
                                                  seller_password)),
            ?assertEqual(item_received,tradepost:introspection_statename(Pid)),
            ?assertEqual([playstation,undefined,seller_password,undefined,
                       undefined],tradepost:introspection_loopdata(Pid))
    end.

withdraw_item(Pid) ->
    fun() ->
            % identified seller and inserted item, withdraw item
            % state should now be pending, loopdata should now contain only password
            tradepost:seller_identify(Pid,seller_password),
            tradepost:seller_insertitem(Pid,playstation,seller_password),
            ?assertEqual(ok,tradepost:withdraw_item(Pid,seller_password)),
            ?assertEqual(pending,tradepost:introspection_statename(Pid)),
            ?assertEqual([undefined,undefined,seller_password,undefined,
                       undefined],tradepost:introspection_loopdata(Pid))
    end.

Now, this is a lot of code duplication per test, and  I shall start rewriting the code into something more wieldy, but first: the failed compilation and the code making it pass.

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> eunit:test(tradepost).
tradepost_tests: identify_seller...*failed*
::error:undef
  in function tradepost:seller_identify/2
    called as seller_identify(<0.39.0>,seller_password)
  in call from tradepost_tests:'-identify_seller/1-fun-1-'/2
  in call from tradepost_tests:'-identify_seller/1-fun-4-'/1

tradepost_tests: insert_item...*failed*
::error:undef
  in function tradepost:seller_identify/2
    called as seller_identify(<0.39.0>,seller_password)
  in call from tradepost_tests:'-insert_item/1-fun-3-'/1

tradepost_tests: withdraw_item...*failed*
::error:undef
  in function tradepost:seller_identify/2
    called as seller_identify(<0.39.0>,seller_password)
  in call from tradepost_tests:'-withdraw_item/1-fun-3-'/1

=======================================================
  Failed: 3.  Skipped: 0.  Passed: 1.
error
2>

Some writing and enjoying later, the module

%%%-------------------------------------------------------------------
%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created :  2 Sep 2010 by Gianfranco <zenon@zen.home>
%%%-------------------------------------------------------------------
-module(tradepost).
-behaviour(gen_fsm).

%% API
-export([start_link/0,introspection_statename/1,introspection_loopdata/1,
         stop/1,seller_identify/2,seller_insertitem/3,withdraw_item/2]).

%% States
-export([pending/2,pending/3,item_received/3]).

%% gen_fsm callbacks
-export([init/1, handle_event/3, handle_sync_event/4, handle_info/3,
         terminate/3, code_change/4]).
-record(state, {object,cash,seller,buyer,time}).

%%% API
start_link() -> gen_fsm:start_link(?MODULE, [], []).

introspection_statename(TradePost) ->
    gen_fsm:sync_send_all_state_event(TradePost,which_statename).
introspection_loopdata(TradePost) ->
    gen_fsm:sync_send_all_state_event(TradePost,which_loopdata).
stop(Pid) -> gen_fsm:sync_send_all_state_event(Pid,stop).

seller_identify(TradePost,Password) ->
    gen_fsm:sync_send_event(TradePost,{identify_seller,Password}).
seller_insertitem(TradePost,Item,Password) ->
    gen_fsm:sync_send_event(TradePost,{insert,Item,Password}).

withdraw_item(TradePost,Password) ->
    gen_fsm:sync_send_event(TradePost,{withdraw,Password}).

%%--------------------------------------------------------------------
pending(_Event,LoopData) -> {next_state,pending,LoopData}.

pending({identify_seller,Password},_Frm,LoopD = #state{seller=Password}) ->
    {reply,ok,pending,LoopD};
pending({identify_seller,Password},_Frm,LoopD = #state{seller=undefined}) ->
    {reply,ok,pending,LoopD#state{seller=Password}};
pending({identify_seller,_},_,LoopD) ->
    {reply,error,pending,LoopD};

pending({insert,Item,Password},_Frm,LoopD = #state{seller=Password}) ->
    {reply,ok,item_received,LoopD#state{object=Item}};
pending({insert,_,_},_Frm,LoopD) ->
    {reply,error,pending,LoopD}.

item_received({withdraw,Password},_Frm,LoopD = #state{seller=Password}) ->
    {reply,ok,pending,LoopD#state{object=undefined}};
item_received({withdraw,_},_Frm,LoopD) ->
    {reply,error,item_received,LoopD}.

%%--------------------------------------------------------------------
handle_sync_event(which_statename, _From, StateName, LoopData) ->
    {reply, StateName, StateName, LoopData};
handle_sync_event(which_loopdata, _From, StateName, LoopData) ->
    {reply,tl(tuple_to_list(LoopData)),StateName,LoopData};
handle_sync_event(stop,_From,_StateName,LoopData) ->
    {stop,normal,ok,LoopData};
handle_sync_event(_E,_From,StateName,LoopData) ->
    {reply,ok,StateName,LoopData}.

%%--------------------------------------------------------------------
init([]) -> {ok, pending, #state{}}.
handle_event(_Event, StateName, State) ->{next_state, StateName, State}.
handle_info(_Info, StateName, State) -> {next_state, StateName, State}.
terminate(_Reason, _StateName, _State) -> ok.
code_change(_OldVsn, StateName, State, _Extra) -> {ok, StateName, State}.

And proof for the functioning of it.

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/ -eval 'eunit:test(tradepost,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly...ok
    tradepost_tests: identify_seller...ok
    tradepost_tests: insert_item...ok
    tradepost_tests: withdraw_item...ok
    [done in 0.015 s]
  [done in 0.015 s]
=======================================================
  All 4 tests passed.
1>

Awesome! And we got a bonus as well: how to run eunit directly from the command line with the -eval command. Cool, oneliners always make you feel more “1337”.

It all looks very good so far, but I will use the third iteration to fixing up the eunit module, it has way to much duplication and could be made more declarative (specifying what should be computed – not how [ which is kind of more abstract <of course someone will oppose>]).

Third Iteration –  Eunit GenFSM DSL (do not worry)

Testing gen_fsm’s should ideally be all about testing state transitions, in-state computations and variables. For this purpose, I would like to have my DSL that handles all of the pesky details for me.

To exemplify, I would like to write the first test

started_properly(Pid) ->
    fun() ->
            ?assertEqual(pending,tradepost:introspection_statename(Pid)),
            ?assertEqual([undefined,undefined,undefined,undefined,undefined],
                         tradepost:introspection_loopdata(Pid))
    end.

as something along these lines

started_properly(Pid) ->
    {"Proper startup test",
     [{statename,is,pending},
      {loopdata,is,[undefined,undefined,undefined,undefined,undefined]}
      ]}.

And, this test

insert_item(Pid) ->
    fun() ->
        % From pending and identified seller, insert item
        % state should now be item_received, loopdata should now contain itm
        tradepost:introspection_statename(Pid),
        tradepost:seller_identify(Pid,seller_password),
        ?assertEqual(ok,tradepost:seller_insertitem(Pid,playstation,
                                              seller_password)),
        ?assertEqual(item_received,tradepost:introspection_statename(Pid)),
        ?assertEqual([playstation,undefined,seller_password,undefined,
                   undefined],tradepost:introspection_loopdata(Pid))
    end.

as something along these lines

insert_item(Pid) ->
    {"Insert Item Test",
      [{state,is,pending},
       {call,tradepost,seller_identify,[Pid,seller_password],ok},
       {call,tradepost,seller_insertitem,[Pid,playstation,seller_password]},
       {state,is,item_received},
       {loopdata,is,[playstation,undefined,seller_password,undefined,undefined]}
      ]}.

This is a symbolic method that defines our DSL in a yet readable way, hiding the logic. For this to work, we need a translation from our DSL to actual EUnit syntax without losing the EUnit machinery. We will also drop the intrusive introspective functions for the usage of the better sys:get_status/1 (thank you Ulf).

Thus, let the journey begin, first with the translation of the Test. This can be done in two ways, either at runtime or at compile time with parse-transforms. I choose the runtime one with translation functions and macros. Ulf Wiger has a neat library for cooler parse transforms, but I shall not use this for now, (the code will probably be revised many times)

The test module has now been changed to

-module(tradepost_tests).
-include_lib("eunit/include/eunit.hrl").
-include("include/eunit_fsm.hrl").

% This is the main point of "entry" for my EUnit testing.
% A generator which forces setup and cleanup for each test in the testset
main_test_() ->
    {foreach,
     fun setup/0,
     fun cleanup/1,
     % Note that this must be a List of TestSet or Instantiator
     [
      % First Iteration
      fun started_properly/1,
      % Second Iteration
      fun identify_seller/1,
      fun insert_item/1,
      fun withdraw_item/1
     ]}.

% Setup and Cleanup
setup()      -> {ok,Pid} = tradepost:start_link(), Pid.
cleanup(Pid) -> tradepost:stop(Pid).

% Pure tests below
% ------------------------------------------------------------------------------
% Let's start simple, I want it to start and check that it is okay.
% I will use the introspective function for this
started_properly(Pid) ->
    ?fsm_test(tradepost,Pid,"Started Properly Test",
      [{state,is,pending},
       {loopdata,is,[undefined,undefined,undefined,undefined,undefined]}
     ]).

% Now, we are adding the Seller API tests
identify_seller(Pid) ->
    ?fsm_test(Pid,"Identify Seller Test",
      [{state,is,pending},
       {call,tradepost,seller_identify,[Pid,seller_password],ok},
       {state,is,pending},
       {loopdata,is,[undefined,undefined,seller_password,undefined,undefined]}
      ]).

insert_item(Pid) ->
    ?fsm_test(Pid,"Insert Item Test",
       [{state,is,pending},
        {call,tradepost,seller_identify,[Pid,seller_password],ok},
        {call,tradepost,seller_insertitem,[Pid,playstation,seller_password],ok},
        {state,is,item_received},
        {loopdata,is,[playstation,undefined,seller_password,undefined,undefined]}
       ]).

withdraw_item(Pid) ->
    ?fsm_test(Pid,"Withdraw Item Test",
       [{state,is,pending},
        {call,tradepost,seller_identify,[Pid,seller_password],ok},
        {call,tradepost,seller_insertitem,[Pid,button,seller_password],ok},
        {state,is,item_received},
        {call,tradepost,seller_withdraw_item,[Pid,seller_password],ok},
        {state,is,pending},
        {loopdata,is,[undefined,undefined,seller_password,undefined,undefined]}
       ]).

And the supporting logic is found in eunit_fsm.hrl

-define(fsm_test(Id,Title,CmdList),
  {Title,fun() -> [ eunit_fsm:translateCmd(Id,Cmd) || Cmd <- CmdList] end}).

together with eunit_fsm.erl

-module(eunit_fsm).
-export([translateCmd/2,get/2]).
-define(Expr(X),??X).
translateCmd(Id,{state,is,X}) ->
    case get(Id,"StateName") of
        X -> true;
        _V ->  .erlang:error({statename_match_failed,
                              [{module, ?MODULE},
                               {line, ?LINE},
                               {expected, X},
                               {value, _V}]})
    end;
translateCmd(_Id,{call,M,F,A,X}) ->
    case apply(M,F,A) of
        X -> ok;
        _V ->  .erlang:error({function_call_match_failed,
                              [{module, ?MODULE},
                               {line, ?LINE},
                               {expression, ?Expr(apply(M,F,A))},
                               {expected, X},
                               {value, _V}]})
    end;
translateCmd(Id,{loopdata,is,X}) ->
    case tl(tuple_to_list(get(Id,"StateData"))) of
        X    -> true;
        _V ->    .erlang:error({loopdata_match_failed,
                                [{module, ?MODULE},
                                 {line, ?LINE},
                                 {expected, X},
                                 {value, _V}]})
    end.

% StateName or StateData
get(Id,Which) ->
    {status,_Pid,_ModTpl, List} = sys:get_status(Id),
    AllData = lists:flatten([ X || {data,X} <- lists:last(List) ]),
    proplists:get_value(Which,AllData).

Now, we have a method of expressing the tests more clearly, next up is the buyer API. Testing if of the essence!

zen:EUnitFSM zenon$ tree .
.
├── ebin
├── include
│   └── eunit_fsm.hrl
├── src
│   └── tradepost.erl
└── test
    ├── eunit_fsm.erl
    └── tradepost_tests.erl

4 directories, 4 files
zen:EUnitFSM zenon$ 

Compile and run

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/ -eval 'eunit:test(tradepost,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly (Started Properly Test)...[0.001 s] ok
    tradepost_tests: identify_seller (Identify Seller Test)...ok
    tradepost_tests: insert_item (Insert Item Test)...ok
    tradepost_tests: withdraw_item (Withdraw Item Test)...ok
    [done in 0.014 s]
  [done in 0.014 s]
=======================================================
  All 4 tests passed.

1>

And so, it works. Next part is adding buyer API and extending the DSL where needed, also maybe rewriting it with Ulf’s awesome parse transform libs.

Erlang EUnit – continuation 3 – Test Control


Knowing Basics of Setup, Cleanup understanding test representation, it’s time to look at EUnit test control. Test control encompassed the ability to specify

  1. If the TestSet should be run in a specific subprocess
  2. If the TestSet should be run in a specific subprocess on a specific node
  3. What timeout a TestSet should have
  4. If the STO’s in the TestSet should be run in a specific predetermined order
  5. If the STO’s in the TestSet should be run in parallell (if possible)
  6. If the STO’s in the TestSet should be run in parallell (if possible), but with the added control that no more than N of them may be run at the same time in parallell

Each of these control specifications are designated by tuples (like most things in EUnit), and may replace any TestSet (single test or deeplist) in the same place where the testset whas previousy placed: But, the replaced testset should then be placed inside the TestSet holder of the tuple.

Replacing a single direct testset as first element in a test generating function (remember the _test_())

testgenerator_test_() -> TESTSET.
testgenerator_test_() -> {CONTROL-TUPLE,TESTSET}.

Replacing a deep list TestSet inside another TestSet.

testgenerator_test_() -> [ TESTSET1, TESTSET2 ].
testgenerator_test_() -> [ TESTSET1, {CONTROL, TESTSET2} ]

without further ado, here comes the examples!

Subprocess Specification for single process and single process on node.

{spawn, TestSet }
{spawn, Node::atom(), TestSet }

Then, may be used as follows

testgeneratorA_test_() -> {spawn, nullarySTO1() }.
testgeneratorB_test_() -> {spawn, [nullarySTO1(),tupleSTO1(),tupleSTO4()]}.
testgeneratorC_test_() -> {spawn, 'eunit@zen', nullarySTO1() }.
testgeneratorD_test_() -> {spawn, 'eunit@zen', [ nullarySTO1() ,
                                          tupleSTO1(),
                                          tupleSTO4()]}.
testgeneratorE_test_() ->
    [ nullarySTO1(),
      nullarySTO2(),
      {spawn, [ {spawn, 'eunit2@zen', tupleSTO1()},
                {spawn, 'eunit2@zen', tupleSTO2()},
                tupleSTO3() ] },
      tupleSTO4() ].

As can be seen, the Control Tuple (CT) can be placed in any place of a TestSet, and puts that replaced testset into itself. Conceptually, it can be thought of as a function, CT(TestSet) which returns the CT with the testset inside.

To prove my point, I shall compile and run it.

zen:EunitBasic4 zenon$ tree .
.
├── ebin
├── src
│   └── mylist.erl
└── test
    └── mylist_tests.erl

3 directories, 2 files

The mylists_tests.erl contains

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).
% Basics
nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end.
nullarySTO2() -> fun nullarySTO1/0.
nullarySTO3() -> fun mylist_tests:nullarySTO1/0.
tupleSTO1() -> { mylist_tests, nullarySTO1 }.
tupleSTO2() -> { mylist_tests, nullarySTO2 }.
tupleSTO3() -> { mylist_tests, nullarySTO3 }.
tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO6() -> ?_test( nullarySTO2() ).
tupleSTO7() -> { 16, nullarySTO3() }.

% Proof of concept
testgeneratorA_test_() -> {spawn, nullarySTO1() }.
testgeneratorB_test_() -> {spawn, [ nullarySTO1() , tupleSTO1(), tupleSTO4()]}.
testgeneratorC_test_() -> {spawn, 'eunit@zen', nullarySTO1() }.
testgeneratorD_test_() -> {spawn, 'eunit@zen', [ nullarySTO1() ,
                                          tupleSTO1(),
                                          tupleSTO4()]}.
testgeneratorE_test_() ->
    [ nullarySTO1(),
      nullarySTO2(),
      {spawn, [ {spawn, 'eunit2@zen', tupleSTO1()},
                {spawn, 'eunit2@zen', tupleSTO2()},
                tupleSTO3() ] },
      tupleSTO4() ].

So, starting 3 shells (one eunit node, and two other nodes), as follows [ note: the other nodes should have the path to the ebins : AND have the code loaded by some means, please see this post to the fun problem issue ]

In orde to automate the code loading, the nl(Module) function is used

nl(Module) -- load module on all nodes

First shell

zen:EunitBasic4 zenon$ erl -pa ebin/ -sname eunit@zen -setcookie eunit
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
(eunit@zen)1>

Second Shell

zen:EunitBasic4 zenon$ erl -pa ebin/ -sname eunit2@zen -setcookie eunit
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
(eunit2@zen)1>

Third shell, starts, connects, loads on all nodes and runs

zen:EUnitBasic4 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitBasic4 zenon$ erl -pa ebin/ -sname base@zen -setcookie eunit
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
(base@zen)1> net_kernel:connect_node(eunit@zen).
true
(base@zen)2> net_kernel:connect_node(eunit2@zen).
true
(base@zen)3> nl(mylist_tests).
abcast
(base@zen)4> eunit:test(mylist,[verbose]).
======================== EUnit ========================
module 'mylist'
  module 'mylist_tests'
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests: nullarySTO1...[0.003 s] ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO3...ok
    mylist_tests:13: tupleSTO4...ok
    [done in 0.070 s]
  [done in 0.070 s]
=======================================================
  All 14 tests passed.
ok
(base@zen)5>


This concludes spawn, as can be seen, all the tests pass, even the node spawned ones. Don’t forget to load the code on the remote nodes first!

Timeout Control

{timeout, Time::number(), Tests}

The effect is that all tests in the TestSet “argument” are given the total time of Time seconds to complete. If the TestSet has not finished during the Time seconds, the TestSet is stopped abruptly without cleanup. Any Setup and Cleanup is also considered during this time.

testgeneratorA_test_() -> {timeout, 1, nullarySTO1() }.
testgeneratorB_test_() -> {timeout, 1, [ nullarySTO1() ,
                                   tupleSTO1(),
                                   tupleSTO4() ] }.
testgeneratorC_test_() ->
    [ nullarySTO1(),
     {timeout, 5, [ {timeout, 1, tupleSTO1()},
                   {timeout, 2, tupleSTO2()},
                   tupleSTO3() ] },
     tupleSTO4() ].

A cool detail is that it is fully legal to do as I did, that is, to nest levels of timeout, which makes total sense if you know some parts may hang indefinitely.

Thus my mylist_tests module now contains

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).

% Basics
nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end.
nullarySTO2() -> fun nullarySTO1/0.
nullarySTO3() -> fun mylist_tests:nullarySTO1/0.
tupleSTO1() -> { mylist_tests, nullarySTO1 }.
tupleSTO2() -> { mylist_tests, nullarySTO2 }.
tupleSTO3() -> { mylist_tests, nullarySTO3 }.
tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO6() -> ?_test( nullarySTO2() ).
tupleSTO7() -> { 16, nullarySTO3() }.

% % Proof of concept
testgeneratorA_test_() -> {timeout, 1, nullarySTO1() }.
testgeneratorB_test_() -> {timeout, 1, [ nullarySTO1() ,
                                   tupleSTO1(),
                                   tupleSTO4() ] }.
testgeneratorC_test_() ->
    [ nullarySTO1(),
     {timeout, 5, [ {timeout, 1, tupleSTO1()},
                   {timeout, 2, tupleSTO2()},
                   tupleSTO3() ] },
     tupleSTO4() ].

And the tests are run as

zen:EUnitBasic4 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitBasic4 zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> eunit:test(mylist,[verbose]).
======================== EUnit ========================
module 'mylist'
  module 'mylist_tests'
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO3...ok
    mylist_tests:13: tupleSTO4...ok
    [done in 0.027 s]
  [done in 0.027 s]
=======================================================
  All 9 tests passed.
ok
2>

This concludes the Timeout demonstration as there is not more to say, sadly :)

Strict Order Of Tests

As is, EUnit is not forced to follow the in-module order or even in-test order of a TestSet, so it’s handy to have a control structure for this.

{inorder, Tests}

As basic as it seems, I shall supply a test for you, with the following module

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).

nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end.
nullarySTO2() -> fun nullarySTO1/0.
nullarySTO3() -> fun mylist_tests:nullarySTO1/0.

tupleSTO1() -> { mylist_tests, nullarySTO1 }.
tupleSTO2() -> { mylist_tests, nullarySTO2 }.
tupleSTO3() -> { mylist_tests, nullarySTO3 }.

tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO6() -> ?_test( nullarySTO2() ).
tupleSTO7() -> { 16, nullarySTO3() }.

generator_test_() ->
    {inorder,
     [
      nullarySTO1(),
      nullarySTO2(),
      nullarySTO3(),
      tupleSTO1(),
      tupleSTO2(),
      tupleSTO3(),
      tupleSTO4(),
      tupleSTO5(),
      tupleSTO6(),
      tupleSTO7()
     ]}.

And compilation, running, etc (you know the drill by now).

zen:EUnitBasic4 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitBasic4 zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> eunit:test(mylist).
  All 10 tests passed.
ok
2> eunit:test(mylist,[verbose]).
======================== EUnit ========================
module 'mylist'
  module 'mylist_tests'
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO3...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests:14: tupleSTO5...ok
    mylist_tests:15: tupleSTO6...ok
    mylist_tests:16: nullarySTO1...ok
    [done in 0.030 s]
  [done in 0.030 s]
=======================================================
  All 10 tests passed.
ok
3>

Now we are dead sure all tests where run in the strict appropriate order.

Parallel Test execution (if possible)

What if we have thousands of tests, which test single part of library routines? Like mylist tests with add, etc? Well, it sure makes sense to run them in parallel! Well, inparallel to the resque!

{inparallel, Tests}

Examples are more talkative and I will supply with such an example

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).

nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end.
nullarySTO2() -> fun nullarySTO1/0.
nullarySTO3() -> fun mylist_tests:nullarySTO1/0.

tupleSTO1() -> { mylist_tests, nullarySTO1 }.
tupleSTO2() -> { mylist_tests, nullarySTO2 }.
tupleSTO3() -> { mylist_tests, nullarySTO3 }.

tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO6() -> ?_test( nullarySTO2() ).
tupleSTO7() -> { 16, nullarySTO3() }.

generator_test_() ->
    {inparallel,
     [
      nullarySTO1(),
      nullarySTO2(),
      nullarySTO3(),
      tupleSTO1(),
      tupleSTO2(),
      tupleSTO3(),
      tupleSTO4(),
      tupleSTO5(),
      tupleSTO6(),
      tupleSTO7()
     ]}.

Lo and behold the run-timeth!

zen:EUnitBasic4 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitBasic4 zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> eunit:test(mylist,[verbose]).
======================== EUnit ========================
module 'mylist'
  module 'mylist_tests'
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO3...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests:14: tupleSTO5...ok
    mylist_tests:15: tupleSTO6...ok
    mylist_tests:16: nullarySTO1...ok
    [done in 0.004 s]
  [done in 0.004 s]
=======================================================
  All 10 tests passed.
ok
2>

Inparallel but with upper bound on parallelism

Maybe you don’t wish more processes to be running in parallel than you have cores, or whatnot, thus, use inparallel with a maximum value of N.

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).

nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end.
nullarySTO2() -> fun nullarySTO1/0.
nullarySTO3() -> fun mylist_tests:nullarySTO1/0.

tupleSTO1() -> { mylist_tests, nullarySTO1 }.
tupleSTO2() -> { mylist_tests, nullarySTO2 }.
tupleSTO3() -> { mylist_tests, nullarySTO3 }.

tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO6() -> ?_test( nullarySTO2() ).
tupleSTO7() -> { 16, nullarySTO3() }.

generator_test_() ->
    {inparallel,2,
     [
      nullarySTO1(),
      nullarySTO2(),
      nullarySTO3(),
      tupleSTO1(),
      tupleSTO2(),
      tupleSTO3(),
      tupleSTO4(),
      tupleSTO5(),
      tupleSTO6(),
      tupleSTO7()
     ]}.

Running and compilation

zen:EUnitBasic4 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitBasic4 zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> eunit:test(mylist,[verbose]).
======================== EUnit ========================
module 'mylist'
  module 'mylist_tests'
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO3...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests:14: tupleSTO5...ok
    mylist_tests:15: tupleSTO6...ok
    mylist_tests:16: nullarySTO1...ok
    [done in 0.015 s]
  [done in 0.015 s]
=======================================================
  All 10 tests passed.
ok
2>

Alas, the runtime is slightly higher, but all went well.

Removing the last question marks

If you where wondering whether all these control tuples play together, then good for you, since here is the answer: YES.

And to prove it, comes the monster module below.

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).

% Basics
nullarySTO1() -> fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end.
nullarySTO2() -> fun nullarySTO1/0.
nullarySTO3() -> fun mylist_tests:nullarySTO1/0.
tupleSTO1() -> { mylist_tests, nullarySTO1 }.
tupleSTO2() -> { mylist_tests, nullarySTO2 }.
tupleSTO3() -> { mylist_tests, nullarySTO3 }.
tupleSTO4() -> ?_test( fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO5() -> ?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).
tupleSTO6() -> ?_test( nullarySTO2() ).
tupleSTO7() -> { 16, nullarySTO3() }.

% % Proof of concept
mixedA_test_() -> {spawn, nullarySTO1() }.
mixedB_test_() -> {spawn, [ {timeout, 1, nullarySTO1()} ,
                            {inparallel, [tupleSTO1(),
                                          tupleSTO4()]}]}.
mixedC_test_() -> {spawn, 'eunit@zen', nullarySTO1() }.
mixedD_test_() -> {spawn, 'eunit@zen', {inorder,
                                        [ nullarySTO1() ,
                                          tupleSTO1(),
                                          tupleSTO4()]}}.
mixedE_test_() ->
    [ nullarySTO1(),
      nullarySTO2(),
      {inparallel, 2, [ {spawn, 'eunit2@zen', tupleSTO1()},
                        {spawn, 'eunit2@zen', tupleSTO2()},
                        tupleSTO3() ] },
      {timeout, 1, tupleSTO4()} ].

As can be seen, the CT can be mixed wildly and are allowed wherever a TestSet is allowed, compiling and running in three shells.

zen:EUnitBasic4 zenon$ erl -pa ebin/ -sname eunit2@zen -setcookie eunit
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] 
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
(eunit2@zen)1>

And the other one

zen:EUnitBasic4 zenon$ erl -pa ebin/ -sname eunit@zen -setcookie eunit
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] 
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
(eunit@zen)1>

And the main shell doing all the fun

zen:EUnitBasic4 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitBasic4 zenon$ erl -pa ebin/ -sname base@zen -setcookie eunit
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] 
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
(base@zen)1> net_kernel:connect_node(eunit2@zen).
true
(base@zen)2> net_kernel:connect_node(eunit@zen).
true
(base@zen)3> nl(mylist_tests).
abcast
(base@zen)4> eunit:test(mylist,[verbose]).
======================== EUnit ========================
module 'mylist'
  module 'mylist_tests'
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests: nullarySTO1...[0.003 s] ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests:13: tupleSTO4...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO3...ok
    mylist_tests:13: tupleSTO4...ok
    [done in 0.063 s]
  [done in 0.063 s]
=======================================================
  All 14 tests passed.
ok
(base@zen)5>

This concludes the Control of EUnit tests, next post will be about testing gen_fsms with EUnit.

EUnit – Spawn on Node – Missing Name Problem


This post is actually a prequel without the first one being posted yet (it’s in draft mode), anyway, if you try doing a spawn with node control, things seem to get a bit out of hand

Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
(t@zen)1>
=ERROR REPORT==== 1-Sep-2010::09:06:20 ===
Error in process <0.56.0> on node 't@zen' with exit value: 
{badarg,[{erlang,atom_to_list,[[]]},
        {eunit_lib,fun_parent,1},
        {eunit_data,parse_function,1},
        {eunit_data,next,1},
        {eunit_data,iter_next,1},
        {eunit_proc,get_next_item,1},
        {eunit_proc,tests_inorder,3},{eunit_proc,with_timeout...

So, what is the cause for this error?

According to the error message, there is a bad usage with the erlang BIF atom_to_list, from the module eunit_lib, in the function fun_parent

{eunit_lib,fun_parent,1}

Now, if you go and read the source (eunit_lib),

%% This library is free software; you can redistribute it and/or modify
%% it under the terms of the GNU Lesser General Public License as
%% published by the Free Software Foundation; either version 2 of the
%% License, or (at your option) any later version.
%%
%% This library is distributed in the hope that it will be useful, but
%% WITHOUT ANY WARRANTY; without even the implied warranty of
%% MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
%% Lesser General Public License for more details.
%%
%% You should have received a copy of the GNU Lesser General Public
%% License along with this library; if not, write to the Free Software
%% Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307
%% USA
%%
%% $Id: eunit_lib.erl 339 2009-04-05 14:10:47Z rcarlsson $
%%
%% @copyright 2004-2007 Mickaël Rémond, Richard Carlsson
%% @author Mickaël Rémond <mickael.remond@process-one.net>
%%   [http://www.process-one.net/]
%% @author Richard Carlsson <richardc@it.uu.se>
%%   [http://user.it.uu.se/~richardc/]
%% @private
%% @see eunit
%% @doc Utility functions for eunit
%% ---------------------------------------------------------------------
%% Get the name of the containing function for a fun. (This is encoded
%% in the name of the generated function that implements the fun.)
fun_parent(F) ->
    {module, M} = erlang:fun_info(F, module),
    {name, N} = erlang:fun_info(F, name),
    case erlang:fun_info(F, type) of
        {type, external} ->
            {arity, A} = erlang:fun_info(F, arity),
            {M, N, A};
        {type, local} ->
            [$-|S] = atom_to_list(N),
            C1 = string:chr(S, $/),
            C2 = string:chr(S, $-),
            {M, list_to_atom(string:sub_string(S, 1, C1 - 1)),
             list_to_integer(string:sub_string(S, C1 + 1, C2 - 1))}
    end.

Where the second line fun_info(F, name) is the culprit. It seems as if the fun does not retain the name for some reason, but fortunately, after discussing this with my senior friend Nicolas, it turns out this is a problem caused by the module from where the fun is sent code not being loaded on the other node where the tests are to be evaluated.

Thus, the simple solution is to make sure the remote node has loaded the code from the tests a priori. This can be achieved with a module_info() or code:load_file/1.

Erlang EUnit – continuation 2 – Test Representation


Having passed through basics and some setup / teardown, it’s time to discuss test representation, a very broad subject which invites for misconceptions.

Basics of Representation

As always, the foundation for the discussion is the EUnit manual pages, where it is stated that a test can be defined in many different ways, and to keep the discussion very short, I shall supply some common examples of what is often seen/used.

Simple Test Objects

Each representation form will be shown with examples based on the mylist module defined in the first post. So, each Simple Test Object (STO) can be defined by

Any nullary function (function with zero arguments)

What this implies is that your STO is encapsulated within a functional object and may be executed  by EUnit. More or less you can put yourself in EUnit’s shoes (some very special shoes) and the user gives you a “black box”, which you simply execute. No need to care about other stuff. Just execute it. Examples of STO’s

nullarySTO1() -> ?assertEqual(6,mylist:sum([1,2,3])).            
fun() -> ?assertEqual(6,mylist:sum([1,2,3])) end.
fun nullarySTO1/0

These are fundamental nullary STO’s

A Module Function Tuple

This is just a different way of reaching the nullary function by the module and function name. Here it is implicit that the functionname is a nullary function. This will not work with {mylist_tests, nullarySTO2 } which will be seen below. Why? Because nullarySTO2() RETURNS a nullary functional object.

tupleSTO1() -> { mylist_tests, nullarySTO1 }.

A Linenumber / STO Tuple (generated by _test(Expr) macro)

According do our best friend, (the user’s guide) “LineNumber is a nonnegative integer …. LineNumber should indicate the source line of the test”. Wow, so it’s the linenumber in the sourcecode for where the tests is. For some reason, this can be written by hand, but luckily the _test(Expr) macro does it for us.

tupleSTO2() ->?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).

Test Sets and Test Generator Basics

One last thing on the basics of representation is that EUnit does not make any “type”-difference between a deep list of STO’s and single STO, that is.

TestSet := [ TestSet ] | STO

So, a Test-Set can be a so called deep list of Test-Sets, to any level, or just a single STO. Also, a Test Generator is a function that ends with _test_() and returns a TestSet. End of story. Like this

my_generator1_test_() -> fun nullarySTO1/0.
my_generator2_test_() -> [ fun nullarySTO1/0, tupleSTO2() ]

What we have seen is all valid erlang EUnit. To prove it, let us place it in a grand EUnit file, compile and run. Below follows the example mylists_tests.erl module, with compilation and running as usual.

EUnit Example File

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").
-compile(export_all).

nullarySTO1() ->?assertEqual(6,mylist:sum([1,2,3])).
nullarySTO2() ->fun() ->?assertEqual(6,mylist:sum([1,2,3])) end. 

tupleSTO1() ->{ mylist_tests, nullarySTO1 }.

tupleSTO2() ->?_test( begin ?assertEqual(6,mylist:sum([1,2,3])) end ).

generator_test_() ->   
    [
     fun nullarySTO1/0,
     nullarySTO2(),
     tupleSTO1(),
     tupleSTO2()
    ].

my_generator_test_() ->
    fun nullarySTO1/0.

That is the end of the module, next is the compilation and running.

zentop:EunitBasic3 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zentop:EunitBasic3 zenon$ erl -pa ebin/
Erlang R14B03 (erts-5.8.4)  [64-bit] [smp:8:8] [rq:8] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.8.4  (abort with ^G)
1> eunit:test(mylist,[verbose]).       
======================== EUnit ========================
module 'mylist'
  module 'mylist_tests'
    mylist_tests: generator_test_...ok
    mylist_tests: nullarySTO2...ok
    mylist_tests: nullarySTO1...ok
    mylist_tests:10: tupleSTO2...ok
    mylist_tests: my_generator_test_...ok
    [done in 0.014 s]
  [done in 0.014 s]
=======================================================
  All 5 tests passed.
ok

Great! Now, as we know how to write, set up and clean STO’s and TestSets. The next lesson is on Test Control!

Erlang EUnit – continuation 1 – Fixtures


This time we will learn how to create a specific environment for our specific tests. As usual, all this can be found in the “EUnit User’s Guide”.

Why would I want to create specific environments?

Consider the case that we wish to perform EUnit tests on on a stateful system (a system with state that is), it could be a server, an fsm, whatever holds and updates a state.

How do I create these environments?

Meet your friend the fixture! There are three kind of major fixtures, Setup / Node / Foreach, this blog entry will only cover the basics of Setup. A fixture is the EUnit name for test-environment. By using fixture statements, we can control how a state (fixture) is created and destroyed. In EUnit terms, this is called

setup/0
cleanup/1

Normally, our EUnit tests run in the same basic environment, as was the case for the mylist EUnit tests, as we shall se next, this is often not desirable when testing a server which contains state (a stateful process).

An example stateful server!

Before (a-hah!) I show some fixture examples, I will show the example code that will be tested using fixtures! It will become clear why the fixture is needed.

-module(numberserver).
-export([start/0,stop/0,op/2,get/0,init/0]).
start() ->
    Pid = spawn_link(?MODULE,init,[]),
    register(?MODULE,Pid),
    ok.
stop() -> ?MODULE ! stop, unregister(?MODULE).
op(Op,Num) -> ?MODULE ! {Op,Num}, ok.
get() ->
    ?MODULE ! {get_result,self()},
    receive
        X -> X
    end.
init() -> loop(basic).
loop(E) ->
    receive
        stop -> ok;
        {get_result,From} ->
            From ! E, loop(E);
        {Op,Num} -> loop(result(Op,E,Num))
    end.
result(_,basic,X) -> X;
result('+',X,Y) -> X + Y;
result('*',X,Y) -> X * Y;
result('-',X,Y) -> X - Y;
result('/',X,Y) -> X / Y.

The code above gives us a small server which performs basic binary operations (operations that take two operands), based on which operator and number is sent with the op/2 function. Compilation , start , usage and stop(ing) is shown below. As usual, compilation and execution is performed from the root directory of my “project”.

zen:EunitBasic2 zenon$ tree .
.
├ ── compile
├── ebin
├── src
│ └── numberserver.erl
└── test

zenon:EunitBasic2$ erlc -o ebin/ src/*.erl
zenon:EunitBasic2$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5 (abort with ^G)
1> numberserver:start().
ok
2> numberserver:op('+',10).
ok
3> numberserver:op('*',2).
ok
4> numberserver:op('-',5).
ok
5> numberserver:op('/',3).
ok
6> numberserver:get().
5.0
7> numberserver:stop().
true
8>
(note the 5.0, this is due to the division which produced a non-integer result).

As you could see, this process obviously maintain some kind of state (hint: the previous computation result), and the subsequent calls use the state as second operand for each binary operation. Thus, when testing the different computations, we wish a clean untouched server for each run. (To exemplify further: assume we are testing basic addition,in one test generating function and later wish to test multiplication, now, in between each test we would ideally want the server to be reset in some way). This is where the fixture comes in.

Basic fixturing (state setup / cleanup)

The most basic fixture example does setup only (=creation only). This would be achieved with the 3-tuple

{ setup,
  SETUP-FUNCTION/0 ,
  TESTSET / INSTANTIATOR/1 }

SETUP-FUNCTION must be a function with arity 0 (zero) and shall perform all necessary operations for creating the encompassing environment.

TEST can be a an EUnit Control operator (more on this in next post), or a test primitive, an INSTANTIATOR/1 is a function receiving the result from the SETUP-FUNCTION and generated instantiated test-sets. I shall use the basic _test(Expr) macro here as is customary (note: there is nothing special about the _test/1 macro, it just returns the the given Expr as a test-function.

first_additon_test_() ->
     { setup,
       fun setup/0,
       fun cleanup/1,
       ?_test(
          begin
              numberserver:op('+',2),
              numberserver:op('*',3),
              ?assertEqual(6,numberserver:get())
          end)}.

This extremely simple test could as well have been written as

-module(numberserver_tests).
-include_lib("eunit/include/eunit.hrl").
first_additon_test_() ->
    numberserver:start(),
    numberserver:op('+',2),
    numberserver:op('*',3),
    ?assertEqual(6,numberserver:get()).

Let’s test this, save the code in numberserver_tests.erl inside the test/ directory, compile and run.

zenon:EunitBasic2$ erlc -o ebin/ src/*.erl test/*.erl
zenon:EunitBasic2$ erl -pa ebin/
zenon:EunitBasic2$ erl

Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5 (abort with ^G)
1> eunit:test(numberserver,[verbose]).
======================== EUnit ========================
module 'numberserver'
module 'numberserver_tests'
numberserver_tests:6: first_additon_test_...ok
[done in 0.004 s]
[done in 0.004 s]
=======================================================
Test passed.
ok

Great, but what happens if we run it again?

2> eunit:test(numberserver,[verbose]).
======================== EUnit ========================
module 'numberserver'
module 'numberserver_tests'
*** context setup failed ***
::error:badarg
in function erlang:register/2
called as register(numberserver,<0.48.0>)
in call from numberserver:start/0

=======================================================
Failed: 0. Skipped: 0. Passed: 0.
One or more tests were cancelled.
error

Uh oh, obviously we can’t register the same numberserver in the same environment again. Seems like we need to do some kind of cleanup (hint hint). The cleanup/1 function is automatically given one argument, the result from the setup.

setup/0 -> X :: any()
cleanup( X :: any() ) -> any()

So, if your setup returns a Pid, you could use that pid in the cleanup to perform stops / kills, whatnot. As is, setup/0 is excuted before any tests are performed, and cleanup/1 is executed after all tests have been performed regardless of test errors or crashes. This would use the 4 tuple setup

{ setup,
  SETUP-FUNCTION/0,
  CLEANUP-FUNCTION/1,
  TEST/INSTANTIATOR }

CLEANUP-FUNCTION/1 is the arity-1 function that reverses the effect of SETUP, and TEST is as usual a test-primitive or an EUnit control term, the INSTANTIATOR/1 receives the same argument as cleanup, namely the result from SETUP.

Setup – Cleanup Fixture Example

-module(numberserver_tests).
-include_lib("eunit/include/eunit.hrl").
first_additon_test_() ->
     { setup,
       fun setup/0,
       fun cleanup/1,
       ?_test(
          begin
              numberserver:op('+',2),
              numberserver:op('*',3),
              ?assertEqual(6,numberserver:get())
          end)}.
first_multiply_test_() ->
     { setup,
       fun setup/0,
       fun cleanup/1,
       ?_test(
          begin
              numberserver:op('*',1),
              numberserver:op('*',3),
              numberserver:op('*',5),
              ?assertEqual(15,numberserver:get())
          end)}.

setup() ->
    numberserver:start().

cleanup(_Pid) ->
    numberserver:stop().

Let’s run it!

zen:EunitBasic2 zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EunitBasic2 zenon$ erl -pa ebin
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5 (abort with ^G)
1> eunit:test(numberserver).
All 2 tests passed.
ok

What is so special about this? Well, for starters, we added one more tests, this second test will now succeed but would have failed previously. Why? Well, because we have a teardown and setup of the numberserver.

This was the basic setup and cleanup, you can find more in the manual pages, next time we will look at test representations.

Erlang EUnit – introduction


Why do I want to have a battery of EUnit tests?

  1. Show that each part of the module is working as expected (Unit testing)
  2. Be able to see if/how new code breaks existing code (Regression testing)

As part of TDD (Test Driven Development) one should ideally write the tests first, and later write the code that makes the tests go through. This will automatically give you a set of Unit + Regression tests, so that each step in your development cycle is “fastened” in a secure place (a “green light”).

How do I write my first EUnit test?

Now, assuming you have not followed TDD and have an untested list-processing module (module is for demonstration purposes only and serves no other reason)

-module(mylist).
-export([sum/1,product/1,odds/1]).

sum([X|R]) -> X + sum(R);
sum([]) -> 0.

product([X|R]) -> X * product(R);
product([]) -> 1.

odds(List) -> odds(List,1).
odds([X|R],N) when N rem 2 == 1 -> [X | odds(R,N+1)];
odds([_|R],N) -> odds(R,N+1);
odds([],_) -> [].

Thus this module could be tested with the separate test module called mylist_tests
(This is good practice since you do not wish to clutter the logic – your real module, with test cases, thus your tests also become portable and regression testing with different versions of code becomes easier).

-module(mylist_tests).
-include_lib("eunit/include/eunit.hrl").

sum_test() ->
    ?assertEqual(0, mylist:sum([])),
    ?assertEqual(0, mylist:sum([0])),
    ?assertEqual(6, mylist:sum([1,2,3,4,-4])).

product_test() ->
    ?assertEqual(1, mylist:product([])),
    ?assertEqual(2, mylist:product([2])),
    ?assertEqual(36, mylist:product([2,3,2,3])).

odds_test() ->
    ?assertEqual([], mylist:odds([])),
    ?assertEqual([1], mylist:odds([1])),
    ?assertEqual([1,3,5], mylist:odds([1,2,3,4,5])).

That was the whole test module, now there are some interesting points with the test module.

  1. The test module name ends with _tests, this is to allow the eunit test function to find the tests for your module by simply referencing the source module (mylist.erl)
  2. The test module does not export any functions explicitly
  3. The test module includes the eunit header file -include_lib(“eunit/include/eunit.hrl”), this is a most important part since this will automatically export all the tests in the module and cause them to be executed once we start.
  4. Test function names end with _test(), this is a requirement for eunit to identify tests

How do I run my eunit tests?

Save the mylists.erl and mylists_tests.erl in the same directory (for this basic guide).

zen:ErlangBlog zenon$ cd EUnitBasic/
zen:EUnitBasic zenon$ erlc *.erl
zen:EUnitBasic zenon$ ls *.beam
mylist.beam mylist_tests.beam

Next, start erl, and run the tests with the test() function, as follows

zen:EUnitBasic zenon$ erl
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.7.5 (abort with ^G)
1> eunit:test(mylist).
All 3 tests passed.
ok

All right! Seems to work. However, we had the test module in the same directory as the source module, which is not good practice.

Testing like a gentleman and a good lady

Make a clear division between test code and source code (well, clearly all of it is source anyway), also we wish our beam files to be stored in a separate directory

zen:EUnitBasic zenon$ rm -iv *.beam
zen:EUnitBasic zenon$ mkdir src
zen:EUnitBasic zenon$ mkdir test
zen:EUnitBasic zenon$ mkdir ebin
zen:EUnitBasic zenon$ mv mylist.erl src/
zen:EUnitBasic zenon$ mv mylist_tests.erl test/

So, what did I do? I removed our previous mess (the beams), created the source directory, the test directory and the ebin directory. Each one of them should contain source modules, test modules and beam files.

Next off, lets compile properly and run the tests, this time standing at the top level

zen:EUnitBasic zenon$ tree .
.
├── ebin
├── src
│ └── mylist.erl
└── test
  └── mylist_tests.erl

3 directories, 2 files

This proves that I am in the top directory of my little “project”.

Next let me compile and output all beams to the ebin/ also I have to tell erlc where to find the source module and the test module.

zen:EUnitBasic zenon$ erlc -o ebin/ src/*.erl test/*.erl

Next, let’s check that all beam files where generated into the ebin/ directory as promised.

zen:EUnitBasic zenon$ tree .
.
├── ebin
│ ├── mylist.beam
│ └── mylist_tests.beam
├── src
│ └── mylist.erl
└── test
  └── mylist_tests.erl

Seems good, let’s run the tests then. From the top level, but telling erl how to find the beams. Also, let’s do it verbally.

zen:EUnitBasic zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]
Eshell V5.7.5 (abort with ^G)
1> eunit:test(mylist,[verbose]).

======================== EUnit ========================
module 'mylist'
module 'mylist_tests'
mylist_tests: sum_test...ok
mylist_tests: product_test...ok
mylist_tests: odds_test...ok
[done in 0.009 s]
[done in 0.009 s]
=======================================================
All 3 tests passed.

ok

Wonderful! This was what we wanted. Now we have the basics set, and next post on this topic will teach us how to do proper test environment setup and cleanup.

All the basis can be found in the EUnit users guide.

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: