Automated Testing – Bringing out the big guns – Part 3


As a continuation on the QuickCheck series, we will now make a serious attempt at refactoring – tests and implementation. First of the much repeated test-code where the generated properties to often fail the preconditions.
Don’t see what I mean? Take a look at the nought and crosses pattern after a ‘make test’

Now, our first adjustment will be to replace the common pattern

{nat(),nat(),nat()}

for our own homegrown generator. The goal is to have a generator which gives us {H,M,S} values within the ranges of reasonable values (0 – 24 system). Defining our own generators is quite easy, we just define a function which returns a 3-tuple. Now, our first generator in this example is a function of arity zero, qctime/0.qctime/0 will return a 3-tuple containing values matching the 24h format. Thus, we implementit with the help of eqc_gen (the QuickCheck generator library) as follows

qctime() -> {eqc_gen:choose(0,23),
            eqc_gen:choose(0,59),
            eqc_gen:choose(0,59)}.

The eqc_gen:choose/2 function will randomly generate a number in the range (inclusive) of the arguments given. Every time you write your own generator, it is a good idea to try it out for yourself in the console with the eqc_gen:sample/1 function. The eqc_gen:sample/1 takes a generator as an argument and gives you a sample of what it generates.

eqc_gen:sample({eqc_gen:choose(0,23),
              eqc_gen:choose(0,59),
              eqc_gen:choose(0,59)}).

We can easily try it out after a ‘make test’ run or by forcing a start as

zen:QCMini zenon$ erl -pa lib/eqc-1.0.1/ebin/
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0]
[hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> eqc_gen:sample({eqc_gen:choose(0,23),
1>               eqc_gen:choose(0,59),
1>               eqc_gen:choose(0,59)}).
{2,45,25}
{0,18,49}
{13,25,36}
{3,51,44}
{16,44,39}
{16,11,32}
{20,19,24}
{21,15,1}
{23,21,38}
{12,36,0}
{15,8,21}
ok

Looking good. All seems in order here. Replacing all the pesky {nat(),nat(),nat()} is now the next step. As we replace them, we can also remove all the preconditions that previously”filtered” their values, so remove all the ?IMPLIES(H < 24, … , ?IMPLIES(S < 60 as well.The change is great as our tests become much more readable. From

prop_time_to_seconds() ->
    ?FORALL({H,M,S},{nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
      clock:hms2s({H,M,S}) == (H*60*60)+(M*60)+S)))).

to

prop_time_to_seconds() ->
    ?FORALL({H,M,S},qctime(),
       clock:hms2s({H,M,S}) == (H*60*60)+(M*60)+S).

However, this will only remove the majority of the duplication. There are still some cases which have some preconditions causing a lot of the properties to be sieved away. The subtrac properties require the subtraction to result in a positive or zero value result, this we can remedy with a value-dependent generator.

qcLT(X) -> eqc_gen:choose(0,X).

Similary we can tuck away the other ?IMPLIES preconditions with smart generators. For the case of prop_subtract_seconds_from_time/0, we write a generator that takes a time/0 3-tuple as argument and returns an appropriate second value for it.

qcLTs({H,M,S}) ->
    eqc_gen:choose(0,H*3600 + M*60 + S).

Now the test can be rewritten from

prop_subtract_seconds_from_time() ->
    ?FORALL({{H,M,S},S2},{qctime(),nat()},
    ?IMPLIES(S2 =< ((H*60*60)+(M*60)+S),
    begin
        T = {H,M,S},
        clock:sub_sec(T,S2) == clock:s2hms((clock:hms2s(T) - S2))
    end)).

to

prop_subtract_seconds_from_time() ->
    ?FORALL(Time,qctime(),
    ?FORALL(Sec,qcLTs(Time),
    begin
        clock:sub_sec(Time,Sec) == clock:s2hms((clock:hms2s(Time) - Sec))
    end)).

As we run the tests frequently while making the changes, we will now observe that the newer and better QuickCheck module helped us find a bug!

zen:QCMini zenon$ make test
prop_subtract_seconds_from_time: Failed! After 1 tests.
{3,2,58}
10172
Shrinking...........(11 times)
{2,0,0}
3601

The output shows that the generated values {2,0,0} together with 3601 will make the test fail. So, if we subtract 3601 seconds from 2 Hours, we would suspect to see the wrong result

1> clock:sub_sec({2,0,0},3601).
{1,-1,59}

And indeed this does look fishy. Taking a second look at the implementation it does look gritty, and a better implementation will be written in a hand-turn (swedish expression)

-spec(sub_sec(time(), nat()) -> time()).
sub_sec(T,S) ->
    sub2(h,sub2(m,sub2(s,{T,S}))).

it relies on the newly added function

sub2(s,{{H,M,S},S2}) ->
    Ssub = S2 rem 60,
    {case {Ssub > S andalso M > 0, Ssub > S andalso M == 0} of
         {true,_} -> {H,M-1,S+60-Ssub};
         {_,true} -> {H-1,M+59,S+60-Ssub};
         _ -> {H,M,S-Ssub}
     end,(S2 - Ssub) div 60};
sub2(m,{{H,M,S},M2}) ->
    Msub = M2 rem 60,
    {case Msub > M andalso H > 0 of
         true -> {H-1,M+60-Msub,S};
         false ->{H,M-Msub,S}
     end,(M2 - Msub) div 60};
sub2(h,{{H,M,S},H2}) -> {H-H2,M,S}.

Running the tests will now pass. sub2/2 is written so as to be composable, making each subtraction for seconds and minutes easier.
Now for another fix in the quickcheck module. There are still too many failed predicates for the prop_sub_time test. Running it will most likely give an output of mostly ‘x’ and some ‘.’, this is due to the failed precondition

    ?IMPLIES(((H*3600) + M*60 + S) >= ((H2*3600) + M2*60 + S2),

Similarly as for the previous test that needs the generated value to be less than the first argument of a subtraction, we can have a dependant generator for the 3-tuple

qcLTt({H,M,S}) ->
    {eqc_gen:choose(0,H),
     eqc_gen:choose(0,M),
     eqc_gen:choose(0,S)}.

This generator will guarantee to generate a time() tuple which is less than or equal to the given one. Problem solved, we can now rewrite the test

prop_sub_time() ->
    eqc:numtests(100,
    ?FORALL({{H,M,S},{H2,M2,S2}},{qctime(),qctime()},
    ?IMPLIES(((H*3600) + M*60 + S) >= ((H2*3600) + M2*60 + S2),
    begin
        T = {H,M,S},
        T2= {H2,M2,S2},
        clock:sub(T,T2) == clock:s2hms(clock:hms2s(T) - clock:hms2s(T2))
    end))).

to

prop_sub_time() ->
    ?FORALL(T,qctime(),
    ?FORALL(T2,qcLTt(T),
    begin
        clock:sub(T,T2) == clock:s2hms(clock:hms2s(T) - clock:hms2s(T2))
    end)).

Running the tests should now show the following

zen:QCMini zenon$ make test
erlc -o ebin/ -pa lib/eqc-1.0.1/ebin/ src/*.erl test/*.erl
erl -pa ebin/ -pa lib/eqc-1.0.1/ebin/ -eval 'eqc:module(clock_eqc).'
Erlang R14B (erts-5.8.1) [source] [smp:4:4] [rq:4] [async-threads:0]
[hipe] [kernel-poll:false]

Eshell V5.8.1  (abort with ^G)
1> prop_time_to_seconds: Starting eqc mini version 1.0.1
(compiled at {{2010,6,13},{11,15,30}})
..................................................................
..................................
OK, passed 100 tests
prop_time_to_minutes: ..............................................

......................................................
OK, passed 100 tests
prop_time_to_hours: ...............................................
.....................................................
OK, passed 100 tests
prop_seconds_to_time: ..............................................
......................................................
OK, passed 100 tests
prop_minutes_to_time: ..............................................
......................................................
OK, passed 100 tests
prop_hours_to_time: ................................................
....................................................
OK, passed 100 tests
prop_subtract_seconds_from_time: .....................................
...............................................................
OK, passed 100 tests
prop_subtract_hours_from_time: .......................................
.............................................................
OK, passed 100 tests
prop_subtract_minutes_from_time: .....................................
...............................................................
OK, passed 100 tests
prop_add_time: ....................................................
................................................
OK, passed 100 tests
prop_sub_time: ....................................................
................................................
OK, passed 100 tests

1>

Perfect. The tests are now readable and make more sense. We even found a bug in the original implementation.

We have now seen how to write simple generators with arity zero, dependant generators that take generated values as arguments and how this helps us reduce the size of the test code as well as make it more readable.
Now follow the final quickcheck implementation with the final code

clock_eqc.erl

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created : 21 Nov 2010 by Gianfranco <zenon@zen.home>
-module(clock_eqc).
-include_lib("eqc/include/eqc.hrl").
-compile(export_all).

% ---------------------------------------------------------------
qctime() -> {eqc_gen:choose(0,23),
             eqc_gen:choose(0,59),
             eqc_gen:choose(0,59)}.

qcLT(X) -> eqc_gen:choose(0,X).

qcLTs({H,M,S}) -> eqc_gen:choose(0,(H*3600)+ (M*60) + S).

qcLTm({H,M,S}) -> eqc_gen:choose(0,H*60+M).    

qcLTt({H,M,S}) ->
    {eqc_gen:choose(0,H),
     eqc_gen:choose(0,M),
     eqc_gen:choose(0,S)}.

prop_time_to_seconds() ->
    ?FORALL({H,M,S},qctime(),
            clock:hms2s({H,M,S}) == (H*60*60)+(M*60)+S).

prop_time_to_minutes() ->
    ?FORALL({H,M,S},qctime(),
            clock:hms2m({H,M,S}) == (H*60)+M+S div 60).

prop_time_to_hours() ->
    ?FORALL({H,M,S},qctime(),
            clock:hms2h({H,M,S}) == H+(M div 60)+(S div (60*60))).

prop_seconds_to_time() ->
    ?FORALL(S,nat(),
    begin
        S2= S rem 60,
        M = (S div 60) rem 60,
        H = ((S div 60) div 60) rem 24,
        clock:s2hms(S) == {H,M,S2}
    end).

prop_minutes_to_time() ->
    ?FORALL(M,nat(),
    begin
        H = (M div 60) rem 24,
        M2= M rem 60,
        clock:m2hms(M) == {H,M2,0}
    end).

prop_hours_to_time() ->
    ?FORALL(H, nat(),
        clock:h2hms(H) == {H rem 24,0,0}).

prop_subtract_seconds_from_time() ->
    ?FORALL(Time,qctime(),
    ?FORALL(Sec,qcLTs(Time),
    begin
        clock:sub_sec(Time,Sec) == clock:s2hms((clock:hms2s(Time) - Sec))
    end)).

prop_subtract_hours_from_time() ->
    eqc:numtests(100,
    ?FORALL({H,M,S},qctime(),
    ?FORALL(H2,qcLT(H),
    begin
        clock:sub_hour({H,M,S},H2) == {H-H2,M,S}
    end))).

prop_subtract_minutes_from_time() ->
    ?FORALL(T,qctime(),
    ?FORALL(M,qcLTm(T),
    begin
        clock:sub_minute(T,M) == clock:s2hms((clock:hms2s(T) - (M*60)))
    end)).

prop_add_time() ->
    ?FORALL({{H,M,S},{H2,M2,S2}},{qctime(),qctime()},
     begin
        T = {H,M,S},
        T2= {H2,M2,S2},
        {HRes,MRes,SRes} = clock:add(T,T2),
        HRes < 24 andalso MRes < 60 andalso SRes < 60 andalso
        {HRes,MRes,SRes} == clock:s2hms(clock:hms2s(T) + clock:hms2s(T2))
    end).

prop_sub_time() ->
    ?FORALL(T,qctime(),
    ?FORALL(T2,qcLTt(T),
    begin
        clock:sub(T,T2) == clock:s2hms(clock:hms2s(T) - clock:hms2s(T2))
    end)).

After the implementation in clock.erl some changes where made to that one as well, and the final module is presented below

clock.erl

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created : 21 Nov 2010 by Gianfranco <zenon@zen.home>
-module(clock).
-export([
         hms2s/1,hms2m/1,hms2h/1,
         s2hms/1,m2hms/1,h2hms/1,
         sub_sec/2,sub_hour/2,sub_minute/2,
         add/2,sub/2
        ]).

-type nat() :: non_neg_integer().
-type time() :: {nat(),nat(),nat()}.

-spec(hms2s(time()) -> nat()).
hms2s({H,M,S}) -> H*3600 + M*60 + S.

-spec(hms2m(time()) -> nat()).
hms2m({H,M,S}) -> H*60+M+(S div 60).

-spec(hms2h(time()) -> nat()).
hms2h({H,M,S}) -> H+(M div 60)+(S div 3600).

-spec(s2hms(nat()) -> time()).
s2hms(S) -> {((S div 60) div 60) rem 24, (S div 60) rem 60, S rem 60}.

-spec(m2hms(nat()) -> time()).
m2hms(M) -> {(M div 60) rem 24, M rem 60, 0}.

-spec(h2hms(nat()) -> time()).
h2hms(H) -> {H rem 24,0,0}.

-spec(sub_sec(time(), nat()) -> time()).
sub_sec(T,S) ->
    sub2(h,sub2(m,sub2(s,{T,S}))).

-spec(sub_minute(time(), nat()) -> time()).
sub_minute(T,M) ->
    sub2(h,sub2(m,{T,M})).

-spec(sub_hour(time(),nat()) -> time()).
sub_hour(T,H) ->
    sub2(h,{T,H}).

-spec(add(time(),time()) -> time()).
add({H,M,S},{H2,M2,S2}) ->
    Sp = S + S2,
    S3 = Sp rem 60,
    S4 = Sp div 60,
    Mp = M + M2 + S4,
    M3 = Mp rem 60,
    H3 = (H + H2 + (Mp div 60)) rem 24,
    {H3,M3,S3}.

-spec(sub(time(),time()) -> time()).
sub(T,{H,M,S}) ->
    {T1,M2} = sub2(s,{T,S}),
    {T2,H2} = sub2(m,{T1,M+M2}),
    sub2(h,{T2,H+H2}).

%% -----------------------------------------------------------------
sub2(s,{{H,M,S},S2}) ->
    Ssub = S2 rem 60,
    {case {Ssub > S andalso M > 0, Ssub > S andalso M == 0} of
         {true,_} -> {H,M-1,S+60-Ssub};
         {_,true} -> {H-1,M+59,S+60-Ssub};
         _ -> {H,M,S-Ssub}
     end,(S2 - Ssub) div 60};
sub2(m,{{H,M,S},M2}) ->
    Msub = M2 rem 60,
    {case Msub > M andalso H > 0 of
         true -> {H-1,M+60-Msub,S};
         false ->{H,M-Msub,S}
     end,(M2 - Msub) div 60};
sub2(h,{{H,M,S},H2}) -> {H-H2,M,S}.

Automated Testing – Bringing out the big guns – Part 2


This is the continuation on how to to TDD with QuickCheck. Last time we had the following methods implemented and tested for the clock library.

-export([hms2s/1,hms2m/1,hms2h/1,
        s2hms/1,m2hms/1,h2hms/1,
        sub_sec/2]).

Missing are the implementations for the sub_min, and sub_hour. It’s always wise to pick the low hanging fruit first, enter sub_hour/2 test.

prop_subtract_hours_from_time() ->
    ?FORALL({H,M,S,H2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H < 24,
    ?IMPLIES(H2 =< H,
    begin
        clock:sub_hour({H,M,S},H2) == {H-H2,M,S}
    end))))).

Needles to say, the testcode is trivial. This test will fail the preconditions many times and it is therefore wise to crank up the amount of times the test is run by using eqc:numtests/2 where the first argument is an integer, the amount of tests, and the second argument is a property. The modified code becomes

prop_subtract_hours_from_time() ->
    eqc:numtests(500,
    ?FORALL({H,M,S,H2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24 andalso H2 < H,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    begin
        clock:sub_hour({H,M,S},H2) == {H-H2,M,S}
    end))))).

I invite you to run the modified version. Now, for the slightly more complicated sub_min/2 function. While doing the test for sub_min/2, I found a bug in the old implementation, this was the error produced while running the test

zen:QCMini zenon$ make test
prop_subtract_minutes_from_time:
Failed! Reason:
{'EXIT',{badarith,[{clock,m2hms,1},
                {clock_eqc,'-prop_subtract_minutes_from_time/0-fun-0-',4},
                {eqc,implies,3},
                {eqc,'-f885_0/2-fun-4-',2},
                {eqc_gen,'-f330_0/2-fun-0-',5},
                {eqc_gen,f195_0,2},
                {eqc_gen,gen,3},
                {eqc,'-f867_0/1-fun-2-',3}]}}
After 1 tests.
{0,0,0,0}

The reason is that hms2m/1 produced a float value! And float values don’t work with div. So, what is needed is to either round the return value from hm2m/1 or to make it return a rounded value. As the clock library is intended to work with natural numbers, it’s best to make the functions return rounded whole values. Now, changing the hms2s/1, hms2m/1 and hms2h/1 functions will require changing the tests. Which is kind of the point of having the tests in the first place. Changing the implementation should alert the programmer that the behaviour has changed. Now the test looks like

prop_subtract_minutes_from_time() ->
    ?FORALL({H,M,S,M2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(M2 =< ((H*60)+M),
    begin
        T = {H,M,S},
        clock:sub_minute(T,M2) == clock:m2hms((clock:hms2m(T) - M2))
    end))))).

The test continues to fail

zen:QCMini zenon$ make test
prop_subtract_minutes_from_time: ...Failed! After 4 tests.
{1,1,1,0}
Shrinking..(2 times)
{0,0,1,0}

This time due to the fact that the model in the QuickCheck test will round the Right Hand Side (RHS) and calculate the LHS without rounding! Clearly, we can’t use the m2hms/1 for this test.  The solution is to go by the smallest possible unit, the second.

   clock:sub_minute(T,M2) == clock:s2hms((clock:hms2s(T) - (M2*60)))

And voila!

zen:QCMini zenon$ make test
prop_subtract_minutes_from_time: .......x..xx.x...............x...x......
.....................................x.......xx.............x.......
x.xx.
OK, passed 100 tests

Of course I made proper changes to the other tests in order to keep all functions to returning whole integer values (not using round!) [left as an exercise].

Now it is left to write a test for the add/2 function, the add function will allow us to add two time() values together, wrapping on 24, so the added value is in the bounds of time() as well.

Due to the intended cyclic behaviour on 24, we will have to make adjustments to the other functions first. Making sure everything is working within the bounds of the 24 hour system (00:00:00 – 23:59:59). This minor modification just means adding some (rem 24) to the _2hms/1 functions and changing some tests. Do this first.

Now the test

prop_add_time() ->
    ?FORALL({H,M,S,H2,M2,S2},{nat(),nat(),nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H2 < 24,
    ?IMPLIES(M2 < 60,
    ?IMPLIES(S2 < 60,
    begin
        T = {H,M,S},
        T2= {H2,M2,S2},
        {HRes,MRes,SRes} = clock:add(T,T2),
        HRes < 24 andalso MRes < 60 andalso SRes < 60 andalso
        {HRes,MRes,SRes} == clock:s2hms(clock:hms2s(T) + clock:hms2s(T2))
    end))))))).

And the proof that the implementation

add({H,M,S},{H2,M2,S2}) ->
    Sp = S + S2,
    S3 = Sp rem 60,
    S4 = Sp div 60,
    Mp = M + M2 + S4,
    M3 = Mp rem 60,
    H3 = (H + H2 + (Mp div 60)) rem 24,
    {H3,M3,S3}.

is working

zen:QCMini zenon$ make test
prop_add_time: ......................................................
........................x.............x.x.xx..xxx.xxxxxxx.xx.x.x.
OK, passed 100 tests

Great! Last API function left,  sub/2, it will be somewhat similar to the sub_sec/2, but the test first of all

prop_sub_time() ->
    ?FORALL({H,M,S,H2,M2,S2},{nat(),nat(),nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H2 < 24,
    ?IMPLIES(M2 < 60,
    ?IMPLIES(S2 < 60,
    ?IMPLIES(((H*3600) + M*60 + S) >= ((H2*3600) + M2*60 + S2),
    begin
        T = {H,M,S},
        T2= {H2,M2,S2},
        {HRes,MRes,SRes} = clock:sub(T,T2),
        HRes >= 0 andalso MRes >= 0 andalso SRes >= 0 andalso
        {HRes,MRes,SRes} == clock:s2hms(clock:hms2s(T) - clock:hms2s(T2))
    end))))))))

Do not fret that this is ugly, the next post on this topic will show you how to fix this up in a good way.

sub({H,M,S},{H2,M2,S2}) ->
    Sres = S - (S2 rem 60),
    Mres = M - (S2 div 60) - M2,
    Hres = H - (M div 60) - H2,
    case {Sres < 0, Mres > 0, Mres == 0} of
        {false,false,false} ->
            {Hres - 1, Mres+60, Sres};
        {true,true,_}  ->
            {Hres, Mres - 1, Sres + 60};
        {true,false,_} ->
            {Hres - 1, Mres + 59, Sres + 60};
        _ ->
            {Hres,Mres,Sres}
    end.

And voila, running the tests for 100 (or like I did 1000) runs makes it pass

zen:QCMini zenon$ make test
prop_sub_time: ....x...x.x..xx.xx...xxxx.x..x.x..x..x..x.xx.x.xx..x..xx.x.
.x......x.xx.xx.xx...x.x..xxx.xx...x.xx.x..x.xx..xxxxxx.xxxx...xx....xx..
xx.xxxxxx.x...xxx.xxx.xx.xx..xxx..xxxx.x.xxxxxxxxxxxxxxxx.x.xxxxx.
(removed lines)
OK, passed 10000 tests

End of this story! We now have the clock lib for working with time() values. Now follows the full code. And next time we will start refactoring and tidying up the clock lib, also the testfile, and write our own generators to make the clock_eqc.erl look better.

The implementation module (clock.erl)

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created : 21 Nov 2010 by Gianfranco <zenon@zen.home>
-module(clock).
-export([
         hms2s/1,hms2m/1,hms2h/1,
         s2hms/1,m2hms/1,h2hms/1,
         sub_sec/2,sub_hour/2,sub_minute/2,
         add/2,sub/2
        ]).

-type nat() :: non_neg_integer().
-type time() :: {nat(),nat(),nat()}.

-spec(hms2s(time()) -> nat()).
hms2s({H,M,S}) -> H*3600 + M*60 + S.

-spec(hms2m(time()) -> nat()).
hms2m({H,M,S}) -> H*60+M+(S div 60).

-spec(hms2h(time()) -> nat()).
hms2h({H,M,S}) -> H+(M div 60)+(S div 3600).

-spec(s2hms(nat()) -> time()).
s2hms(S) -> {((S div 60) div 60) rem 24, (S div 60) rem 60, S rem 60}.

-spec(m2hms(nat()) -> time()).
m2hms(M) -> {(M div 60) rem 24, M rem 60, 0}.

-spec(h2hms(nat()) -> time()).
h2hms(H) -> {H rem 24,0,0}.

-spec(sub_sec(time(), nat()) -> time()).
sub_sec({H,M,S},S2) ->
    Sres = S - (S2 rem 60),
    Mres = M - (S2 div 60),
    Hres = H - (M div 60),
    case {Sres < 0, Mres > 0, Mres == 0} of
        {false,false,false} -> {Hres - 1, Mres+60, Sres};
        {true,true,_}  -> {Hres, Mres - 1, Sres + 60};
        {true,false,_} -> {Hres - 1, Mres + 59, Sres + 60};
        _ -> {Hres,Mres,Sres}
    end.

-spec(sub_hour(time(),nat()) -> time()).
sub_hour({H,M,S},H2) ->
    {H-H2,M,S}.

-spec(sub_minute(time(), nat()) -> time()).
sub_minute({H,M,S},M2) ->
    case M2 > M of
        true -> {H-1,(M+60)-M2,S};
        false -> {H,M-M2,S}
    end.

-spec(add(time(),time()) -> time()).
add({H,M,S},{H2,M2,S2}) ->
    Sp = S + S2,
    S3 = Sp rem 60,
    S4 = Sp div 60,
    Mp = M + M2 + S4,
    M3 = Mp rem 60,
    H3 = (H + H2 + (Mp div 60)) rem 24,
    {H3,M3,S3}.

-spec(sub(time(),time()) -> time()).
sub({H,M,S},{H2,M2,S2}) ->
    Sres = S - (S2 rem 60),
    Mres = M - (S2 div 60) - M2,
    Hres = H - (M div 60) - H2,
    case {Sres < 0, Mres > 0, Mres == 0} of
        {false,false,false} -> {Hres - 1, Mres+60, Sres};
        {true,true,_}  -> {Hres, Mres - 1, Sres + 60};
        {true,false,_} -> {Hres - 1, Mres + 59, Sres + 60};
        _ -> {Hres,Mres,Sres}
    end.

The test module (clock_eqc.erl)

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created : 21 Nov 2010 by Gianfranco <zenon@zen.home>
-module(clock_eqc).
-include_lib("eqc/include/eqc.hrl").
-compile(export_all).

prop_time_to_seconds() ->
    ?FORALL({H,M,S},{nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
      clock:hms2s({H,M,S}) == (H*60*60)+(M*60)+S)))).

prop_time_to_minutes() ->
    ?FORALL({H,M,S},{nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
      clock:hms2m({H,M,S}) == (H*60)+M+S div 60)))).

prop_time_to_hours() ->
    ?FORALL({H,M,S},{nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    clock:hms2h({H,M,S}) == H+(M div 60)+(S div (60*60)))))).

prop_seconds_to_time() ->
    ?FORALL(S,nat(),
    begin
        S2= S rem 60,
        M = (S div 60) rem 60,
        H = ((S div 60) div 60) rem 24,
        clock:s2hms(S) == {H,M,S2}
    end).

prop_minutes_to_time() ->
    ?FORALL(M,nat(),
    begin
        H = (M div 60) rem 24,
        M2= M rem 60,
        clock:m2hms(M) == {H,M2,0}
    end).

prop_hours_to_time() ->
    ?FORALL(H, nat(),
        clock:h2hms(H) == {H rem 24,0,0}).

prop_subtract_seconds_from_time() ->
    ?FORALL({H,M,S,S2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H < 24,
    ?IMPLIES(S2 =< ((H*60*60)+(M*60)+S),
    begin
        T = {H,M,S},
        clock:sub_sec(T,S2) == clock:s2hms((clock:hms2s(T) - S2))
    end))))).

prop_subtract_hours_from_time() ->
    eqc:numtests(100,
    ?FORALL({H,M,S,H2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24 andalso H2 < H,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    begin
        clock:sub_hour({H,M,S},H2) == {H-H2,M,S}
    end))))).

prop_subtract_minutes_from_time() ->
    ?FORALL({H,M,S,M2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(M2 =< ((H*60)+M),
    begin
        T = {H,M,S},
        clock:sub_minute(T,M2) == clock:s2hms((clock:hms2s(T) - (M2*60)))
    end))))).

prop_add_time() ->
    ?FORALL({H,M,S,H2,M2,S2},{nat(),nat(),nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H2 < 24,
    ?IMPLIES(M2 < 60,
    ?IMPLIES(S2 < 60,
    begin
        T = {H,M,S},
        T2= {H2,M2,S2},
        {HRes,MRes,SRes} = clock:add(T,T2),
        HRes < 24 andalso MRes < 60 andalso SRes < 60 andalso
        {HRes,MRes,SRes} == clock:s2hms(clock:hms2s(T) + clock:hms2s(T2))
    end))))))).

prop_sub_time() ->
    eqc:numtests(1000,
    ?FORALL({H,M,S,H2,M2,S2},{nat(),nat(),nat(),nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H2 < 24,
    ?IMPLIES(M2 < 60,
    ?IMPLIES(S2 < 60,
    ?IMPLIES(((H*3600) + M*60 + S) >= ((H2*3600) + M2*60 + S2),
    begin
        T = {H,M,S},
        T2= {H2,M2,S2},
        {HRes,MRes,SRes} = clock:sub(T,T2),
        HRes >= 0 andalso MRes >= 0 andalso SRes >= 0 andalso
        {HRes,MRes,SRes} == clock:s2hms(clock:hms2s(T) - clock:hms2s(T2))
    end))))))))).

Next time

Cleaning up and refactoring of the modules. Generators for our cases and extra material on quickcheck.
Cheers
/G

Automated Testing – Bringing out the big guns – Part 1


Since the last gen_fsm test post already tip-toed on the subject of  Automated Testing, I thought it would be natural to go into QuickCheck and  “bring out the big”  guns.

Enter QuickCheck.

What is quickcheck?

QuickCheck for Erlang is a commercial tool from QuviQ for Property Based Testing by automatic generation of testdata that is fed into property specifications. Behind this we find John Hughes and Thomas Arts. Written in Erlang.

For the Haskell fans out there, I hope you recognize QuickCheck from the Haskell history, through this incredibly well-designed page (irony). As a side-note, I did firewall testing with the Haskell QuickCheck some time ago for a company called Witsbits.

This post will use the free QuickCheck mini version.

How do I install it?

This is seriously easy, and takes under 10 seconds (no honestly). Unzip the eqcmini.zip archive and you have your “installation”. As the product is shipped with the beamfiles, you don’t need to (you actually can’t) compile anything.

How do I use it?

You need three things in order to get QuickChecking (qc’ing : queue-see-ing)

  1. A module including the quickcheck application header file, 

    -include_lib(“eqc/include/eqc.hrl”).

  2. Inside the same module, properties you will run your test on, each property must be exported.
  3. A module with the actual implementation of what we wish to test.

This is very similar as to EUnit testing. We have a test-module consisting of tests and the EUnit header file, and an implementation file with the actual code.

The idea is that the test module exports the properties that will be tested by quickcheck.

How can I do TDD with QuickCheck?

For the purpose of this example, we shall develop parts of a small clock library that makes it possible to add / subtract time with its basis in the time() format.

As usual,  we will probably a standard layout with the minimals, ebin/ src/ test/  and a makefile.

zen:QCMini zenon$ tree -L 2.
.
├── Makefile
├── ebin
├── lib
│   └── eqc-1.0.1
├── src
│   └── clock.erl
└── test
    └── clock_eqc.erl

5 directories, 3 files

As can be seen, the quickcheck directory is put into the lib/ directory, and I keep the now empty clock module in src/ with the test module in test/

The clock_eqc.erl contains the following initial code

-module(clock_eqc).
-include_lib("eqc/include/eqc.hrl").
-compile(export_all).

prop_time_to_seconds() ->
    ?FORALL({H,M,S},{nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
             clock:hms2s({H,M,S}) == (H*60*60)+(M*60)+S)))).

First we see that the the property has arity 0 (zero). The name also begins with prop_. The ?FORALL is a quickcheck macro

?FORALL(X,Gen,Prop)

with the first element being the pattern; in this case the 3-tuple {H, M, S} which will be pattern matched for value binding to generated values from the Generator. The second element is the Generator from where values (3-tuple of natural numbers) are generated and bound to the Pattern and the third being the Property. For this example, the property is everything from the next line (?IMPLIES…) to the end.

The ?IMPLIES macro has two elements. A precondition (for this example: the (H < 24)) works as a filter, only allowing H values with values in the range 1 – 23, it discard all tests for which the precondition does not hold and runs the tests for which the precondition is valid. Second element is the Property (for this exmple: everything from ?IMPLIES(M < 60)… and onward).

For this code piece, we see that there is one Pattern and generator, three natural numbers, then 3 preconditions, ensuring that only sane values are given for (H)ours, (M)inutes and (S)econds. When the sifted values are accepted, the last part can be thought of as the property: The function call hms2s/1 must return the amount of seconds for the 3-tuple. Here we are doing a crucial thing, which is basic to quickcheck. We are comparing the function value with our model.

The clock.erl module is totally blank except for the module declaration, and the Make file holds

QCLIB := lib/eqc-1.0.1/ebin/
all:
        erlc -o ebin/ -pa $(QCLIB) src/*.erl test/*.erl
.PHONY: test
test:   all
        erl -pa ebin/ -pa $(QCLIB) -eval 'eqc:module(clock_eqc).'

Let’s run the first make test (with trunkated output)

zen:QCMini zenon$ make test
1> prop_time_to_seconds: Starting eqc mini version 1.0.1
(compiled at {{2010,6,13},{11,15,30}})
Failed! Reason:
{'EXIT',{undef,[{clock,hms2s,[{0,0,0}]},
                {clock_eqc,'-prop_time_to_seconds/0-fun-0-',3},
                {eqc,implies,3},
                {eqc,'-f885_0/2-fun-4-',2},
                {eqc_gen,'-f330_0/2-fun-0-',5},
                {eqc_gen,f195_0,2},
                {eqc_gen,gen,3},
                {eqc,'-f867_0/1-fun-2-',3}]}}
After 1 tests.
{0,0,0}

1>

Not too unexpected, there is no function hms2s exported. Let’s write and export it.

-module(clock).
-export([hms2s/1]).

hms2s({H,M,S}) -> H*60*60 + M*60 + S.

Now, let’s run test

zen:QCMini zenon$ make test
1> prop_time_to_seconds: Starting eqc mini version 1.0.1
(compiled at {{2010,6,13},{11,15,30}})
...................................................................
...........x......xx.......x.....x.x...
OK, passed 100 tests

1>

Passed 100, seems good. The ‘x’ we see are cases that where discarded due to failed preconditions (H < 24, M < 60, S < 60).

Next test, we want to do the same for minutes and hours (only showing the isolated tests in themselves, not the whole module each time).

prop_time_to_minutes() ->
    ?FORALL({H,M,S},{nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
      clock:hms2m({H,M,S}) == (H*60)+M+S/60)))).

And the trivial implementation (leaving out rest of module)

hms2m({H,M,S}) -> H*60+M+S/60.

with running

zen:QCMini zenon$ make test
Eshell V5.8.1  (abort with ^G)
1> prop_time_to_seconds: Starting eqc mini version 1.0.1
(compiled at {{2010,6,13},{11,15,30}})
...................................................................
............x..........xxx..x........x.
OK, passed 100 tests
prop_time_to_minutes: ...............................................
.........................................x............
OK, passed 100 tests

1>

As the last in the trio goes, hms2h left, after that one, we will implement the converse functions, returning time() type tuples from (H)ours, (M)inutes and (S)econds.

In good TDD style, first the test.

prop_time_to_hours() ->
    ?FORALL({H,M,S},{nat(),nat(),nat()},
    ?IMPLIES(H < 24,
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    clock:hms2h({H,M,S}) == H+M/60+(S/(60*60)))))).

and then the implementation

hms2h({H,M,S}) -> H+(M/60)+(S/(60*60)).

Now for the reverse functions, of course, the test first, in true TDD

prop_seconds_to_time() ->
    ?FORALL(S,nat(),
    begin
        S2 = S rem 60,
        M = (S div 60) rem 60,
        H = (S div 60) div 60,
        clock:s2hms(S) == {H,M,S2}
    end).

A very interesting thing we see from the above code is that when doing property driven development is that in developing the model for the property, one may find the solution to the implementation. In contrast to EUnit tests, this is one step closer to the implementation.

s2hms(S) -> {(S div 60) div 60, (S div 60) rem 60, S rem 60}.

And so the test run

zen:QCMini zenon$ make test
Eshell V5.8.1  (abort with ^G)
1> prop_time_to_seconds: Starting eqc mini version 1.0.1
(compiled at {{2010,6,13},{11,15,30}})
.............................................................
......................x......x.....x..x..x.xx.
OK, passed 100 tests
prop_time_to_minutes: ..........................................
..............................x......xx....x...xx.xx..........x....
OK, passed 100 tests
prop_time_to_hours: ...........................................
....................................x.....x.x...x.........x...
OK, passed 100 tests
prop_seconds_to_time: ...........................................
.........................................................
OK, passed 100 tests

Now, I will do a quick fast forward for the rest of the reverse functions, m2hms/1 and h2hms/1. I will present the tests and the solutions.

prop_minutes_to_time() ->
    ?FORALL(M,nat(),
    begin
        H = M div 60,
        M2= M rem 60,
        clock:m2hms(M) == {H,M2,0}
    end).

prop_hours_to_time() ->
    ?FORALL(H, nat(),
        clock:h2hms(H) == {H,0,0}).

And the implementation that can trivially be taken from the model

m2hms(M) -> {M div 60, M rem 60, 0}.

h2hms(H) -> {H,0,0}.

You can take my word for that this will pass the tests, or try it yourself with make test. Next is the addition and subtraction functions.  They will allow us to subtract and add seconds, minutes and hours from a given time() 3-tuple.

First, we write a test, based on the previously (seemingly) okay functions, for the internal model

prop_subtract_seconds_from_time() ->
    ?FORALL({H,M,S,S2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H < 24,
    ?IMPLIES(S2 =< ((H*60*60)+(M*60)+S),
    begin
        T = {H,M,S},
        clock:sub_sec(T,S2) == clock:s2hms((clock:hms2s(T) - S2))
    end))))).

Here we see that some sanity is made sure of, by never subtracting more time than possible from a time(), with the final precondition. I now purposefully write some dirty code for this

sub_sec({H,M,S},S2) ->
    Hsub = (S div 60) div 60,
    Msub = S2 div 60,
    Ssub = S2 rem 60,
    { H - Hsub, M - Msub, S - Ssub }.

Oh! The test fails, and I remove all the other output that was okay

zen:QCMini zenon$ make test
1>
prop_subtract_seconds_from_time: ......Failed! After 7 tests.
{1,0,1,2}
Shrinking..(2 times)
{1,0,0,1}

Here we see that QuickCheck generated a minimal failing case with Hour = 1, Minutes = 0, Seconds = 0 and Subtracting 1 second fails. Using these values with the model (in the console)

1> clock:s2hms((clock:hms2s({1,0,0}) - 1)).
{0,59,59}
2>

we see that the model seems sane. Removing one second from an hour produces 59 minutes and 59 seconds. All okay. Now what about my broken implementation? Well, obviously it did not account for “lending”. Since we want the actual implementation to be quicker than the model, it would be cheating to copy the model.

Some coding

sub_sec({H,M,S},S2) ->
    Sres  = S - (S2 rem 60),
    Mres = M - (S2 div 60),
    Hres = H - (M div 60),
    case {Sres < 0, Mres > 0, Mres == 0} of
        {false,false,false} -> {Hres - 1, Mres+60, Sres};
        {true,true,_}  -> {Hres, Mres - 1, Sres + 60};
        {true,false,_} -> {Hres - 1, Mres + 59, Sres + 60};
        _ -> {Hres,Mres,Sres}
    end.

and the test

zen:QCMini zenon$ make test
Eshell V5.8.1  (abort with ^G)
1> prop_time_to_seconds: Starting eqc mini version 1.0.1
(compiled at {{2010,6,13},{11,15,30}})
prop_subtract_seconds_from_time: ..............................x..........
.....................................xxx..xx...............x.x..xx...
OK, passed 100 tests

Yes. Nice! But how sure are we that this is not running trivial tests? Well, just check WHAT it is testing, by collecting the testdata! Note the added eqc:collect/2 function for the line after the last precondition.

prop_subtract_seconds_from_time() ->
    ?FORALL({H,M,S,S2},{nat(),nat(),nat(),nat()},
    ?IMPLIES(M < 60,
    ?IMPLIES(S < 60,
    ?IMPLIES(H < 24,
    ?IMPLIES(S2 =< ((H*60*60)+(M*60)+S),
    eqc:collect({H,M,S,S2},
    begin
        T = {H,M,S},
        clock:sub_sec(T,S2) == clock:s2hms((clock:hms2s(T) - S2))
    end)))))).

Running the tests should now spew a lot of test statistics similar to this

OK, passed 100 tests
3% {0,0,0,0}
2% {3,3,0,2}
1% {22,23,28,21}
1% {21,6,24,1}
1% {20,15,13,11}
1% {20,6,8,27}
1% {20,0,9,12}
1% {19,25,30,29}
1% {19,18,9,7}
1% {18,13,25,25}
.....

with A LOT of lines, so if you don’t trust your model OR your implementation, you are welcome to check out the used values by hand. For now, we are good.

I hope this first post on QuickCheck shows that you CAN do TDD with QuickCheck and also wet your appetite for QC if it was your first encounter with it. Next time I will continue on this clock lib and try to show more QC stuff.

Cheers

/G

Testing gen_fsm – Part 6 ATG engine – Implementation


Finally, after all this thinking and all the discussing with ourselves, the moment of implementation is here. Of course, I have been writing this up during the ever so minimal spare time, and will present the final result for your pleasure, and also walk you through the interesting bits and pieces while commenting it. Enjoy!

Overview

The ATG machine consists of 2 application modules and 2 user-supplied modules

  • atg_parsetrans.erl
  • atg_machine.erl
  • test module
  • gen_fsm implementation module

The general idea is that the test module will include the atg_parsetrans as a parse transformation, and the state representation of the gen_fsm (often a record). The test module will then define a set of 3-tuples which act as a kind of Hoare-triples. The atg_machine is then used for using the test module-defined rules and auxiliary functions to randomly test sequences of transitions through the gen_fsm.

In detail – atg_parsetrans.erl

The atg_parsetrans module is the parse_transform library that your test-module should import (for your convenience).  atg_parsetrans will convert the kind-of Hoare Triples to an atg_machine friendly format. The transformation of your test module will then be roughly

  • Encase all expressions in fun’s
  • Store a string representation of the original expression, used for failure reports
  • Put the fun encased Expr and the String representation  into internal atg record
  • Drop the ‘$RULES’ tuple and replace with a list of all the records

The code follows

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% The Automated Test Generator (ATG) parse transformer
%%% Transforms the 3-tuples in the list as second elements of
%%% {'$RULES',[RULES]} into funs() -> RULE end, and creates accessors
%%% for state etc.
%%% Created : 20 Oct 2010 by Gianfranco <zenon@zen.home>
-module(atg_parsetrans).
-export([parse_transform/2]).

%% t(X)        = transformation of X
%% t('$ID')   => erlang:get('$ID')
%% t('$STATE')=> (fun......end)()
%% t('$PROGRAM_RESULT') => erlang:get('$PROGRAM_RESULT')
%% t({'$RULES',[RULE]})=> [t(R) || R <- [RULE]]
parse_transform([File,Module|Rest],_Options) ->
%% Transform with sneaked in rule_fun record
    transform([File,Module,abstr_rule_fun_record()|Rest]).

%% For inserting the used record into the file :)
abstr_rule_fun_record() ->
    {attribute,1,record,
      {rule_fun,
       [{record_field,1,{atom,1,function}},
        {record_field,1,{atom,1,string}}]}}.

%% All occurrences of '$STATE' must be replaced with this
%% call.
% ((fun() ->
%    {status,_,_,[_,_,_,_,Misc]} = sys:get_status(erlang:get('$ID')),
%    AllDatas = lists:flatten([ X || {data,X} <- Misc]),
%    proplists:get_value("StateData",AllData)
%  end)())
transform({atom,_Line,'$STATE'}) ->
    StrFunExp =
        "((fun() ->
            {status,_,_,[_,_,_,_,Misc]} = sys:get_status(erlang:get('$ID')),
            AllDatas = lists:flatten([ X || {data,X} <- Misc]),
            proplists:get_value(\"StateData\",AllDatas)
        end)()).",
    {ok,FunTkns,_} = erl_scan:string(StrFunExp),
    {ok,[FunForm]} = erl_parse:parse_exprs(FunTkns),
    FunForm;
%% All calls to use the other atoms acting as variables need to be changed to
%% proper erlang:get/1 from the process dictionary
transform({atom,_Line,'$PROGRAM_RESULT'}) ->
    StrFunExp = "erlang:get('$PROGRAM_RESULT').",
    {ok,FunTkns,_} = erl_scan:string(StrFunExp),
    {ok,[FunForm]} = erl_parse:parse_exprs(FunTkns),
    FunForm;
transform({atom,_Line,'$ID'}) ->
    StrFunExp = "erlang:get('$ID').",
    {ok,FunTkns,_} = erl_scan:string(StrFunExp),
    {ok,[FunForm]} = erl_parse:parse_exprs(FunTkns),
    FunForm;
transform({tuple,_,[{atom,_,'$RULES'},RuleList]}) ->
    rule_transform(RuleList);
transform(X) when is_tuple(X) ->
    list_to_tuple(lists:map(fun transform/1,tuple_to_list(X)));
transform(X) when is_list(X) ->
    lists:map(fun transform/1,X);
transform(X) -> X.

rule_transform({cons,L1,{tuple,L2,[A,B,C]},Rest}) ->
    {cons,L1,
     {tuple,L2,
      lists:map(
      fun({Expr,OrigExpr}) ->
              ExprStr   = erl_prettypr:format(Expr),
              OrigExprSt= erl_prettypr:format(OrigExpr),
              StructStr = "#rule_fun{function = fun()-> "++ExprStr++" end, "
                          "string = \""++OrigExprSt++"\"}.",
              {ok,Tkns,_} = erl_scan:string(StructStr),
              {ok,[Form]} = erl_parse:parse_exprs(Tkns),
              Form
      end,[{transform(A),A},{transform(B),B},{transform(C),C}])},
     rule_transform(Rest)};
rule_transform(X) -> X.


In order to fully understand this, the prerequisite knowledge is ‘The Abstract Format’. The key parts to know is that the entry-point of this module is the exported one (of course) and that it works on a list of abstract format terms. I then let each function clause of my transform/1 function take care of each specific thing that interests me.

transform({atom,_Line,'$STATE'}) ->
transform({atom,_Line,'$PROGRAM_RESULT'}) ->
transform({atom,_Line,'$ID'}) ->
transform({tuple,_,[{atom,_,'$RULES'},RuleList]}) ->

Besides this, the last three clauses work for traversing the Abstract Format

transform(X) when is_tuple(X) ->
    list_to_tuple(lists:map(fun transform/1,tuple_to_list(X)));
transform(X) when is_list(X) ->
    lists:map(fun transform/1,X);

and identity for let-through on the stuff that is not interesting.

transform(X) -> X.

Also, there is a specific function for traversing the list of 3-tuples holding the rules.  That function is interesting as it shows how I rewrite the code to be fun()-encapsulated and stringified. For this, I use erl_prettypr a very handy Abstract Form to String library, erl_scan the Erlang tokenizer (I hope you understand parsers and programming languages), and the erl_parse for generating replacement Abstract Form code.

Away we go

rule_transform({cons,L1,{tuple,L2,[A,B,C]},Rest}) ->
    {cons,L1,
     {tuple,L2,
      lists:map(
      fun({Expr,OrigExpr}) ->
              ExprStr   = erl_prettypr:format(Expr),
              OrigExprSt= erl_prettypr:format(OrigExpr),
              StructStr = "#rule_fun{function = fun()-> "++ExprStr++" end, "
                          "string = \""++OrigExprSt++"\"}.",
              {ok,Tkns,_} = erl_scan:string(StructStr),
              {ok,[Form]} = erl_parse:parse_exprs(Tkns),
              Form
      end,[{transform(A),A},{transform(B),B},{transform(C),C}])},
     rule_transform(Rest)};
rule_transform(X) -> X.

As can be seen, this function replaces each tuple element Expr with a #rule_fun record where the function field is bound to a fun encasing a parse-transformed version of the original Expr, and the string field is bound to the original un-parse-transformed Expr.

In detail – atg_machine.erl

The atg_machine is the whole .. yeah, machinery. Pumping through the different #rule_fun records, finding the matching ones, selecting one at random and executing.  The coarse steps of the atg_machine are

  • Check if end condition is met
  • Check if maximum iterations are met
  • Find matching set of accessible Rules
  • Pick one Rule at random from accessible rules
  • Execute the program of the picked Rule
  • Check postcondition of the picked Rule

One specially interesting thing is that if no rule can be applied before any endcondition is met, it will be considered a failure.

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% The Automated Test Generator (ATG) machinery, which handles
%%% rule execution and selecetion.
%%% Created : 20 Oct 2010 by Gianfranco <zenon@zen.home>
-module(atg_machine).
-export([run/3]).
-define(MAX_RUNS,100).

-record(rule_fun,{function :: fun() ,
               string   :: string() }).

-record(atg_triple,{pre  :: #rule_fun{},
                prog :: #rule_fun{},
                post :: #rule_fun{}}).

-record(atg_state,{remaining :: integer(),
               passed    :: integer(),
               log       :: [#atg_triple{}],
               state_log :: [any()],
               rules     :: [{#rule_fun{},#rule_fun{},#rule_fun{}}],
               stop_pred :: fun()}).

%------------------------------------------------------------------------------
-spec run([#rule_fun{}],fun(),fun()) -> ok | {error,any()} | true.
run(BaseSetOfRules, GenFsm_Setup,StopPred) ->
    case (catch GenFsm_Setup()) of
        {'EXIT',Reason} -> exit({setup,Reason});
        Result ->
            {A,B,C} = now(),
            random:seed(A,B,C),
            bind_to_id(Result),
            run_until(#atg_state{remaining = ?MAX_RUNS,
                                 passed    = 0,
                                 log       = [],
                                 state_log = [],
                                 rules     = BaseSetOfRules,
                                 stop_pred = StopPred})
    end.

%------------------------------------------------------------------------------
run_until(#atg_state{remaining=0}) -> report_success();
run_until(State) -> check_stopcondition(State).

check_stopcondition(#atg_state{stop_pred = StopPred} = State) ->
    case (catch StopPred()) of
        {'EXIT',Reason} -> exit({stop_predicate,Reason});
        true -> report_success();
        false -> check_hitset(State)
    end.

check_hitset(#atg_state{rules = RuleSet} = State) ->
    HitSet = [ X || X={P,_,_} <- RuleSet, (P#rule_fun.function)() ],
    case HitSet of
        [] -> exit({empty_hitset,fsm_state()});
        _ -> pick_rule(HitSet,State)
    end.

pick_rule(HitSet,State) ->
    {X,Y,Z} = lists:nth(random:uniform(length(HitSet)),HitSet),
    Triple = #atg_triple{pre=X, prog=Y, post=Z},
    use_rule(Triple,State).

use_rule(Triple = #atg_triple{prog=Y},State) ->
    io:format(".",[]),
    case (catch (Y#rule_fun.function)()) of
        {'EXIT',Reason} -> report_error(program,{Reason,Triple},State);
        Result ->
            bind_to_program_result(Result),
            test_postcondition(Triple,State)
    end.

test_postcondition(Triple = #atg_triple{post=Z},
                   #atg_state{remaining = Remaining,
                              passed = Passed,
                              log = Log,
                              state_log = SLog}=State) ->
    case (catch (Z#rule_fun.function)()) of
        {'EXIT',Reason} -> report_error(postcondition,{Reason,Triple},State);
        false-> report_error(postcondition,{postcondition_false,Triple},State);
        true -> run_until(State#atg_state{remaining=Remaining-1,
                                          passed=Passed+1,
                                          log=[Triple|Log],
                                          state_log=[fsm_state()|SLog]})
    end.

%------------------------------------------------------------------------------
bind_to_id(Value) -> erlang:put('$ID',Value).
bind_to_program_result(Value) -> erlang:put('$PROGRAM_RESULT',Value).    

report_success() ->
    io:format("~n~n*** Automated Testing Successfull ***~n~n",[]),
    ok.

report_error(Phase,{Cause,Triple},#atg_state{passed=Passed,
                                             log=Log,
                                             state_log=SLog}) ->
    io:format("~n~n*** ERROR: Automated Testing Failure ***~n"
              " Failed after: ~p ~n"
              " Failure phase: ~p~n"
              " Cause: ~p~n"
              " Rule: ~p~n"
              " State: ~p~n"
              " Log: ~p~n"
              " States: ~p~n"
              " $ID: ~p~n"
              " $PROGRAM_RESULT: ~p~n",
              [Passed,Phase,Cause,triple_to_str(Triple),
               fsm_state(),[triple_to_str(T) || T <- Log],SLog,
               erlang:get('$ID'),
               erlang:get('$PROGRAM_RESULT')]),
    % Kill the gen_fsm?
    exit(erlang:get('$ID'),kill),
    {error,Cause}.

fsm_state() ->
    ((fun() ->
              {status,_,_,[_,_,_,_,Misc]} = sys:get_status(erlang:get('$ID')),
              AllDatas = lists:flatten([ X || {data,X} <- Misc]),
              proplists:get_value("StateData",AllDatas)
      end)()).   

triple_to_str(#atg_triple{pre=#rule_fun{string=PreStr},
                          prog=#rule_fun{string=ProgStr},
                          post=#rule_fun{string=PostStr}}) ->
    {PreStr,ProgStr,PostStr}.

Some interesting things to discuss could be that this machinery does not need parse_transforming and never needs to know the internal format of the gen_fsm. So, it’s gen_fsm implementation agnostic. It will run until the first thing happens of [condition failure, end condition success, 100 iterations passed].

Other than that, I believe the code is fairly self explanatory. Of course, be welcome to ask in the comments.

A test module

A user supplied test module (call it what you want, for example: demo.erl) should define that parse-transformation is to be done with the atg_parsetrans module.

-compile({parse_transform, atg_parsetrans}).

The test module must also define the internally used state record of the gen_fsm if such a record is used and it will be very handy if the following functions are exported from the test module

-export([rules/0,is_done/0,start/0]).

why are these functions so handy, and what should they contain?

The rules() function is an obvious way to pass the Rule-list to the machinery, thus, this function is defined to return {‘$RULES’,  [....]} where the dots ‘.’ in the list are the 3-tuples. Something like this

rules() ->
  {'$RULES',
    [
     { not reg(seller), do_reg(seller), reg(seller,'$PROGRAM_RESULT')},
     { not reg(buyer), do_reg(buyer),reg(buyer,'$PROGRAM_RESULT')}
    ]}.

The is_done/0 function will serve as the end predicate for the machinery, so it will know when it’s “done”, can be set to return false if MAX_ITERATIONS of tests are wanted.
Example could be

is_done() -> '$STATE'#state.deal_done == true.

And the final export start/0 will serve as the function that return the value that will be bound for the gen_fsm accessing.
Example

start() ->
    {ok,Pid} = tradepost:start_link(),
    Pid.

Thus, we understand, that the majority of the work lies in writing the rules/0 function and all the 3-tuples defining the rules. The programmer will have to tip in some extra work into the predicates.  Now for the example test module.

Usage examples

My own test module (demo.erl) will be posted below, together with the success output and some error examples when we do some bad modifications to the tradepost.erl code.
First the demo.erl code

-module(demo).
-export([rules/0,is_done/0,start/0]).
-compile({parse_transform, atg_parsetrans}).

%% Must be included for gen_fsm
-record(state,{item,cash,seller,buyer,seller_accept,buyer_accept,deal_done}).

start() ->
    {ok,Pid} = tradepost:start_link(),
    Pid.

% ------------------------------------------------------------------------------
% List of all rules - mock code!
% ------------------------------------------------------------------------------
rules() ->
    {'$RULES',
     [
      % Seller and buyer registration,
      { not reg(seller), do_reg(seller), reg(seller,'$PROGRAM_RESULT')},
      { not reg(buyer),  do_reg(buyer),  reg(buyer,'$PROGRAM_RESULT')},
      { reg(seller), insert(item), item_is('$PROGRAM_RESULT')},
      { reg(buyer),  insert(cash),  cash_is('$PROGRAM_RESULT')},
      { tradepost_has(cash), remove(cash), not tradepost_has(cash) },
      { tradepost_has(item), remove(item), not tradepost_has(item) },
      { tradepost_has(cash) andalso
        tradepost_has(item), closedeal(), return_value_is([item,100])}
     ]
    }.

% Deal is done when deal_done is true
is_done() -> '$STATE'#state.deal_done == true.

reg(seller) -> '$STATE'#state.seller =/= undefined;
reg(buyer) ->  '$STATE'#state.buyer =/= undefined.

reg(seller,Passwd) -> '$STATE'#state.seller == Passwd;
reg(buyer,Passwd) -> '$STATE'#state.buyer == Passwd.

% The identifier of the gen_fsm is accessed through '$ID'
do_reg(buyer) ->
    Password = generate_passwd(buyer),
    tradepost:buyer_identify('$ID',Password),
    Password;
do_reg(seller) ->
    Password = generate_passwd(seller),
    tradepost:seller_identify('$ID',Password),
    Password. % This line ensures that '$PROGRAM_RESULT' is bound to
              % the generated password, for the Postcondition to access.

insert(item) ->
    Password = '$STATE'#state.seller,
    Item = generate_item(),
    tradepost:seller_insert('$ID',Item,Password),
    Item; % Ensure '$PROGRAM_RESULT is Item
insert(cash) ->
    Password = '$STATE'#state.buyer,
    Cash = generate_cash(),
    tradepost:buyer_insert('$ID',Cash,Password),
    Cash. % Ensure '$PROGRAM_RESULT is Cash

% Check that item is the specified item (argument).
item_is(GivenItem) -> '$STATE'#state.item == GivenItem.

% Check that cash is specified amount (argument)
cash_is(GivenAmount) -> '$STATE'#state.cash == GivenAmount.

% Testing state to contain cash xor item
tradepost_has(cash) -> '$STATE'#state.cash =/= undefined;
tradepost_has(item) -> '$STATE'#state.item =/= undefined.

remove(cash) ->
    Passwd = '$STATE'#state.buyer,
    tradepost:buyer_withdraw('$ID',Passwd);
remove(item) ->
    Passwd = '$STATE'#state.seller,
    tradepost:seller_withdraw('$ID',Passwd).

% Deal closing, is a parallelized action.
% therefore I have to bind the value of '$ID' to the atom '$ID' in the
% new process without writing '$ID' as an atom when doing so! o<:D
closedeal() ->
    ID   = '$ID',
    Self = self(),
    spawn_link(fun() ->
                       erlang:put(list_to_atom("$ID"),ID),
                       Passwd = '$STATE'#state.buyer,
                       ItemAndCash = tradepost:get_contents('$ID',Passwd),
                       Self ! tradepost:buyer_deal('$ID',Passwd,ItemAndCash)
               end),
    spawn_link(fun() ->
                       erlang:put(list_to_atom("$ID"),ID),
                       Passwd = '$STATE'#state.seller,
                       ItemAndCash = tradepost:get_contents('$ID',Passwd),
                       Self ! tradepost:seller_deal('$ID',Passwd,ItemAndCash)
               end),
    [receive X -> X end,receive Y -> Y end].                            

return_value_is(X)-> lists:sort(X) == lists:sort('$PROGRAM_RESULT').

generate_passwd(buyer) -> buyer;
generate_passwd(seller) -> seller.

generate_item() -> item.
generate_cash() -> 100.

There, this is the test module I have for the tradepost.erl module, and here comes the compilation and running. I have already compile atg_parsetrans and atg_machine to ebin/ and therefore add ebin/ to the path when compiling demo.erl

zen:EUnitFSM zenon$ erlc -o ebin/ src/demo.erl -pa ebin/
zen:EUnitFSM zenon$ erl -pa ebin/
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> atg_machine:run(demo:rules(),fun demo:start/0, fun demo:is_done/0).
...........

*** Automated Testing Successfull ***

ok
2>

The 11 dots seen show that 11 cycles where run before the is_done/0 predicate returned true. Now, if we wish to see some errors, let’s make a small modification (that actually was a bug I discovered thanks to my own tool) in tradepost.erl

On line 118, change the last don’t care to undefined.

{cash_received,item,Pwd,_,_,_} -> {ok, cash_received, undef(item,LoopD)};

like this, and save + recompile

{cash_received,item,Pwd,_,_,undefined} -> {ok, cash_received, undef(item,LoopD)};

now, running should produce the nice error output format

16> atg_machine:run(demo:rules(),fun demo:start/0, fun demo:is_done/0).
........

*** ERROR: Automated Testing Failure ***
 Failed after: 7
 Failure phase: postcondition
 Cause: postcondition_false
 Rule: {"tradepost_has(item)","remove(item)","not tradepost_has(item)"}
 State: {state,item,100,seller,buyer,undefined,undefined,false}
 Log: [{"reg(buyer)","insert(cash)","cash_is('$PROGRAM_RESULT')"},
       {"tradepost_has(cash)","remove(cash)","not tradepost_has(cash)"},
       {"reg(buyer)","insert(cash)","cash_is('$PROGRAM_RESULT')"},
       {"reg(seller)","insert(item)","item_is('$PROGRAM_RESULT')"},
       {"not reg(buyer)","do_reg(buyer)","reg(buyer, '$PROGRAM_RESULT')"},
       {"reg(seller)","insert(item)","item_is('$PROGRAM_RESULT')"},
       {"not reg(seller)","do_reg(seller)","reg(seller, '$PROGRAM_RESULT')"}]
 States: [{state,item,100,seller,buyer,undefined,undefined,false},
          {state,item,undefined,seller,buyer,undefined,undefined,false},
          {state,item,100,seller,buyer,undefined,undefined,false},
          {state,item,undefined,seller,buyer,undefined,undefined,false},
          {state,item,undefined,seller,buyer,undefined,undefined,false},
          {state,item,undefined,seller,undefined,undefined,undefined,false},
          {state,undefined,undefined,seller,undefined,undefined,undefined,
                 false}]
 $ID: <0.88.0>
 $PROGRAM_RESULT: error
** exception exit: killed
17>

Now, first of, the 8 dots say that 8 cycles where performed, and the Error output says that we failed on the 8th cycle, that is, after 7 completed. The phase of the error was the postcondition of the selected rule. The selected rule is shown on the “Rule:” line. The current state in which we failed is shown after “State:”.

The “Log:” shows a list of all the processed and passed rules, most recent nearest the top of the screen. This is also true for the “States:” output, which serves as a log for us. The internal process dictionary bindings for ‘$ID’ and ‘$PROGRAM_RESULT’ are also shown.
In short, almost everything we need for debugging is here.

Now for a really terrifyingly delicious comment! I had to run this several (6 times) before this error was shown.

Why? Because the testing is randomised. And some of the corner cases will thus not be triggered easily during short runs.  Let us undo the faulty change and introduce an easier one into the program. Let’s forget that buyer inserts cash and not item. (Line 42, change cash  to item : and recompile!)

4> atg_machine:run(demo:rules(),fun demo:start/0, fun demo:is_done/0).
...

*** ERROR: Automated Testing Failure ***
 Failed after: 2
 Failure phase: postcondition
 Cause: postcondition_false
 Rule: {"reg(buyer)","insert(cash)","cash_is('$PROGRAM_RESULT')"}
 State: {state,undefined,undefined,seller,buyer,undefined,undefined,false}
 Log: [{"not reg(buyer)","do_reg(buyer)","reg(buyer, '$PROGRAM_RESULT')"},
       {"not reg(seller)","do_reg(seller)","reg(seller, '$PROGRAM_RESULT')"}]
 States: [{state,undefined,undefined,seller,buyer,undefined,undefined,false},
          {state,undefined,undefined,seller,undefined,undefined,undefined,
                 false}]
 $ID: <0.43.0>
 $PROGRAM_RESULT: 100
** exception exit: killed
5>

Bam! Reading the error reason, it becomes apparent that something is not working in the cash insertion! the cash in the state does not match ‘$PROGRAM_RESULT’ which was 100! Looking at the State, it’s undefined. Bummer! :)

This was the final part of the home-written ATG, I will presumably put in on git if the demand is high enough.

Cheers

/G

EUnit – Common special errors


Now and then, people ask me EUnit questions, and sometimes the problems consist of less obvious nuts and bolts.  So, let’s explore some special and yet common errors.

For the purpose of this post, I will be using my previously seen numberserver (reposted here for your convenience).

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created :  4 Oct 2010 by Gianfranco <zenon@zen.home>
-module(numberserver).
-export([start/0,stop/0,op/2,get/0,init/0]).
start() ->
    Pid = spawn_link(?MODULE,init,[]),
    register(?MODULE,Pid),
    ok.
stop() -> ?MODULE ! stop, unregister(?MODULE).
op(Op,Num) -> ?MODULE ! {Op,Num}, ok.
get() ->
    ?MODULE ! {get_result,self()},
    receive
        X -> X
    end.
init() -> loop(basic).
loop(E) ->
    receive
        stop -> ok;
        {get_result,From} ->
            From ! E, loop(E);
        {Op,Num} -> loop(result(Op,E,Num))
    end.
result(_,basic,X) -> X;
result('+',X,Y) -> X + Y;
result('*',X,Y) -> X * Y;
result('-',X,Y) -> X - Y;
result('/',X,Y) -> X / Y.

Enough with the reminiscence.

*** test module not found ***   ::ok

This one is very special, and also very common. People sometimes forget that Instantiators (remember, instantiators are functions that take the setup/0 result as input argument, and returns a TestSet or Simple Test Object) must return TestSet or Simple Test Object. Classic erroneous code follows.

example_a_test_() ->
    {setup,
     fun()  -> numberserver:start() end,
     fun(_) -> numberserver:stop() end,
     % Using instantiator, not returning: Test xor Simple Test Object!
     fun(_) ->
         numberserver:op('+',1),
         ?assertEqual(1,numberserver:get())
     end
    }.

and the reduced error output for this

1> ======================== EUnit ========================
module 'numberserver'
  module 'numberserver_tests'
undefined
*** test module not found ***
::ok

=======================================================
  Failed: 0.  Skipped: 0.  Passed: 0.
One or more tests were cancelled.

This error often (most understandably!) confuses people as it does not say: bad test descriptor or something more helpful. The correct code could have been written as

example_a_test_() ->
    {setup,
     fun()  -> numberserver:start() end,
     fun(_) -> numberserver:stop() end,
     fun(_) ->
        fun() ->
             numberserver:op('+',1),
             ?assertEqual(1,numberserver:get())
        end
     end
     }.

Test passed.                (yet it’s wrong!)

Sometimes people use {setup,….}, {foreach,…} or any other fixture inside a _test() [simple test function] instead of a _test_() [test-generating function]. EUnit will then seem to execute the test, but does not do it. Proof follows below, with a faulty test-case and the output.

-module(numberserver_tests).
-include_lib("eunit/include/eunit.hrl").

example_b_test() ->
    {setup,
     fun()  -> numberserver:start() end,
     fun(_) -> numberserver:stop() end,
     fun() ->
             numberserver:op('+',3),
             numberserver:op('*',4),
             ?assertMatch(13,numberserver:get())
     end}.

And the console, deceives us with

zen:EUnitProblems zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitProblems zenon$ erl -pa ebin/ -eval 'eunit:test(numberserver,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'numberserver'
  numberserver_tests: example_b_test (module 'numberserver_tests')...ok
  [done in 0.002 s]
=======================================================
  Test passed.

1>

This shows us how IMPORTANT, it is to ALWAYS have a failing test first.  If we would have a failing test first, because it’s obvious that it’s failing, this type of errors can not sneak onto us.

Correct code would have been

-module(numberserver_tests).
-include_lib("eunit/include/eunit.hrl").

example_b_test_() ->
    {setup,
     fun()  -> numberserver:start() end,
     fun(_) -> numberserver:stop() end,
     fun() ->
             numberserver:op('+',3),
             numberserver:op('*',4),
             ?assertMatch(13,numberserver:get())
     end}.

and console output

zen:EUnitProblems zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitProblems zenon$ erl -pa ebin/ -eval 'eunit:test(numberserver,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'numberserver'
  module 'numberserver_tests'
    numberserver_tests: example_b_test_...*failed*
::error:{assertMatch_failed,[{module,numberserver_tests},
                           {line,11},
                           {expression,"numberserver : get ( )"},
                           {expected,"13"},
                           {value,12}]}
  in function numberserver_tests:'-example_b_test_/0-fun-2-'/0

    [done in 0.003 s]
  [done in 0.003 s]
=======================================================
  Failed: 1.  Skipped: 0.  Passed: 0.

This post will be updated as time goes and people ask questions, etc. So check it out once in a while!

 

Cheers

/G

Testing gen_fsm – Part 5 ATG Engine


The first post on the Automatest Test Generator for gen_fsm testing gave us a good idea on Rule basis, what we should brainstorm on now, is how the internal machinery should be for this to work as expected. By now this seems to take on a form outside of EUnit, and I shall therefore drop the EUnit prefix on the titles (for now). Without further ado, let us start writing some free-style mock code of our machine, firs off, a basic skeleton.

Second Iteration – Machinery Brainstorming

%------------------------------------------------------------------------------
% Minimal for now, there is a lot of bells and whistles that can be added.
%------------------------------------------------------------------------------
run(BaseSetOfRules, GenFsm_Setup) ->
    case (catch GenFsm_Setup()) of
        {'EXIT',Reason} -> report_error(setup,Reason);
        Result ->
            bind_to_id(Result),
            run_until(BaseSetOfRules,100)
    end.

%------------------------------------------------------------------------------
% No more iterations to run, and no crashes, all is okay
run_until(_RulesSet, 0) -> report_success();
% An iteration: Find valid rules (== positive precondition) and run one of them
run_until(RuleSet,Iterations) when Iterations > 0 ->
    HitSet = [ X || X={Precond,_,_} <- RuleSet, Precond() ],
    {_,Program,PostCondition} = lists:nth(random:uniform(length(HitSet)),HitSet),
    case (catch Program()) of
        {'EXIT',Reason} -> report_error(program,Reason);
        Result ->
            bind_to_program_result(Result),
            % Test postcondition
            case (catch PostCondition()) of
                {'EXIT',Reason2} -> report_error(postcondition,Reason2);
                true -> run_until(RuleSet,Iterations-1);
                false-> report_error(postcondition,postcondition_false)
            end
    end.

In the code seen above, the BaseSetOfRules would be the result of the previously written rule() function, a list of rules. The second argument GenFsm_Setup would correspond to some kind of setup function to run before starting (think in {setup,fun setup/0,fun cleanup/0} terms). This function call should give a result which can be used to bind with ‘$ID’.

A very important detail is that the predicates PostCondition and PreCondition must not change the State of the gen_fsm!

Given the code above, we kind of have an idea of the machinery and it’s syntax. The natural question is then, how do we put this together through parse_transforming? The special parts that need consideration is the binding of values  to

  • ‘$ID’
  • ‘$PROGRAM_RESULT’
  • ‘$STATE’

And also the accessing of these values. Of course, another detail worth mentioning is the fact that the machinery seems to treat the Precondition, Program and Postcondition as they where all function objects. This brain-storm design implies that all Expressions given in the previous Rule post are encapsulated into a fun during the parse_transforming (aha!).

Let this discussion of value binding and accessing be the focus for the next iteration.

Third Iteration – Binding and Accessing

For the binding of values, the erlang process dictionary fits perfectly. But before going further, let me light up the usual warning lamps about using the process dictionary and tell you how bad it is and how you should never do it, and give you the link to the info about it. Now that we got that out of the way, here is the simple implementation the machinery needs to pull this of.

% Use the process dictionary of the machinery for these special values
bind_to_id(Value) -> erlang:put('$ID',Value).
bind_to_program_result(Value) -> erlang:put('$PROGRAM_RESULT',Value).

Great, but what about the ‘$STATE’ binding, do we need the process dictionary for it?The answer is: No. The gen_fsm state is a fresh info value that needs to be  requests each time the ‘$STATE’ atom is seen in the user (rule) code,  thus we would expect the parse_transform to exchange the  ‘$STATE’ atom with a special function call.

What about accessing the values bound to these atoms? Well, for the process dictionary bound ones, just let the parse_transform exchange all occurrences with process dictionary lookups. Looking at the first brainstorm syntax, and how it evolves:

 % Old brain-storm syntax
   { not reg(seller), do_reg(seller), reg(seller,generated_pwd)},

 % Updated brain-storm syntax
   { not reg(seller), do_reg(seller), reg(seller,'$PROGRAM_RESULT')},

 % Parse_transform ed
   { fun() -> not reg(seller) end ,
     fun() -> do_reg(seller) end,
     fun() -> reg(seller,erlang:get('$PROGRAM_RESULT') end },

As for the ‘$STATE’ atom, we would expect the parse_transform to replace it with something roughly similar to this “do not try this at home kids – I am a professional“.

% Old brain-storm syntax
 reg(seller) -> '$STATE'#state.seller =/= undefined;

% Parse_transform:d into this
 reg(seller) ->
   ((fun() ->
      {status,_,_,[_,_,_,_,Misc]} = sys:get_status(erlang:get('$ID')),
      AllDatas = lists:flatten([ X || {data,X} <- Misc]),
      proplists:get_value("StateData",AllData)
     end)())#state.seller =/= undefined;

Let me guide you through it in case you need it. According to our manual pages, the sys:get_status will return a nifty status tuple where the last element is a list which in turn has a last element which is a proplist. That proplist contains the state representation which can be accessed through the Key “StateData”. For our purposes, this is great. Please note: “… Callback modules for gen_server and gen_fsm can also customise the value of Misc by exporting a format_status/2 function that contributes module-specific information…“.

This state accessing might have to be done in several places with close proximity, thus I encase it into an instantly executed functional object (fun), which is parenthesised in order to make it usable by any type accessing method of choice.

Now we have a kind of good idea on how we would like the Rules to be written. Also we kind of know how we would like the machinery to work, thus, the next iteration should be the start of the  actual implementation.

Cheers

/G

Meanwhile ….


Awaiting the time to finish my next post, I created this vi rage post with the help from the rage editor :)

Enjoy

The next post will be here soon, also, I’ll write a clarifying post on common EUnit format mistakes.

EUnit Testing gen_fsm – Part 4 Automated Test Machinery


Using the idea from the last post, we have a machinery which takes a set of Rules as input, a gen_fsm, setup and cleanup. To develop this, there is once again a big question, will this be done with parse transforms or using macros and functions? To try and answer this question along with how it should work, let’s write some mock-code. This will be the first iteration of the ATG (Automated Test Generator).

First Iteration – Brainstorm

For this first iteration, it gets easier if we break it down into some use-cases, first off, the battery of Rules,

% ------------------------------------------------------------------------------
% List of all rules - mock code!
% ------------------------------------------------------------------------------
rules() ->
    [
     % Seller and buyer registration,
     % The postcondition needs to access the password which was generated in
     % the program so that we can verify that the tradepost holds the correct
     % registration.
     { not reg(seller), do_reg(seller), reg(seller,generated_pwd)},
     { not reg(buyer),  do_reg(buyer),  reg(buyer,generated_pwd)},

     % Insert item / cash, should insert random amount of cash and random
     % items. The PostCond must be able to test the existence of that item / cash
     % and thus must have access to the generated item / cash.
     { reg(seller), insert(item), item_is(inserted_item)},
     { reg(buyer),  insert(cash),   cash_is(inserted_amount)},

     % Removal of cash or item, remove needs to use the password which
     % the tradepost currently has.
     { tradepost_has(cash), remove(cash), not tradepost_has(cash) },
     { tradepost_has(item), remove(item), not tradepost_has(item) },

     % Close a deal, we must check that the _returned value_ is the correct one.
     % We must pass on the returned value to the postcond. The Postcond must
     % then also have access to the original item and cash.
     { tradepost_has(cash)
       andalso tradepost_has(item), closedeal(), return_value_is(??)}
    ].

As can be seen, I added some comments on thoughts that deserve discussion. It will often be the case that the postcondition needs to refer to a value which is the result of the Program. Such an example can be the test for correct registration. For that case, we do not only need access to the state, but also the generated password which was used. How to fix it? Bind the Program result to a special atom ‘$PROGRAM_RESULT’, also, bind the state to a special atom ‘$STATE’. Problem solved. Please note that such a solution , is leaning towards parse-transforms.

% The State is accessed through the special  atom
% '$STATE', and the module must include the state record.
reg(seller) -> '$STATE'#state.seller =/= undefined;
reg(buyer) ->  '$STATE'#state.buyer =/= undefined.

% Passwd can be result from Program, thus the call in
% the Rule may then be { ..... , reg(seller,'$PROGRAM_RESULT')}
reg(seller,Passwd) -> '$STATE'#state.seller == Passwd;
reg(buyer,Passwd) -> '$STATE'#state.buyer == Passwd.

Another special binding added to this machinery, is the identifier for the gen_fsm, this could be a registered name or a Pid. The usage of this is shown below, where a registration is performed, and the password is returned for the Postcondition.

% The identifier of the gen_fsm is accessed through '$ID'
do_reg(seller) ->
    Password = generate_passwd(),
    tradepost:seller_identify('$ID',Password),
    Password. % This line ensures that '$PROGRAM_RESULT' is bound to
             % the generated password, for the Postcondition to access.

How the ‘$ID’ is bound, is unclear for now, but that can be taken care of later, remember that we are just writing mock code to get an idea of what we possibly want. Going from the top to bottom, how would we love to see the insertion?

% This of course also uses the $ID, but also the state to access the password.
% seems only fair to return the inserted item for this function as that
% probably is what we wish, ... always.
insert(item) ->
    Password = '$STATE'#state.seller,
    Item = generate_item(),
    tradepost:seller_insert('$ID',Item,Password),
    Item; % Ensure '$PROGRAM_RESULT is Item
insert(cash) ->
    Password = '$STATE'#state.buyer,
    Cash = generate_cash(),
    tradepost:buyer_insert('$ID',Cash,Password),
    Cash; % Ensure '$PROGRAM_RESULT is Cash

Coupled to this, Now, how to test that the cash is the amount we wish, and that the item is the sought one?

% Check that item is the specified item (argument).
item_is(GivenItem) -> '$STATE'#state.item == GivenItem.

% Check that cash is specified amount (argument)
cash_is(GivenAmount) -> '$STATE'#state.cash == GivenAmount.

Or simpler, just test existence?

% Testing state to contain cash xor item
tradepost_has(cash) -> '$STATE'#state.cash =/= undefined;
tradepost_has(item) -> '$STATE'#state.item =/= undefined.

Likewise, removal should be easy with the help of ‘$ID’ and ‘$STATE’

% Utilizing both state and id, but nice thing is that result
% of withdraw function will be bound to $PROGRAM_RESULT in this mockup.
remove(cash) ->
    Passwd = '$STATE'#state.buyer,
    tradepost:buyer_withdraw('$ID',Passwd);
remove(item) ->
    Passwd = '$STATE'#state.seller,
    tradepost:seller_withdraw('$ID',Passwd);

Nearing the end of the first iteration brainstorming, what remains is a way to program the deal closing and a way to test the return value of this (a way to test the return value of the deal closing), ‘$PROGRAM_RESULT’ is still our friend, but it would be good to have some kind of parallelism. Once again, how would we like to see this written?

% Deal closing, is a parallelized action.
% Return value should still be bound to '$PROGRAM_RESULT'
% Could be bound in the form [ A , B ] for this example.
% Inspiration from the parallel keyword in EUnit.
closedeal() ->
    {'$PARALLEL',
     [ begin
           Pass = '$STATE'#state.buyer,
           ItemAndCash = tradepost:get_contents('$ID',Pass),
           tradepost:buyer_deal('$ID',Pass,ItemAndCash)
       end,
       begin
           Pass = '$STATE'#state.seller,
           ItemAndCash = tradepost:get_contents('$ID',Pass),
           tradepost:seller_deal('$ID',Pass,ItemAndCash)
       end]}.

This seems to do the trick,  the idea is to pass a list of  functions which are executed in parallel, the result of each one is bound to an element in the generated list which is bound to ‘$PROGRAM_RESULT’. As this seems that this covers the first iteration of brainstorming, it could be nice to sum it up.

Summary of First Brainstorming session

The Automated Test Generator machinery has a set of Rules, each Rule is modeled as a 3-tuple, The first element in the 3-tuple is the Precondition that has to be met for the Rule to possibly take action. The second element is a Program which is executed, iff the Precondition holds and the Rule is chosen. The third element is a Postcondition that must hold once the Program of the Rule is executed.

The different special syntax elements which have been identified now are

  • ‘$STATE’,  the internal state of the gen_fsm being tested.
  • ‘$ID’,  identifier of the gen_fsm, can be a registered name, a registered name on a node or a Pid.
  • ‘$PROGRAM_RESULT’, the result of the Program (middle element in the Rule)
  • ‘$PARALLEL’, keyword for marking that functions are to be executed in parallel, and the joint result is bound to ‘$PROGRAM_RESULT’

Besides these special syntactical elements, we should try structuring the thoughts a bit more.

rule() = { precondition(), program(), postcondition() }

precondition() = expr() -> true | false.
program()    = expr() -> any().
postcondition()= expr() -> true | false.

Maybe this definition feels a bit weak and lame, but as for now, it captures what we wish, next brainstorming session should focus on the machinery of our automated test generator.

Cheers

EUnit Testing gen_fsm – Part 3


This third part will add the deal closure API as well as refactoring the code, we will also see the inception of the Automated Test Generating machinery.

Sixth Iteration – Deal closing API and the beginning of the ATG machinery

Buyer and Seller can now insert their items, and wish to close the deal, for this we need some kind of “security”. To implement this, the buyer and seller must both agree on the item and the cash. They both query the tradepost for it’s contents, get the item name and the cash amount, then if they agree, they can send an okay to the tradepost with the same item name and the same cash and their Pid. If the tradepost contains an item with that name, and that cash, the tradepost sends the item to the buyer and the cash to the seller. The tradepost then terminates.

For this parallelised receive, the syntax of the symbolic language is extended with

{parallel,call,MODULE,FUNCTION,ARGLIST,REPLYMATCH}

Which gives the new code in eunit_fsm.erl as

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created : 19 Sep 2010 by Gianfranco <zenon@zen.home>
-module(eunit_fsm).
-export([translateCmd/2,get/2]).
-define(Expr(X),??X).
translateCmd(Id,{state,is,X}) ->
    case get(Id,"StateName") of
        X -> true;
        _V ->  .erlang:error({statename_match_failed,
                              [{module, ?MODULE},
                               {line, ?LINE},
                               {expected, X},
                               {value, _V}]})
    end;
translateCmd(_Id,{call,M,F,A,X}) ->
    case apply(M,F,A) of
        X -> ok;
        _V ->  .erlang:error({function_call_match_failed,
                              [{module, ?MODULE},
                               {line, ?LINE},
                               {expression, "apply("++atom_to_list(M)++","++atom_to_list(F)++","++
                                ?Expr(A)},
                               {expected, X},
                               {value, _V}]})
    end;
translateCmd(Id,{loopdata,is,X}) ->
    case tl(tuple_to_list(get(Id,"StateData"))) of
        X    -> true;
        _V ->    .erlang:error({loopdata_match_failed,
                                [{module, ?MODULE},
                                 {line, ?LINE},
                                 {expected, X},
                                 {value, _V}]})
    end;
translateCmd(Id,{parallel,call,M,F,A,X}) ->
    spawn_link(fun() -> translateCmd(Id,{call,M,F,A,X}) end).             

% StateName or StateData
get(Id,Which) ->
    {status,_Pid,_ModTpl, List} = sys:get_status(Id),
    AllData = lists:flatten([ X || {data,X} <- lists:last(List) ]),
    proplists:get_value(Which,AllData).

To pull this off, we need to extend the state with a holder for the agreed parts Pids, this will reveal a flaw with the current syntax that has not been so evident until now (bu surely nagging) that will need to be countered.

closedeal(Pid) ->
    ?fsm_test(Pid,
              "Seller Identifies, Buyer Identifies, Seller Inserts Item "
              ", Buyer Inserts Cash, Seller Gets Contents, Buyer Gets Contents "
              ", Buyer Agrees, Seller Agrees ",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {call,tradepost,get_contents,[Pid,s],{ring,100}},
               {call,tradepost,get_contents,[Pid,b],{ring,100}},
               {parallel,call,tradepost,seller_deal,[Pid,s,{ring,100}],100},
               {parallel,call,tradepost,buyer_deal,[Pid,b,{ring,100}],ring}
              ]).

Writing the code and also modifying the state record, the result is

1> ======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly (Started Properly Test)...[0.001 s] ok
    tradepost_tests: identify_seller (Identify Seller Test)...ok
    tradepost_tests: insert_item (Insert Item Test)...ok
    tradepost_tests: withdraw_item (Withdraw Item Test)...ok
    tradepost_tests: identify_buyer (Identify Buyer Test)...ok
    tradepost_tests: insert_cash (Insert Cash Test)...ok
    tradepost_tests: withdraw_cash (Withdraw Cash Test)...ok
    tradepost_tests: interleaving1 (Seller Identifies, Inserts Item, then Buyer Identiefies)...ok
    tradepost_tests: interleaving2 (Buyer Identifies, Inserts Item, then Seller Identiefies)...ok
    tradepost_tests: interleaving3 (Buyer Identifies, Seller Identifies, buyer inserts cash)...ok
    tradepost_tests: interleaving4 (Seller Identifies, Buyer Identifies, Seller inserts item)...ok
    tradepost_tests: interleaving5 (Seller Identifies, Seller inserts item, Buyer Identifies,Buyer Inserts Cash)...ok
    tradepost_tests: interleaving6 (Seller Identifies, Seller inserts item, Buyer Identifies,Seller Withdraws Item, Buyer Inserts Cash, Seller Inserts ItemBuyer Withdraws Cash, Seller Withdraws Item)...ok
    tradepost_tests: closedeal (Seller Identifies, Buyer Identifies, Seller Inserts Item , Buyer Inserts Cash, Seller Gets Contents, Buyer Gets Contents , Buyer Agrees, Seller Agrees )...ok
    [done in 0.045 s]
  [done in 0.045 s]
=======================================================
  All 14 tests passed.

1>

Greatness, but, in order for this to work, some modification had to be done to all previous tests! We had to extend the state-record with new fields, and the result is that the loopdata assertion had to be changed in each test! What if we had 1000 previous tests? *pulls hair in desperation*. Also, after this success, I made some refactoring, something that _should_ be done after each tests + code + success. This is also a great thing, once you have a nice big set of tests, refactoring can always be tested to preserve the logic by running your tests *thumbs up*.

Anyway, here is the cut and paste version of  tradepost.erl

%%%-------------------------------------------------------------------
%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created :  2 Sep 2010 by Gianfranco <zenon@zen.home>
%%%-------------------------------------------------------------------
-module(tradepost).
-behaviour(gen_fsm).
% ------------------------------------------------------------------------------
% API
-export([start_link/0,stop/1,seller_identify/2,seller_insert/3,seller_withdraw/2,
         buyer_identify/2,buyer_insert/3,buyer_withdraw/2,get_contents/2,
         seller_deal/3,buyer_deal/3]).
% ------------------------------------------------------------------------------
% States
-export([pending/3,item_received/3,cash_received/3]).
% ------------------------------------------------------------------------------
% gen_fsm callbacks
-export([init/1, handle_event/3, handle_sync_event/4, handle_info/3,terminate/3,
         code_change/4]).
% ------------------------------------------------------------------------------
% State representation
-record(state, {item,cash,seller,buyer,seller_accept,buyer_accept}).

% ------------------------------------------------------------------------------
% API
start_link() -> gen_fsm:start_link(?MODULE, [], []).
stop(Pid) -> gen_fsm:sync_send_all_state_event(Pid,stop).
seller_identify(TP,Pwd) ->
    gen_fsm:sync_send_all_state_event(TP,{identify,seller,Pwd}).
seller_insert(TP,Item,Pwd) ->
    gen_fsm:sync_send_event(TP,{insert,item,Item,Pwd}).
seller_withdraw(TP,Pwd) ->
    gen_fsm:sync_send_event(TP,{withdraw,item,Pwd}).
seller_deal(TP,Pwd,{Item,Cash}) ->
    gen_fsm:sync_send_all_state_event(TP,{accept,seller,Pwd,{Item,Cash}}).
buyer_deal(TP,Pwd,{Item,Cash}) ->
    gen_fsm:sync_send_all_state_event(TP,{accept,buyer,Pwd,{Item,Cash}}).
buyer_identify(TP,Pwd) ->
    gen_fsm:sync_send_all_state_event(TP,{identify,buyer,Pwd}).
buyer_insert(TP,Amount,Pwd) ->
    gen_fsm:sync_send_event(TP,{insert,cash,Amount,Pwd}).
buyer_withdraw(TP,Pwd) ->
    gen_fsm:sync_send_event(TP,{withdraw,cash,Pwd}).
get_contents(TP,Pwd) ->
    gen_fsm:sync_send_all_state_event(TP,{get_contents,Pwd}).
% ------------------------------------------------------------------------------
% Internal States
pending({insert,item,Item,Pwd},_,LoopD) when Pwd == LoopD#state.seller,
                                             LoopD#state.seller =/= undefined ->
    {reply,ok,item_received,LoopD#state{item=Item}};
pending({insert,cash,Amount,Pwd},_,LoopD) when Pwd == LoopD#state.buyer;
                                             LoopD#state.buyer =/= undefined ->
    {reply,ok,cash_received,LoopD#state{cash=Amount}};
pending(_E,_From,LoopD) -> {reply,error,pending,LoopD}.

item_received({withdraw,What,Pwd},_,LoopD) ->
    withdraw(What,Pwd,LoopD,item_received);
item_received({insert,cash,Amount,Pwd},_,LoopD = #state{buyer=Pwd})->
    {reply,ok,cash_received,LoopD#state{cash=Amount}};
item_received({insert,item,Item,Pwd},_,LoopD = #state{seller=Pwd})->
    {reply,ok,item_received,LoopD#state{item=Item}};
item_received(_E,_From,LoopD) -> {reply,error,item_received,LoopD}.

cash_received({withdraw,What,Pwd},_,LoopD) ->
    withdraw(What,Pwd,LoopD,cash_received);
cash_received({insert,cash,Amount,Pwd},_,LoopD = #state{buyer=Pwd})->
    {reply,ok,cash_received,LoopD#state{cash=Amount}};
cash_received({insert,item,Item,Pwd},_Frm, LoopD = #state{seller=Pwd})->
    {reply,ok,item_received,LoopD#state{item=Item}};
cash_received(_E,_From,LoopD) -> {reply,error,cash_received,LoopD}.
% ------------------------------------------------------------------------------
% Callbacks with Logic
handle_sync_event({identify,Who,Pwd},_From,StateName,LoopD) ->
    {Reply,NextState} =
        case {Who , LoopD#state.seller, LoopD#state.buyer } of
            {seller, undefined, _ } -> {ok,LoopD#state{seller=Pwd}};
            {buyer, _ , undefined } -> {ok,LoopD#state{buyer=Pwd}};
            _ -> {error,LoopD}
        end,
    {reply,Reply,StateName,NextState};
handle_sync_event(stop,_From,_,LoopData) -> {stop,normal,ok,LoopData};
handle_sync_event({get_contents,Pwd},_Frm,StateName,LoopData) ->
    Reply =
        case {LoopData#state.seller,LoopData#state.buyer} of
            {Pwd,_} -> {LoopData#state.item,LoopData#state.cash};
            {_,Pwd} -> {LoopData#state.item,LoopData#state.cash}
        end,
    {reply,Reply,StateName,LoopData};
handle_sync_event({accept,seller,Pwd,{Item,Cash}},From,StateName,LoopD)
  when Pwd == LoopD#state.seller; LoopD#state.seller =/= undefined ->
    accept(LoopD#state.buyer_accept, {Item,Cash,From}, LoopD#state.item,
           LoopD#state.cash, seller, StateName, LoopD);
handle_sync_event({accept,buyer,Pwd,{Item,Cash}},From,StateName,LoopD)
  when Pwd == LoopD#state.buyer; LoopD#state.buyer =/= undefined ->
    accept(LoopD#state.seller_accept, {Item,Cash,From}, LoopD#state.item,
           LoopD#state.cash, buyer, StateName, LoopD);
handle_sync_event(_E,_From,StateName,LoopData) ->
    {reply,error,StateName,LoopData}.
% ------------------------------------------------------------------------------
% Obligatory others
init([]) -> {ok, pending, #state{}}.
handle_event(_Event, StateName, State) ->{next_state, StateName, State}.
handle_info(_Info, StateName, State) -> {next_state, StateName, State}.
terminate(_Reason, _StateName, _State) -> ok.
code_change(_OldVsn, StateName, State, _Extra) -> {ok, StateName, State}.

% ------------------------------------------------------------------------------
% Withdraw cash xor item - grossly compressed  :(
-spec withdraw(item | cash, any(), #state{}, atom()) ->
    {reply,ok|error,atom(),#state{}}.
withdraw(What,Pwd,LoopD=#state{seller=S,buyer=B,item = I,cash=C},State) ->
  {Reply,NextState,NewLoopData} =
   case { State, What, S, B, I, C} of
    {item_received, item, Pwd, _ , _, undefined } -> {ok, pending, undef(item,LoopD)};
    {item_received, item, Pwd, _ , _, _V } -> {ok, cash_received, undef(item,LoopD)};
    {item_received, cash, _ , Pwd, _, _} -> {ok, item_received, undef(cash,LoopD)};
    {cash_received, item, Pwd, _, _, undefined} -> {ok, cash_received, undef(item,LoopD)};
    {cash_received, cash, _, Pwd, undefined, _ } -> {ok, pending, undef(cash,LoopD)};
    {cash_received, cash, _ , Pwd, _V, _} -> {ok, item_received, undef(cash,LoopD)};
     _ -> {error,State,LoopD}
  end,
    {reply,Reply,NextState,NextLoopData}.
% ------------------------------------------------------------------------------
% Accept a deal
-spec accept({atom(), integer(), {reference(),pid()}},
             {atom(), integer(), {reference(),pid()}},
             atom(), integer(), seller | buyer, atom(),#state{}) ->
    {reply, atom() | integer | {error,not_same},  atom(), #state{}} |
    {next_state, atom(), #state{}}.
accept(AgreeOther, {ItemSelf,CashSelf,Self}, TradePostItem, TradePostCash,
       Who,StateName,LoopD) ->
    % Check if Other and Self have now agreed on the same thing
    AgreeThis  = {ItemSelf, CashSelf, Self},
    Contains   = {TradePostItem, TradePostCash},
    case { AgreeOther , AgreeThis,  Contains} of
        % Same Object And Cash, send cash to buyer And item to seller
        { {Item, Cash, Other}, {Item, Cash, Self}, {Item, Cash} } ->
            {SelfEnd,OtherEnd} =
                case Who of
                    seller -> {Cash,Item};
                    buyer -> {Item,Cash}
                end,
            gen_fsm:reply(Other,OtherEnd),
            {reply,SelfEnd,StateName,LoopD};
        % Not agreed on same
        { {_A,_B,Other}, {_C,_D,Self}, _ } ->
            gen_fsm:reply(Other,{error,not_same}),
            {reply,{error,not_same},StateName,LoopD};
        % Only self has agreed so far
        { undefined, _ , _ } ->
            NewLoopD =
            case Who of
                seller -> LoopD#state{seller_accept={ItemSelf,CashSelf,Self}};
                buyer -> LoopD#state{buyer_accept={ItemSelf,CashSelf,Self}}
            end,
            {next_state,StateName,NewLoopD}
    end.

-spec undef(item | cash, #state{}) -> #state{}.
undef(item,LoopData) -> LoopData#state{item=undefined};
undef(cash,LoopData) -> LoopData#state{cash=undefined}.

And the modified test module tradepost_tests.erl

% @author Gianfranco <zenon@zen.home>
% @copyright (C) 2010, Gianfranco
% Created :  6 Sep 2010 by Gianfranco <zenon@zen.home>
-module(tradepost_tests).
-include_lib("eunit/include/eunit.hrl").
-include("include/eunit_fsm.hrl").

% This is the main point of "entry" for my EUnit testing.
% A generator which forces setup and cleanup for each test in the testset
main_test_() ->
    {foreach,
     fun setup/0,
     fun cleanup/1,
     % Note that this must be a List of TestSet or Instantiator
     [
      % First Iteration
      fun started_properly/1,
      % Second Iteration
      fun identify_seller/1,
      fun insert_item/1,
      fun withdraw_item/1,
      % Fourth iteration
      fun identify_buyer/1,
      fun insert_cash/1,
      fun withdraw_cash/1,
      % Fifth iteration
      fun interleaving1/1,
      fun interleaving2/1,
      fun interleaving3/1,
      fun interleaving4/1,
      fun interleaving5/1,
      fun interleaving6/1,
      % Sixth Iteration
      fun closedeal/1
     ]}.

% Setup and Cleanup
setup()      -> {ok,Pid} = tradepost:start_link(), Pid.
cleanup(Pid) -> tradepost:stop(Pid).

% Pure tests below
% ------------------------------------------------------------------------------
% Let's start simple, I want it to start and check that it is okay.
% I will use the introspective function for this
started_properly(Pid) ->
    ?fsm_test(Pid,"Started Properly Test",
              [{state,is,pending},
               {loopdata,is,[undefined,undefined,undefined,undefined,
                            undefined,undefined]}
              ]).

% Now, we are adding the Seller API tests
identify_seller(Pid) ->
    ?fsm_test(Pid,"Identify Seller Test",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,seller_password],ok},
               {state,is,pending},
               {loopdata,is,[undefined,undefined,seller_password,undefined,
                             undefined,undefined]}
              ]).

insert_item(Pid) ->
    ?fsm_test(Pid,"Insert Item Test",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,seller_password],ok},
               {call,tradepost,seller_insert,[Pid,playstation,seller_password],ok},
               {state,is,item_received},
               {loopdata,is,[playstation,undefined,seller_password,undefined,
                             undefined,undefined]}
              ]).

withdraw_item(Pid) ->
    ?fsm_test(Pid,"Withdraw Item Test",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,seller_password],ok},
               {call,tradepost,seller_insert,[Pid,button,seller_password],ok},
               {state,is,item_received},
               {call,tradepost,seller_withdraw,[Pid,seller_password],ok},
               {state,is,pending},
               {loopdata,is,[undefined,undefined,seller_password,undefined,
                             undefined,undefined]}
              ]).

identify_buyer(Pid) ->
    ?fsm_test(Pid,"Identify Buyer Test",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
               {state,is,pending},
               {loopdata,is,[undefined,undefined,undefined,buyer_password,
                            undefined,undefined]}
              ]).

insert_cash(Pid) ->
    ?fsm_test(Pid,"Insert Cash Test",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
               {call,tradepost,buyer_insert,[Pid,100,buyer_password],ok},
               {state,is,cash_received},
               {loopdata,is,[undefined,100,undefined,buyer_password,
                            undefined,undefined]}
              ]).

withdraw_cash(Pid) ->
    ?fsm_test(Pid,"Withdraw Cash Test",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
               {call,tradepost,buyer_insert,[Pid,100,buyer_password],ok},
               {call,tradepost,buyer_withdraw,[Pid,buyer_password],ok},
               {loopdata,is,[undefined,undefined,undefined,buyer_password,
                            undefined,undefined]}
              ]).

interleaving1(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Inserts Item, then Buyer Identiefies",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insert,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {loopdata,is,[ring,undefined,s,b,undefined,undefined]}
              ]).

interleaving2(Pid) ->
    ?fsm_test(Pid,"Buyer Identifies, Inserts Item, then Seller Identiefies",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,buyer_insert,[Pid,100,b],ok},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {loopdata,is,[undefined,100,s,b,undefined,undefined]}
              ]).

interleaving3(Pid) ->
    ?fsm_test(Pid,"Buyer Identifies, Seller Identifies, buyer inserts cash",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_insert,[Pid,100,b],ok},
               {loopdata,is,[undefined,100,s,b,undefined,undefined]}
              ]).

interleaving4(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Buyer Identifies, Seller inserts item",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_insert,[Pid,ring,s],ok},
               {loopdata,is,[ring,undefined,s,b,undefined,undefined]}
              ]).

interleaving5(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Seller inserts item, Buyer Identifies,"
              "Buyer Inserts Cash",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insert,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,buyer_insert,[Pid,100,b],ok},
               {loopdata,is,[ring,100,s,b,undefined,undefined]}
              ]).

interleaving6(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Seller inserts item, Buyer Identifies,"
              "Seller Withdraws Item, Buyer Inserts Cash, Seller Inserts Item"
              "Buyer Withdraws Cash, Seller Withdraws Item",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insert,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_withdraw,[Pid,s],ok},
               {call,tradepost,buyer_insert,[Pid,100,b],ok},
               {call,tradepost,seller_insert,[Pid,ring,s],ok},
               {call,tradepost,buyer_withdraw,[Pid,b],ok},
               {call,tradepost,seller_withdraw,[Pid,s],ok},
               {loopdata,is,[undefined,undefined,s,b,undefined,undefined]}
              ]).

closedeal(Pid) ->
    ?fsm_test(Pid,
              "Seller Identifies, Buyer Identifies, Seller Inserts Item "
              ", Buyer Inserts Cash, Seller Gets Contents, Buyer Gets Contents "
              ", Buyer Agrees, Seller Agrees ",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_insert,[Pid,ring,s],ok},
               {call,tradepost,buyer_insert,[Pid,100,b],ok},
               {call,tradepost,get_contents,[Pid,s],{ring,100}},
               {call,tradepost,get_contents,[Pid,b],{ring,100}},
               {parallel,call,tradepost,seller_deal,[Pid,s,{ring,100}],100},
               {parallel,call,tradepost,buyer_deal,[Pid,b,{ring,100}],ring}
              ]).

With this success, it’s time to hunt the real cornercase-bugs using Automated Test Generation, a very strong piece of machinery, that requieres some setup. Oh, and just in case, if you forgot how it all fits together…

zen:EUnitFSM zenon$ tree .
.
├── ebin
│   ├── eunit_fsm.beam
│   ├── tradepost.beam
│   └── tradepost_tests.beam
├── include
│   └── eunit_fsm.hrl
├── src
│   └── tradepost.erl
└── test
    ├── eunit_fsm.erl
    └── tradepost_tests.erl

4 directories, 7 files
zen:EUnitFSM zenon$ 

Automated Test Generation – inception

As we wish to generate complex (almost the same as long) test sequences, and do not wish to enter them by hand, we want to have a piece of machinery which makes this for us. An immediate question that comes to mind is then, how does the machinery know which sequences are actually valid? Well, that is for us to know and express. Here we can draw inspiration from from Hoare Logic and use that notion for our automated test generator. A good candidate for this would be a syntax similar to something as

Rule = { PreCondPred, Program, PostCondPred }

PreCondPred = fun FSM-State x SetupResult -> Boolean

Program = [Action]

Action = {call,M,F,A,ExpectedResult} |
        {parallel, call, M, F, A, ResultExpectedResult}

PostCondPred = fun FSM-State x SetupResult -> Boolean

We would thus want the machinery to behave as follows. Given a current gen_fsm state, and the list of all rules, return a list of all rules for which the PreCondPred holds. From this list of rules, choose one randomly, and apply execute Program. After execution, check if PostCondPred holds true. If not, record failure. On Success, repeat process with random selection of valid rule and application, etc.

For this to work, we need to write our own generator, for this, we can peek at the EUnit User’s Guide section on Lazy Generators. However, this should be discussed in the next part of this series. For this purpose, the iterations have ended.

Ps: I apologize for the long delay before this post came, but I had a lot to do at the sidelines.

EUnit Testing gen_fsm – Part 2


Last post we saw the symbolic (somewhat DSL) for the gen_fsm testing, using that as help we shall continue with the Buyer API. Technically, we are now doing the fourth iteration.

Fourth Iteration – Buyer API

The buyer wishes to be identified like the seller, to deposit cash and to withdraw cash. Thus his/her usage is similar. By good TDD, we will write the tests first, adding them to tradepost_tests.erl

The new added instantiators

% This is the main point of "entry" for my EUnit testing.
% A generator which forces setup and cleanup for each test in the testset
main_test_() ->
    {foreach,
     fun setup/0,
     fun cleanup/1,
     % Note that this must be a List of TestSet or Instantiator
     [
      % First Iteration
      fun started_properly/1,
      % Second Iteration
      fun identify_seller/1,
      fun insert_item/1,
      fun withdraw_item/1,
      % Fourth iteration
      fun identify_buyer/1,
      fun insert_cash/1,
      fun withdraw_cash/1
     ]}.

And their implementations

identify_buyer(Pid) ->
    ?fsm_test(Pid,"Identify Buyer Test",
       [{state,is,pending},
        {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
        {state,is,pending},
        {loopdata,is,[undefined,undefined,undefined,buyer_password,undefined]}
      ]).

insert_cash(Pid) ->
    ?fsm_test(Pid,"Insert Cash Test",
       [{state,is,pending},
        {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
        {call,tradepost,buyer_insertcash,[Pid,100,buyer_password],ok},
        {state,is,cash_received},
        {loopdata,is,[undefined,100,undefined,buyer_password,undefined]}
       ]).

withdraw_cash(Pid) ->
    ?fsm_test(Pid,"Withdraw Cash Test",
       [{state,is,pending},
        {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
        {call,tradepost,buyer_insertcash,[Pid,100,buyer_password],ok},
        {call,tradepost,buyer_withdrawcash,[Pid,buyer_password],ok},
        {loopdata,is,[undefined,undefined,undefined,buyer_password,undefined]}
       ]).

The changes in tradepost.erl

%% API
-export([start_link/0,stop/1,seller_identify/2,seller_insertitem/3,
         seller_withdraw_item/2,buyer_identify/2,buyer_insertcash/3,
         buyer_withdrawcash/2]).
%% States
-export([pending/2,pending/3,item_received/3,cash_received/3]).

buyer_identify(TradePost,Password) ->
    gen_fsm:sync_send_event(TradePost,{identify_buyer,Password}).
buyer_insertcash(TradePost,Amount,Password) ->
    gen_fsm:sync_send_event(TradePost,{insert_cash,Amount,Password}).
buyer_withdrawcash(TradePost,Password) ->
    gen_fsm:sync_send_event(TradePost,{withdraw_cash,Password}).    

pending({identify_buyer,Password},_Frm,LoopD = #state{buyer=Password}) ->
    {reply,ok,pending,LoopD};
pending({identify_buyer,Password},_Frm,LoopD = #state{buyer=undefined}) ->
    {reply,ok,pending,LoopD#state{buyer=Password}};
pending({identify_buyer,_},_,LoopD) ->
    {reply,error,pending,LoopD};

pending({insert_cash,Amount,Password},_Frm,LoopD = #state{buyer=Password}) ->
    {reply,ok,cash_received,LoopD#state{cash=Amount}};
pending({insert_cash,_,_},_Frm,LoopD) ->
    {reply,error,pending,LoopD}.

cash_received({withdraw_cash,Password},_From,LoopD = #state{buyer=Password}) ->
    {reply,ok,pending,LoopD#state{cash=undefined}};
cash_received({withdraw_cash,_},_From,LoopD) ->
    {reply,error,cas_received,LoopD}.

Compiling and running

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/ -eval 'eunit:test(tradepost,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly (Started Properly Test)...ok
    tradepost_tests: identify_seller (Identify Seller Test)...ok
    tradepost_tests: insert_item (Insert Item Test)...[0.001 s] ok
    tradepost_tests: withdraw_item (Withdraw Item Test)...[0.001 s] ok
    tradepost_tests: identify_buyer (Identify Buyer Test)...ok
    tradepost_tests: insert_cash (Insert Cash Test)...[0.001 s] ok
    tradepost_tests: withdraw_cash (Withdraw Cash Test)...ok
    [done in 0.024 s]
  [done in 0.024 s]
=======================================================
  All 7 tests passed.

1>

Wow. Great. So the buyer and the seller can now deposit and retract their respective parts. Awesome. However, there are some intentionally left out parts (and yes, I assume more than one of you has been thinking and cringing about it) – the interleaving of the actions. That is left for the fifth iteration.

Fifth Iteration – Interleaving of Actions

Much straight forward  - what if the buyer identifies and inserts the item, and the buyer wishes to identify after this?

Why, let’s write a test scenario for it (and of course it will fail).

interleaving1(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Inserts Item, then Buyer Identiefies",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,a],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,a],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok}
              ]).

And just as we knew, it would blow

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/ -eval 'eunit:test(tradepost,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly (Started Properly Test)...[0.001 s] ok
    tradepost_tests: identify_seller (Identify Seller Test)...ok
    tradepost_tests: insert_item (Insert Item Test)...ok
    tradepost_tests: withdraw_item (Withdraw Item Test)...ok
    tradepost_tests: identify_buyer (Identify Buyer Test)...ok
    tradepost_tests: insert_cash (Insert Cash Test)...ok
    tradepost_tests: withdraw_cash (Withdraw Cash Test)...ok
    tradepost_tests: interleaving1 (Seller Identifies, Inserts Item, then Buyer Identiefies)...*skipped*
undefined
*unexpected termination of test process*
::{function_clause,[{tradepost,item_received,
                               [{identify_buyer,b},
                                {<0.69.0>,#Ref<0.0.0.113>},
                                {state,ring,undefined,a,undefined,...}]},
                    {gen_fsm,handle_msg,7},
                    {proc_lib,init_p_do_apply,3}]}

The issue is of course that it is not possible to identify oneself in any other state than the pending one. Do we consider this a flaw or as part of the system design?  For this example, we shall regard it to be a flaw. And the true design should be that either buyer or seller must be able to identify themselves once before inserting their part and closing the deal.

Let’s add some more tests that we know should pass, like reversing the roles, and adding more interleaving of actions. As will be seen, this causes a lot of code, and we discover the need for some Automated Test Generation.

interleaving1(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Inserts Item, then Buyer Identiefies",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {loopdata,is,[ring,undefined,s,b]}
              ]).

interleaving2(Pid) ->
    ?fsm_test(Pid,"Buyer Identifies, Inserts Item, then Seller Identiefies",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {loopdata,is,[undefined,100,s,b]}
              ]).

interleaving3(Pid) ->
    ?fsm_test(Pid,"Buyer Identifies, Seller Identifies, buyer inserts cash",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {loopdata,is,[undefined,100,s,b]}
              ]).

interleaving4(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Buyer Identifies, Seller inserts item",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {loopdata,is,[ring,undefined,s,b]}
              ]).

interleaving5(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Seller inserts item, Buyer Identifies,"
              "Buyer Inserts Cash",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {loopdata,is,[ring,100,s,b]}
              ]).

These are not all the tests (some 6 more are hidden). For the future (for some future iteration), we would like to specify which state transitions are legal, which functions cause these transitions, and ultimately let the machine generate them for us, run the sequences and test whether all is good.

Also, while failing, it would be tremenduously nice if the automatic test generation would show us a trace of the failing run.

Fixing up the problem with identification, a lot of the tests run through, however a new problem is evident.

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/ -eval 'eunit:test(tradepost,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4]
[async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly (Started Properly Test)...ok
    tradepost_tests: identify_seller (Identify Seller Test)...ok
    tradepost_tests: insert_item (Insert Item Test)...ok
    tradepost_tests: withdraw_item (Withdraw Item Test)...ok
    tradepost_tests: identify_buyer (Identify Buyer Test)...ok
    tradepost_tests: insert_cash (Insert Cash Test)...ok
    tradepost_tests: withdraw_cash (Withdraw Cash Test)...ok
    tradepost_tests: interleaving1 (Seller Identifies, Inserts Item, then Buyer Identiefies)...ok
    tradepost_tests: interleaving2 (Buyer Identifies, Inserts Item, then Seller Identiefies)...ok
    tradepost_tests: interleaving3 (Buyer Identifies, Seller Identifies, buyer inserts cash)...ok
    tradepost_tests: interleaving4 (Seller Identifies, Buyer Identifies, Seller inserts item)...ok
    tradepost_tests: interleaving5 (Seller Identifies, Seller inserts item, Buyer Identifies,Buyer Inserts Cash)...*skipped*
undefined
*unexpected termination of test process*
::{function_clause,[{tradepost,item_received,
                               [{insert_cash,100,a},
                                {<0.85.0>,#Ref<0.0.0.153>},
                                {state,ring,undefined,a,a,...}]},
                    {gen_fsm,handle_msg,7},
                    {proc_lib,init_p_do_apply,3}]}

=ERROR REPORT==== 5-Sep-2010::22:33:23 ===
** State machine <0.83.0> terminating
** Last message in was {'$gen_sync_event',
                           {<0.85.0>,#Ref<0.0.0.153>},
                           {insert_cash,100,a}}
** When State == item_received
**      Data  == {state,ring,undefined,a,a,undefined}
** Reason for termination =
** {function_clause,[{tradepost,item_received,
                                [{insert_cash,100,a},
                                 {<0.85.0>,#Ref<0.0.0.153>},
                                 {state,ring,undefined,a,a,undefined}]},
                     {gen_fsm,handle_msg,7},
                     {proc_lib,init_p_do_apply,3}]}
=======================================================
  Failed: 0.  Skipped: 0.  Passed: 11.
One or more tests were cancelled.

1>

It is not possible to insert the cash after the item has been inserted!(?) Clearly, there is an interleaving problem between item insertion and cash insertion. The true design  should be that either buyer or seller must be able to insert xor withdraw their item / cash before closing the deal, irrespective of the other parts item / cash.

An interesting thing to note is that we got this failing tests because this test was longer. It triggered more transitions, and was in a sense, more complex. This is another thing we wish to get for free from an automated test generating engine.

Wishing to test what we just discussed, we add a longer test that should serve as a green light once it goes through.

interleaving6(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Seller inserts item, Buyer Identifies,"
              "Seller Withdraws Item, Buyer Inserts Cash, Seller Inserts Item"
              "Buyer Withdraws Cash, Seller Withdraws Item"
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_withdraw_item,[Pid,s],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {call,tradepost,seller_insertitem[Pid,ring,s],ok},
               {call,tradepost,buyer_withdrawcash,[Pid,b],ok},
               {call,tradepost,seller_withdraw_item,[Pid,s],ok},
               {loopdata,is,[undefined,undefined,s,b]}
              ]).

A little later …

zen:EUnitFSM zenon$ erlc -o ebin/ src/*.erl test/*.erl
zen:EUnitFSM zenon$ erl -pa ebin/ -eval 'eunit:test(tradepost,[verbose]).'
Erlang R13B04 (erts-5.7.5) [source] [64-bit] [smp:4:4] [rq:4] [async-threads:0] [hipe] [kernel-poll:false]

Eshell V5.7.5  (abort with ^G)
1> ======================== EUnit ========================
module 'tradepost'
  module 'tradepost_tests'
    tradepost_tests: started_properly (Started Properly Test)...ok
    tradepost_tests: identify_seller (Identify Seller Test)...ok
    tradepost_tests: insert_item (Insert Item Test)...[0.001 s] ok
    tradepost_tests: withdraw_item (Withdraw Item Test)...[0.001 s] ok
    tradepost_tests: identify_buyer (Identify Buyer Test)...[0.001 s] ok
    tradepost_tests: insert_cash (Insert Cash Test)...[0.001 s] ok
    tradepost_tests: withdraw_cash (Withdraw Cash Test)...[0.001 s] ok
    tradepost_tests: interleaving1 (Seller Identifies, Inserts Item, then Buyer Identiefies)...ok
    tradepost_tests: interleaving2 (Buyer Identifies, Inserts Item, then Seller Identiefies)...ok
    tradepost_tests: interleaving3 (Buyer Identifies, Seller Identifies, buyer inserts cash)...[0.001 s] ok
    tradepost_tests: interleaving4 (Seller Identifies, Buyer Identifies, Seller inserts item)...ok
    tradepost_tests: interleaving5 (Seller Identifies, Seller inserts item, Buyer Identifies,Buyer Inserts Cash)...[0.001 s] ok
    tradepost_tests: interleaving6 (Seller Identifies, Seller inserts item, Buyer Identifies,Seller Withdraws Item, Buyer Inserts Cash, Seller Inserts ItemBuyer Withdraws Cash, Seller Withdraws Item)...[0.001 s] ok
    [done in 0.041 s]
  [done in 0.041 s]
=======================================================
  All 13 tests passed.

1>

That done, the code of tradepost.erl is now

%%%-------------------------------------------------------------------
%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created :  2 Sep 2010 by Gianfranco <zenon@zen.home>
%%%-------------------------------------------------------------------
-module(tradepost).
-behaviour(gen_fsm).

%% API
-export([start_link/0,stop/1,seller_identify/2,seller_insertitem/3,
         seller_withdraw_item/2,buyer_identify/2,buyer_insertcash/3,
         buyer_withdrawcash/2]).

%% States
-export([pending/3,item_received/3,cash_received/3]).

%% gen_fsm callbacks
-export([init/1, handle_event/3, handle_sync_event/4, handle_info/3,
         terminate/3, code_change/4]).
-record(state, {object,cash,seller,buyer}).

%%%===================================================================
%%% API
start_link() -> gen_fsm:start_link(?MODULE, [], []).

stop(Pid) -> gen_fsm:sync_send_all_state_event(Pid,stop).

seller_identify(TradePost,Password) ->
    gen_fsm:sync_send_all_state_event(TradePost,{identify_seller,Password}).
seller_insertitem(TradePost,Item,Password) ->
    gen_fsm:sync_send_event(TradePost,{insert,Item,Password}).
seller_withdraw_item(TradePost,Password) ->
    gen_fsm:sync_send_event(TradePost,{withdraw,Password}).

buyer_identify(TradePost,Password) ->
    gen_fsm:sync_send_all_state_event(TradePost,{identify_buyer,Password}).
buyer_insertcash(TradePost,Amount,Password) ->
    gen_fsm:sync_send_event(TradePost,{insert_cash,Amount,Password}).
buyer_withdrawcash(TradePost,Password) ->
    gen_fsm:sync_send_event(TradePost,{withdraw_cash,Password}).    

%%--------------------------------------------------------------------
pending({insert,Item,Password},_Frm,LoopD = #state{seller=Password}) ->
    {reply,ok,item_received,LoopD#state{object=Item}};
pending({insert_cash,Amount,Password},_Frm,LoopD = #state{buyer=Password}) ->
    {reply,ok,cash_received,LoopD#state{cash=Amount}};
pending(_E,_From,LoopD) ->
    {reply,error,pending,LoopD}.

item_received({withdraw,Password},_Frm,LoopD = #state{seller=Password}) ->
    NewState = case LoopD#state.cash of
                   undefined -> pending;
                   _ -> cash_received
               end,
    {reply,ok,NewState,LoopD#state{object=undefined}};
item_received({withdraw_cash,Password},_Frm,LoopD = #state{buyer=Password}) ->
    NewState = case LoopD#state.object of
                   undefined -> pending;
                   _ -> item_received
               end,
    {reply,ok,NewState,LoopD#state{cash=undefined}};
item_received({insert_cash,Amount,Password},_Frm,LoopD = #state{buyer=Password})->
    {reply,ok,cash_received,LoopD#state{cash=Amount}};
item_received({insert,Item,Password},_Frm,LoopD = #state{seller=Password})->
    {reply,ok,item_received,LoopD#state{object=Item}};
item_received(_E,_From,LoopD) ->
    {reply,error,item_received,LoopD}.

cash_received({withdraw_cash,Password},_From,LoopD = #state{buyer=Password}) ->
    NewState = case LoopD#state.object of
                   undefined -> pending;
                   _ -> item_received
               end,
    {reply,ok,NewState,LoopD#state{cash=undefined}};
cash_received({withdraw,Password},_Frm,LoopD = #state{seller=Password}) ->
    NewState = case LoopD#state.cash of
                   undefined -> pending;
                   _ -> cash_received
               end,
    {reply,ok,NewState,LoopD#state{object=undefined}};
cash_received({insert_cash,Amount,Password},_Frm,LoopD = #state{buyer=Password})->
    {reply,ok,cash_received,LoopD#state{cash=Amount}};
cash_received({insert,Item,Password},_Frm,LoopD = #state{seller=Password})->
    {reply,ok,item_received,LoopD#state{object=Item}};
cash_received(_E,_From,LoopD) ->
    {reply,error,cash_received,LoopD}.

%%--------------------------------------------------------------------
handle_sync_event({identify_seller,Pass},_From,StateName,
                  LoopData=#state{seller=undefined}) ->
    {reply,ok,StateName,LoopData#state{seller=Pass}};
handle_sync_event({identify_buyer,Pass},_From,StateName,
                  LoopData=#state{buyer=undefined}) ->
    {reply,ok,StateName,LoopData#state{buyer=Pass}};
handle_sync_event(stop,_From,_StateName,LoopData) ->
    {stop,normal,ok,LoopData};
handle_sync_event(_E,_From,StateName,LoopData) ->
    {reply,error,StateName,LoopData}.

%%--------------------------------------------------------------------
init([]) -> {ok, pending, #state{}}.
handle_event(_Event, StateName, State) ->{next_state, StateName, State}.
handle_info(_Info, StateName, State) -> {next_state, StateName, State}.
terminate(_Reason, _StateName, _State) -> ok.
code_change(_OldVsn, StateName, State, _Extra) -> {ok, StateName, State}.

And the time has inevitably come for the deal closure API (next up – in the third part), and for your convenience, the full tradepost_tests.erl

%%% @author Gianfranco <zenon@zen.home>
%%% @copyright (C) 2010, Gianfranco
%%% Created :  6 Sep 2010 by Gianfranco <zenon@zen.home>
-module(tradepost_tests).
-include_lib("eunit/include/eunit.hrl").
-include("include/eunit_fsm.hrl").

% This is the main point of "entry" for my EUnit testing.
% A generator which forces setup and cleanup for each test in the testset
main_test_() ->
    {foreach,
     fun setup/0,
     fun cleanup/1,
     % Note that this must be a List of TestSet or Instantiator
     [
      % First Iteration
      fun started_properly/1,
      % Second Iteration
      fun identify_seller/1,
      fun insert_item/1,
      fun withdraw_item/1,
      % Fourth iteration
      fun identify_buyer/1,
      fun insert_cash/1,
      fun withdraw_cash/1,
      % Fifth iteration
      fun interleaving1/1,
      fun interleaving2/1,
      fun interleaving3/1,
      fun interleaving4/1,
      fun interleaving5/1,
      fun interleaving6/1
     ]}.

% Setup and Cleanup
setup()      -> {ok,Pid} = tradepost:start_link(), Pid.
cleanup(Pid) -> tradepost:stop(Pid).

% Pure tests below
% ------------------------------------------------------------------------------
% Let's start simple, I want it to start and check that it is okay.
% I will use the introspective function for this
started_properly(Pid) ->
    ?fsm_test(Pid,"Started Properly Test",
              [{state,is,pending},
               {loopdata,is,[undefined,undefined,undefined,undefined]}
              ]).

% Now, we are adding the Seller API tests
identify_seller(Pid) ->
    ?fsm_test(Pid,"Identify Seller Test",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,seller_password],ok},
               {state,is,pending},
               {loopdata,is,[undefined,undefined,seller_password,undefined]}
              ]).

insert_item(Pid) ->
    ?fsm_test(Pid,"Insert Item Test",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,seller_password],ok},
               {call,tradepost,seller_insertitem,[Pid,playstation,seller_password],ok},
               {state,is,item_received},
               {loopdata,is,[playstation,undefined,seller_password,undefined]}
              ]).

withdraw_item(Pid) ->
    ?fsm_test(Pid,"Withdraw Item Test",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,seller_password],ok},
               {call,tradepost,seller_insertitem,[Pid,button,seller_password],ok},
               {state,is,item_received},
               {call,tradepost,seller_withdraw_item,[Pid,seller_password],ok},
               {state,is,pending},
               {loopdata,is,[undefined,undefined,seller_password,undefined]}
              ]).

identify_buyer(Pid) ->
    ?fsm_test(Pid,"Identify Buyer Test",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
               {state,is,pending},
               {loopdata,is,[undefined,undefined,undefined,buyer_password]}
              ]).

insert_cash(Pid) ->
    ?fsm_test(Pid,"Insert Cash Test",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,buyer_password],ok},
               {state,is,cash_received},
               {loopdata,is,[undefined,100,undefined,buyer_password]}
              ]).

withdraw_cash(Pid) ->
    ?fsm_test(Pid,"Withdraw Cash Test",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,buyer_password],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,buyer_password],ok},
               {call,tradepost,buyer_withdrawcash,[Pid,buyer_password],ok},
               {loopdata,is,[undefined,undefined,undefined,buyer_password]}
              ]).

interleaving1(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Inserts Item, then Buyer Identiefies",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {loopdata,is,[ring,undefined,s,b]}
              ]).

interleaving2(Pid) ->
    ?fsm_test(Pid,"Buyer Identifies, Inserts Item, then Seller Identiefies",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {loopdata,is,[undefined,100,s,b]}
              ]).

interleaving3(Pid) ->
    ?fsm_test(Pid,"Buyer Identifies, Seller Identifies, buyer inserts cash",
              [{state,is,pending},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {loopdata,is,[undefined,100,s,b]}
              ]).

interleaving4(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Buyer Identifies, Seller inserts item",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {loopdata,is,[ring,undefined,s,b]}
              ]).

interleaving5(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Seller inserts item, Buyer Identifies,"
              "Buyer Inserts Cash",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {loopdata,is,[ring,100,s,b]}
              ]).

interleaving6(Pid) ->
    ?fsm_test(Pid,"Seller Identifies, Seller inserts item, Buyer Identifies,"
              "Seller Withdraws Item, Buyer Inserts Cash, Seller Inserts Item"
              "Buyer Withdraws Cash, Seller Withdraws Item",
              [{state,is,pending},
               {call,tradepost,seller_identify,[Pid,s],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_identify,[Pid,b],ok},
               {call,tradepost,seller_withdraw_item,[Pid,s],ok},
               {call,tradepost,buyer_insertcash,[Pid,100,b],ok},
               {call,tradepost,seller_insertitem,[Pid,ring,s],ok},
               {call,tradepost,buyer_withdrawcash,[Pid,b],ok},
               {call,tradepost,seller_withdraw_item,[Pid,s],ok},
               {loopdata,is,[undefined,undefined,s,b]}
              ]).

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: