EUnit Testing gen_fsm – Part 4 Automated Test Machinery

Using the idea from the last post, we have a machinery which takes a set of Rules as input, a gen_fsm, setup and cleanup. To develop this, there is once again a big question, will this be done with parse transforms or using macros and functions? To try and answer this question along with how it should work, let’s write some mock-code. This will be the first iteration of the ATG (Automated Test Generator).

First Iteration – Brainstorm

For this first iteration, it gets easier if we break it down into some use-cases, first off, the battery of Rules,

% ------------------------------------------------------------------------------
% List of all rules - mock code!
% ------------------------------------------------------------------------------
rules() ->
     % Seller and buyer registration,
     % The postcondition needs to access the password which was generated in
     % the program so that we can verify that the tradepost holds the correct
     % registration.
     { not reg(seller), do_reg(seller), reg(seller,generated_pwd)},
     { not reg(buyer),  do_reg(buyer),  reg(buyer,generated_pwd)},

     % Insert item / cash, should insert random amount of cash and random
     % items. The PostCond must be able to test the existence of that item / cash
     % and thus must have access to the generated item / cash.
     { reg(seller), insert(item), item_is(inserted_item)},
     { reg(buyer),  insert(cash),   cash_is(inserted_amount)},

     % Removal of cash or item, remove needs to use the password which
     % the tradepost currently has.
     { tradepost_has(cash), remove(cash), not tradepost_has(cash) },
     { tradepost_has(item), remove(item), not tradepost_has(item) },

     % Close a deal, we must check that the _returned value_ is the correct one.
     % We must pass on the returned value to the postcond. The Postcond must
     % then also have access to the original item and cash.
     { tradepost_has(cash)
       andalso tradepost_has(item), closedeal(), return_value_is(??)}

As can be seen, I added some comments on thoughts that deserve discussion. It will often be the case that the postcondition needs to refer to a value which is the result of the Program. Such an example can be the test for correct registration. For that case, we do not only need access to the state, but also the generated password which was used. How to fix it? Bind the Program result to a special atom ‘$PROGRAM_RESULT’, also, bind the state to a special atom ‘$STATE’. Problem solved. Please note that such a solution , is leaning towards parse-transforms.

% The State is accessed through the special  atom
% '$STATE', and the module must include the state record.
reg(seller) -> '$STATE'#state.seller =/= undefined;
reg(buyer) ->  '$STATE'#state.buyer =/= undefined.

% Passwd can be result from Program, thus the call in
% the Rule may then be { ..... , reg(seller,'$PROGRAM_RESULT')}
reg(seller,Passwd) -> '$STATE'#state.seller == Passwd;
reg(buyer,Passwd) -> '$STATE'#state.buyer == Passwd.

Another special binding added to this machinery, is the identifier for the gen_fsm, this could be a registered name or a Pid. The usage of this is shown below, where a registration is performed, and the password is returned for the Postcondition.

% The identifier of the gen_fsm is accessed through '$ID'
do_reg(seller) ->
    Password = generate_passwd(),
    Password. % This line ensures that '$PROGRAM_RESULT' is bound to
             % the generated password, for the Postcondition to access.

How the ‘$ID’ is bound, is unclear for now, but that can be taken care of later, remember that we are just writing mock code to get an idea of what we possibly want. Going from the top to bottom, how would we love to see the insertion?

% This of course also uses the $ID, but also the state to access the password.
% seems only fair to return the inserted item for this function as that
% probably is what we wish, ... always.
insert(item) ->
    Password = '$STATE'#state.seller,
    Item = generate_item(),
    Item; % Ensure '$PROGRAM_RESULT is Item
insert(cash) ->
    Password = '$STATE'#state.buyer,
    Cash = generate_cash(),
    Cash; % Ensure '$PROGRAM_RESULT is Cash

Coupled to this, Now, how to test that the cash is the amount we wish, and that the item is the sought one?

% Check that item is the specified item (argument).
item_is(GivenItem) -> '$STATE'#state.item == GivenItem.

% Check that cash is specified amount (argument)
cash_is(GivenAmount) -> '$STATE' == GivenAmount.

Or simpler, just test existence?

% Testing state to contain cash xor item
tradepost_has(cash) -> '$STATE' =/= undefined;
tradepost_has(item) -> '$STATE'#state.item =/= undefined.

Likewise, removal should be easy with the help of ‘$ID’ and ‘$STATE’

% Utilizing both state and id, but nice thing is that result
% of withdraw function will be bound to $PROGRAM_RESULT in this mockup.
remove(cash) ->
    Passwd = '$STATE'#state.buyer,
remove(item) ->
    Passwd = '$STATE'#state.seller,

Nearing the end of the first iteration brainstorming, what remains is a way to program the deal closing and a way to test the return value of this (a way to test the return value of the deal closing), ‘$PROGRAM_RESULT’ is still our friend, but it would be good to have some kind of parallelism. Once again, how would we like to see this written?

% Deal closing, is a parallelized action.
% Return value should still be bound to '$PROGRAM_RESULT'
% Could be bound in the form [ A , B ] for this example.
% Inspiration from the parallel keyword in EUnit.
closedeal() ->
     [ begin
           Pass = '$STATE'#state.buyer,
           ItemAndCash = tradepost:get_contents('$ID',Pass),
           Pass = '$STATE'#state.seller,
           ItemAndCash = tradepost:get_contents('$ID',Pass),

This seems to do the trick,  the idea is to pass a list of  functions which are executed in parallel, the result of each one is bound to an element in the generated list which is bound to ‘$PROGRAM_RESULT’. As this seems that this covers the first iteration of brainstorming, it could be nice to sum it up.

Summary of First Brainstorming session

The Automated Test Generator machinery has a set of Rules, each Rule is modeled as a 3-tuple, The first element in the 3-tuple is the Precondition that has to be met for the Rule to possibly take action. The second element is a Program which is executed, iff the Precondition holds and the Rule is chosen. The third element is a Postcondition that must hold once the Program of the Rule is executed.

The different special syntax elements which have been identified now are

  • ‘$STATE’,  the internal state of the gen_fsm being tested.
  • ‘$ID’,  identifier of the gen_fsm, can be a registered name, a registered name on a node or a Pid.
  • ‘$PROGRAM_RESULT’, the result of the Program (middle element in the Rule)
  • ‘$PARALLEL’, keyword for marking that functions are to be executed in parallel, and the joint result is bound to ‘$PROGRAM_RESULT’

Besides these special syntactical elements, we should try structuring the thoughts a bit more.

rule() = { precondition(), program(), postcondition() }

precondition() = expr() -> true | false.
program()    = expr() -> any().
postcondition()= expr() -> true | false.

Maybe this definition feels a bit weak and lame, but as for now, it captures what we wish, next brainstorming session should focus on the machinery of our automated test generator.


Leave a comment

1 Comment

  1. Testing gen_fsm – Part 5 ATG Engine | Erlang, Testing and TDD

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: