Experiment #1 - LightPL-ACME compared to PL-AspectualACME

Contents

1. Questions
2. Hypotheses
3. Dependent and independent variables
4. Execution
5. Threats to validity
6. Analysis of the results

1. Questions and Metrics

Q1:	Is LightPL-ACME efficient in terms of time for sufficiently describing the GingaForAll SPL when compared to PL-AspectualACME?
Q2:	What is the degree of complexity for describing the GingaForAll SPL using LightPL-ACME when compared to PL-AspectualACME?
Q3:	What is the degree of expressiveness for describing the GingaForAll SPL using LightPL-ACME when compared to PL-AspectualACME?
Q4:	What is the degree of scalability for describing the GingaForAll SPL using LightPL-ACME when compared to PL-AspectualACME?
Q5:	What is the degree of understanding regarding the GingaForAll SPL architecture description using LightPL-ACME when compared to PL-AspectualACME?

The respective metrics used for each question (Q1 to Q5) are:

M1:	Efficiency in terms of the time spent for describing the GingaForAll SPL.
M2:	Complexity in terms of difficulty provided by the ADL for describing the GingaForAll SPL.
M3:	Expressiveness in terms of the sufficient representation of the elements related to the GingaForAll SPL.
M4:	Scalability in terms of the fact that the ADL enables to describe larger systems without promoting a major impact in the architectural description, i.e. it is possible to establish an acceptable trade-off between increasing the size of the SPL and its respective products and the degree of difficulty for describing the SPL architecture and elements.
M5:	Ease of understanding of the GingaForAll SPL architectural description.

2. Hypotheses
The null and alternative hypotheses (H1 to H5) corresponding to the questions Q1 to Q5 using the metrics M1 to M5 are synthetized as follows.

H1 (Q1-M1):	The time spent for describing the GingaForAll SPL using the LightPL-ACME ADL is (equal / smaller than / greater than) to the time spent for making the description using the PL-AspectualACME ADL.
H2 (Q2-M2):	The degree of complexity for describing the GingaForAll SPL using the LightPL-ACME ADL is (equal / smaller than / greater than) to the degree of complexity for making the description using the PL-AspectualACME ADL.
H3 (Q3-M3):	The degree of expressiveness for describing the GingaForAll SPL using the LightPL-ACME ADL is (equal / smaller than / greater than) to the degree of expressiveness for making the description using the PL-AspectualACME ADL.
H4 (Q4-M4):	The degree of scalability for describing the GingaForAll SPL using the LightPL-ACME ADL is (equal / smaller than / greater than) to the degree of scalability for making the description using the PL-AspectualACME ADL.
H5 (Q5-M5):	The degree of understanding the GingaForAll SPL architecture description in LightPL-ACME is (equal / smaller than / greater than) to the degree of understanding of the description in PL-AspectualACME.

3. Dependent and independent variables
The goal of this controlled experiment is to evaluate the applicability, expressiveness, and scalability of the LightPL-ACME ADL compared to the PL-AspectualACME ADL for describing the GingaForAll SPL. In this experiment, there is an independent variable, which is the strategy for architectural description using ADLs, and it has two related alternatives: (i) to specify the architectural description using the LightPL-ACME ADL, and; (ii) to specify the architectural description using the PL-AspectualACME ADL. In addition, there are two dependent variables to be measured in the experiment and that are related to the GingaForAll SPL architectural description using the ADLs: (i) efficiency; (ii) complexity; (iii) expressiveness; (iv) scalability, and; (v) ease of understanding.


4. Execution
The controlled experiment was performed in three phases. In the first one, the participants were trained in order to level their knowledge. Before beginning the second phase of the experiment, which consisted in describing the GingaForAll SPL using the LightPL-ACME and PL-AspectualACME ADLs, the execution environment was prepared by installing the Eclipse IDE plug-ins regarding the tools used for specifying the architectural descriptions. Despite PL-AspectualACME does not directly have an associated tool, the architectural description using this ADL was made by using the AspectualACME Studio tool, which supports architectural descriptions in AspectualACME, the ADL on which PL-AspectualACME is based. In this experiment, it is very important to highlight that the tools were used only for specifying the architectural descriptions using the ADLs and checking their syntax, and not for assessing issues regarding the usability of these tools.

To start the second phase of the experiment, the participants answered an initial form in order to characterize the profile of each participant and collect personal information about professional expertise and skills in software development and familiarity with SPL and ADLs concepts. In addition, part of the GingaForAll SPL feature model was provided to the members of Group I as well as two products resulted from combinations of some of these features. Next, the members of Group I were randomly grouped in ten pairs and raffles were performed in order to define the order in which the participants would make the GingaForAll SPL architecture description (i.e. firstly with LightPL-ACME or PL-AspectualACME), but still warranting that five pairs would begin with the LightPL-ACME ADL and the others would begin with the PL-AspectualACME ADL. The time spent to make each architectural description using the evaluated ADLs was timed and registered by the experimenters, so that the members of Group I individually answered another form in which they recorded their opinion about each of the ADLs and listed difficulties, facilities, and other relevant observations at the end.

Finally, in the third phase of the experiment, the architectural descriptions specified by the members of Group I were analyzed by the members of Group II, which tried to build the GingaForAll SPL feature model from the provided descriptions and identify the features that would be present in the considered products in order to assess the degree of understanding of these descriptions. These artefacts were compared with the originally provided to the members of Group I and the members of Group II also answered a form in which their impressions about the architectural descriptions using the evaluated ADLs were recorded. All materials used in this experiment are available by following these links:

GingaForAll artifacts provided to the participants

Questionnaires answered by the participants (in Portuguese)

5. Threats to validity
As threats to the validity of the performed experiment, two as-pects can be enumerated. The first one concerns to the methodology that was adopted in this experiment. The number of participants (only twenty-five people) is small and characterizes a non-representative sample from the population, thus making impossible to establish any kind of generalization. This small sample size also avoided the use of statistical methodologies for analysing the results, thus being another threat to validity. In addition, it is necessary to perform the experiment taking into account others SPLs in order to verify if the results would be the same observed for the GingaForAll SPL.

The second aspect of threat to validity is inherent to the participants. The results are limited to their profiles (somewhat homogeneous), which would probably not observed if the experiment would be performed in real-world conditions since it could have participants with different expertise levels. It is also possible that the results may be affected by making a different organization of the participants (e.g. keeping them in individual work rather than putting them in groups, as done in this experiment) or postponing the duration of the experiment, which could discourage and/or physically and/or mentally tire the participants. Finally, another threat to validity is the fact that the experiment was performed only once, so that the results could be somewhat affected whether the roles of the participants change, e.g. if the members of Group I were in Group II and vice-versa.


6. Analysis of the Results

Q1 - Is LightPL-ACME efficient in terms of time for sufficiently describing the GingaForAll SPL when compared to PL-AspectualACME? Result: The time spent for describing the GingaForAll SPL using LightPL-ACME was smaller than the time spent for describing it using PL-AspectualACME. More precisely, the average time for describing the GingaForAll SPL using LightPL-ACME was 28 minutes, whereas the average time spent for PL-AspectualACME was 62 minutes.

Q1 - What is the degree of complexity for describing the GingaForAll SPL using LightPL-ACME when compared to PL-AspectualACME? Result: The participants were unanimous when reporting that PL-AspectualACME is very difficult and verbose for describing the GingaForAll SPL compared to LightPL-ACME.

Q3 - What is the degree of expressiveness for describing the GingaForAll SPL using LightPL-ACME when compared to PL-AspectualACME? Result: The participants unanimously reported that the degree of expressiveness of PL-AspectualACME is very low when compared to LightPL-ACME.

Q4 - What is the degree of scalability for describing the GingaForAll SPL using LightPL-ACME when compared to PL-AspectualACME? Result: The members of Group I unanimously reported that such increase hampers the architectural description in PL-AspectualACME when compared to LightPL-ACME.

Q5 - What is the degree of understanding regarding the GingaForAll SPL architecture description using LightPL-ACME when compared to PL-AspectualACME? Result: The participants stated that the degree of understanding of descriptions in LightPL-ACME is much greater than the descriptions in PL-AspectualACME. Additionally, the members of this group reported that it was considerably easy to build the feature model and identify the GingaForAll SPL products from the descriptions in LightPL-ACME compared to the descriptions in PL-AspectualACME.