Experimentation in Software Engineering

Lecturer (Coordinator):
Sira Vegas
Natalia Juristo


Second semester




Software engineering technologies are not being properly assessed, i.e., practitioners do not really know whether or not a technology is effective, and, if they do, they cannot be sure how effective and applicable it is. This lack of proper assessment is seriously undermining industry’s capability to produce competitive and quality software.

Experimental software engineering (ESE) is a branch of software engineering that aims to produce reliable information for practitioners about which technologies should be used in software development projects. ESE uses empirical studies (experiments, quasi-experiments, case studies, etc.) to assess the effectiveness of software technologies.

The objective of this subject is to train students in the basic skills required to apply empirical methods. We focus on experiments as they are the most mature and well-understood type of empirical study in SE. Students will learn how to perform, analyse, aggregate and replicate experiments (in industry and academic environments).

Learning Goals

  • Understand the current limitation of software engineering technologies because they are not researched scientifically
  • Understand the meaning of experimental laboratory and experiment in software engineering
  • Know the different elements of a software engineering experiments: response variable, factors, levels, etc.
  • Design experiments for software engineering technologies: randomization, other control strategies, types of designs, etc.
  • Understand software engineering experiment reports
  • Apply basic statistical data analysis techniques (i.e. t-test, ANOVA, sample size calculation)


  1. Introduction to experimental software engineering
    1. Basics of experimentalism
    2. The scientific method
    3. Scientific Rules: Cause-effect relationships
    4. Scientific immaturity of software engineering
  2. Laboratory and experiment
    1. The concept of laboratory
    2. The concept of experiment
    3. A lab for software engineering
    4. An experiment for software engineering
  3. Elements of an experiment
    1. Response variables
    2. Factors and levels
    3. Types of empirical studies
  4. Designing experiments
    1. Types of variables
    2. Types of control
    3. Validity
  5. Data analysis
    1. Basics of inferential statistics
    2. Parametric tests for independent samples
    3. Parametric tests for related samples
    4. Non-parametric tests

Recommended reading

  • Natalia Juristo; Ana Moreno: "Basics of software engineering experimentation". Kluwer 2001
  • Claes Wohlin et al.: "Experimentation in software engineering: An introduction". Kluwer 2000


  • Basic knowledge of statistics.

Assessment method

  • Students will be assessed on the basis of assignments only. Students shall not have to sit an exam.
  • The assessment of assignments will depend on: (1) a presentation made by students and (2) the correctness of the results.
  • The final grade will be calculated using a weighted average.

Tuition language


Subject-Specific Competences

Code, description and proficiency level for each subject-specific competence
Code Competence Proficiency Level
CEM1 Examination of the state of the art to identify research problems related to the design, construction, use and evaluation of complex software-intensive sociotechnical systems A
CEM3 Application of relevant research methods to open problems in the field of software engineering related to both the particular features of the software and software development management A

Learning Outcomes

Code, description and proficiency level for each subject learning outcome
Code Learning Outcome Associated competences Proficiency level
RA-IS-23 Understand the application of the experimental paradigm in software engineering CEM1, CEM3 A
RA-IS-24 Design software engineering experiments, including experiment replications CEM1, CEM3 A

Learning Guide

Learning Guide: Experimentation in Software Engineering