Experimentation in Software Engineering

Lecturer (Coordinator):
Sira Vegas
svegas@fi.upm.es
Lecturer:
Natalia Juristo
natalia@fi.upm.es

Semester

Second semester

Credits

4 ECTS

Outline

Software engineering technologies are not being properly evaluated. That is, professionals do not know for sure whether a technology is effective or not and, if so, cannot be sure how effective and applicable it is. This lack of proper evaluation undermines the ability of the industry to produce competitive quality software.

Experimental software engineering (ESE) is a discipline of Software Engineering that aims to produce reliable information for professionals about what technologies should be used in software development projects. ESE uses empirical studies (experiments, quasi-experiments, case studies, etc.) to evaluate the effectiveness of technologies for software development.

The objective of this subject is to train students in the basic skills required to apply empirical methods. We focus on experiments as they are the most mature and well-understood type of empirical study in SE. Students will learn how to perform, analyse, aggregate and replicate experiments (in industry and academic environments).

Learning Goals

  • Understand the current limitation of software engineering technologies because they are not researched scientifically
  • Understand the meaning of experimental laboratory and experiment in software engineering
  • Know the different elements of a software engineering experiments: response variable, factors, levels, etc.
  • Design experiments for software engineering technologies: randomization, other control strategies, types of designs, etc.
  • Understand software engineering experiment reports
  • Apply basic statistical data analysis techniques (i.e. t-test, ANOVA, sample size calculation)

Syllabus

  1. Introduction to experimental software engineering
    1. Basics of experimentalism
    2. The scientific method
    3. Scientific Rules: Cause-effect relationships
    4. Scientific immaturity of software engineering
  2. Laboratory and experiment
    1. The concept of laboratory
    2. The concept of experiment
    3. A lab for software engineering
    4. An experiment for software engineering
  3. Elements of an experiment
    1. Response variables
    2. Factors and levels
    3. Types of empirical studies
  4. Designing experiments
    1. Types of variables
    2. Types of control
    3. Validity

Recommended reading

Prerequisites

  • Basic knowledge of statistics.

Tuition language

English

Subject-Specific Competences

Code, description and proficiency level for each subject-specific competence
Code Competence Proficiency Level
CEM1 Examination of the state of the art to identify research problems related to the design, construction, use and evaluation of complex software-intensive sociotechnical systems A
CEM3 Application of relevant research methods to open problems in the field of software engineering related to both the particular features of the software and software development management A

Learning Outcomes

Code, description and proficiency level for each subject learning outcome
Code Learning Outcome Associated competences Proficiency level
RA-IS-23 Understand the application of the experimental paradigm in software engineering CEM1, CEM3 A
RA-IS-24 Design software engineering experiments, including experiment replications CEM1, CEM3 A
RA-IS-35 Aggregate Software Engineering experiments CEM3 A

Learning Guide

Learning Guide: Experimentation in Software Engineering