Empirical Software Engineering

(Winter 2022)

This course is heavily inspired by the (2018) 17803: Empiricial Methods at Carnegie Mellon University, (2020) CSC485: Empirical SE: Bridging Research and Practice at University of Victoria, and (2014) CSC2130: Empirical Research Methods for Computer Scientists at University of Toronto. Many thanks to Prof. Bogdan Vasilescu, Prof. Margaret-Anne Storey, and Prof. Steve Easterbrook for generously sharing the course materials and providing valuable advices. Check out the Winter2021 version of this course.


Recent decades have witnessed the emergence of two important trends in software systems. First, an increasing number of software technologies have a significant social component (e.g. end-user programming, collaborative development tools). Second, software systems have increased significantly in complexity and size challenging traditional development and testing approaches. Empirical research methods play a key role in the evaluation of tools and technologies, and in testing the social and technical theories they embody.

This course provides an overview and hands-on experience with a core of qualitative and quantitative empirical research methods, including interviews, qualitative coding, survey design, and large-scale mining and analysis of data. There will be extensive reading with occasional student presentations about the reading in class, weekly homework assignments, and a semester-long research project for which students must prepare in-class kickoff and final presentations as well as a final report.

We will focus on software engineering related research questions in readings and assignments. Students will mine and integrate data from and across online software repositories (e.g., GitHub and Stack Overflow) and employ a spectrum of data analysis techniques, ranging from statistical modeling to social network analysis. For the final research project, we encourage students to come up with a research question of interest to themselves. The delivery will be a research paper, and one or more empirical methods presented in class have to be part of the paper.

Learning Goals

The learning goals describe what I want students to know or be able to do by the end of the semester. I evaluate whether learning goals have been achieved through assignments, written project reports, and in-class presentations. All learning goals are roughly written in a form "after taking this class, the student should be able to ...".

Logistics and People

Lectures: Thursday 10:00-12:00 EST
Office hour: by appointment

Instructor: Shurui Zhou
TA1: Hossein Yousefi
TA2: Shashank Motepalli

Learning in a Global Pandemic

This course offers a place where students come together to discuss and debate ideas, learn about themselves and each other, and build community. Although we cannot come together in person this semester, the course team will work hard to support you and your learning, and do our best to cultivate a sense of community.
Professor Brandon Bayne of religious studies at the University of North Carolina at Chapel Hill produced a new syllabus for the summer 2020 edition of his course, addressing the pandemic directly. We’ve borrowed his ideas, with gratitude to Professor Bayne:
  • Nobody signed up for this.
  • - Not for the sickness, not for the social distancing, not for the sudden end of our collective lives together on campus
    - Not for an online class, not for teaching remotely, not for learning from home, not for mastering new technologies, not for varied access to learning materials
  • The humane option is the best option.
  • - We are going to prioritize supporting each other as humans
    - We are going to prioritize simple solutions that make sense for the most
    - We are going to prioritize sharing resources and communicating clearly
  • We cannot just do the same thing online.
  • - Some assignments are no longer possible
    - Some expectations are no longer reasonable
    - Some objectives are no longer valuable
  • We will foster intellectual nourishment, social connection, and personal accommodation.
  • - Accessible asynchronous content for diverse access, time zones, and contexts
    - Optional synchronous discussion to learn together and combat isolation
  • We will remain flexible and adjust to the situation.
  • - Nobody knows where this is going and what we’ll need to adapt
    - Everybody needs support and understanding in this unprecedented moment

    Important Notes on Information Security Risk and Teaching Remotely from UofT. If you are a citizen of another country, and/or accessing your courses at the University of Toronto from a jurisdiction outside of Canada, please note that you may be subject to the laws of the country in which you are residing, or any country of which you have citizenship. The University of Toronto has a long-established commitment to freedom of expression, with this right enabled by an environment valuing respect, diversity, and inclusion. In your classes, you may be assigned readings, or discuss topics that are against the law in other jurisdictions. I encourage you to become familiar with any local laws that may apply to you and any potential impact on you if course content and information could be considered illegal, controversial, or politically sensitive. If you have any concerns about these issues, please contact your instructor directly to discuss with them.


    The following schedule describes the current planing status and the covered concepts. It is subject to change and will be updated as the semester progresses, especially to help focus on requested topics or support learning.

    Week Date Topic
    1 1/13 Introduction
    2 1/20 Research Basics & Research Questions (1)
    3 1/27 Research Questions (2) & LiteratureReview & Theory
    4 2/3 Interview studies
    5 2/10 Grounded Theory
    6 2/17 Survey & Validity & Ethics
    [X] 2/21-25 Reading Week
    7 3/3 Experimentation (1) & Descriptive statistics
    8 3/10 Hypothesis Testing & T-test & ANOVA & Chi-square statistics
    9 3/17 Experimentation (2)
    10 3/24 Time Series Analysis
    11 3/31 Case study & Final Presentation 1
    12 4/7 Final Presentation 2
    13 4/14 Final Presentation 3

    Course Syllabus

    The course uses Quercus for announcements, homework submission, grading, Piazza for questions, discussion, and supplementary documents; slides will be posted both here and on Quercus. See the pdf version of syllabus here.

    Expected Course Workload: Student workload should average to about 10 hours per week for ALL course-related activities (reference), including “contact” and/or “viewing” time, assignment/project completion, self-study, etc. In general, 3 hours/week will be spent in class and 7 hours on reading and project. Please feel free to give the course staff feedback on how much time the course is taking for you.

    Communication: We make announcements through Quercus, including clarifying homework assignments and other interactions. The instructors and TAs are reachable by email; see above for information on how to contact us.

    Textbook: Various readings throughout the semester available online or through the library; we do not have a single text book but rather assemble readings from different sources.

    Assignments & Grading
    Following the FASE UofT Grading Policy, the evaluation of the course will be based on the following distribution:

    Course Policies

    Late work policy: There are no late days. Exceptions to this policy will be made only in extraordinary circumstances, almost always involving a family or medical emergency---with your academic advisor or the Dean of Student Affairs requesting the exception on your behalf. Accommodations are possible if requested at least 3 days in advance.

    Notice of video recording and sharing (Download permissible; re-use prohibited) This course, including your participation, will be recorded on video and will be available to students in the course for viewing remotely and after each session. Course videos and materials belong to your instructor, the University, and/or other source depending on the specific facts of each situation, and are protected by copyright. In this course, you are permitted to download session videos and materials for your own academic use, but you should not copy, share, or use them for any other purpose without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.

    Academic Integrity Policies and Collaboration: The usual policies apply, especially the University Policy on Academic Integrity and the Code of Behaviour on Academic Matters at UofT.

    We expect that group members collaborate with one another, but that groups work independently from one another, not exchanging results with other groups. Within groups, we expect that you are honest about your contribution to the group's work. This implies not taking credit for others' work and not covering for team members that have not contributed to the team. Otherwise, our expectations regarding academic honestly and collaboration for group work are the same as for individual work, substituting elevated to the level of "group."

    Here are some examples of behavior that are inappropriate:

    If any of your work contains any statement that was not written by you, you must put it in quotes and cite the source. If you are paraphrasing an idea you read elsewhere, you must acknowledge the source. Using existing material without proper citation is plagiarism, a form of cheating. If there is any question about whether the material is permitted, you must get permission in advance. We will be using automated systems to detect software plagiarism.

    It is not considered cheating to clarify vague points in the assignments, lectures, lecture notes; to give help or receive help in using the computer systems, compilers, debuggers, profilers, or other facilities; or to discuss ideas at a very high level, without referring to or producing code.

    Any violation of this policy is cheating. The minimum penalty for cheating (including plagiarism) will be a zero grade for the whole assignment. Cheating incidents will also be reported through University channels, with possible additional disciplinary action (see the above-linked University Policy on Academic Integrity).

    If you have any question about how this policy applies in a particular situation, ask the instructors or TAs for clarification."

    Note that the instructors respect honesty in these (and indeed most!) situations.

    Inclusivity, Accommodations & Mental Health Support

    Inclusivity Statement: You belong here. The University of Toronto commits to all students, faculty and staff that you can learn, work and create in a welcoming, respectful and inclusive environment. In this class, we embrace the broadest range of people and encourage their diverse perspectives. This team environment is how we will innovate and improve our collective academic success. You can read the evidence for this approach here.

    We expect each of us to take responsibility for the impact that our language, actions and interactions have on others. Engineering denounces discrimination, harassment and unwelcoming behaviour in all its forms. You have rights under the Ontario Human Rights Code. If you experience or witness any form of harassment or discrimination, including but not limited to, acts of racism, sexism, Islamophobia, anti-Semitism, homophobia, transphobia, ableism and ageism, please tell someone so we can intervene. Engineering takes these reports extremely seriously. You can talk to anyone you feel comfortable approaching, including your professor or TA, an academic advisor, our Assistant Dean, Diversity, Inclusion and Professionalism, the Engineering Equity Diversity & Inclusion Action Group, any staff member or a U of T Equity Office.

    You are not alone. Here you can find a list of clubs and groups that support people who identify in many diverse ways. Working together, we can all achieve our full potential.

    Syllabus Statement on Accommodations: The University of Toronto supports accommodations for students with diverse learning needs, which may be associated with mental health conditions, learning disabilities, autism spectrum, ADHD, mobility impairments, functional/fine motor impairments, concussion or head injury, blindness and low vision, chronic health conditions, addictions, deafness and hearing loss, communication disorders and/or temporary disabilities, such as fractures and severe sprains, or recovery from an operation. If you have a learning need requiring an accommodation the University of Toronto recommends that students register as soon as possible with Accessibility Services at here.
    Phone: 416-978-8060
    Email: accessibility.services@utoronto.ca

    Mental Health Statement: As a university student, you may experience a range of health and/or mental health challenges that could result in significant barriers to achieving your personal and academic goals. Please note, the University of Toronto and the Faculty of Applied Science & Engineering offer a wide range of free and confidential services that could assist you during these times.
    As a U of T Engineering student, you have an Academic Advisor (undergraduate students) or a Graduate Administrator (graduate students) who can support you by advising on personal matters that impact your academics. Other resources that you may find helpful are listed on the U of T Engineering Mental Health & Wellness webpage, and a small selection are also included here:

  • Accessibility Services & the On-Location Advisor
  • Graduate Engineering Council of Students’ Mental Wellness Commission
  • Health & Wellness and the On-Location Health & Wellness Engineering Counsellor
  • Inclusion & Transition Advisor
  • U of T Engineering Learning Strategist and Academic Success
  • My Student Support Program (MySSP)
  • Registrar’s Office
  • SKULE Mental Wellness
  • Scholarships & Financial Aid Office & Advisor
  • If you find yourself feeling distressed and in need of more immediate support resources, consider reaching out to the counsellors at My Student Support Program (MySSP) or visiting the Feeling Distressed webpage.