Overview

Recent decades have witnessed the emergence of two important trends in software systems. First, an increasing number of software technologies have a significant social component (e.g. end-user programming, collaborative development tools). Second, software systems have increased significantly in complexity and size challenging traditional development and testing approaches. Empirical research methods play a key role in the evaluation of tools and technologies, and in testing the social and technical theories they embody.

This course provides an overview and hands-on experience with a core of qualitative and quantitative empirical research methods, including interviews, qualitative coding, survey design, and large-scale mining and analysis of data. There will be extensive reading with occasional student presentations about the reading in class, weekly homework assignments, and a semester-long research project for which students must prepare in-class kickoff and final presentations as well as a final report.

We will focus on software engineering related research questions in readings and assignments. Students will mine and integrate data from and across online software repositories (e.g., GitHub and Stack Overflow) and employ a spectrum of data analysis techniques, ranging from statistical modeling to social network analysis. For the final research project, we encourage students to come up with a research question of interest to themselves. The delivery will be a research paper, and one or more empirical methods presented in class have to be part of the paper.

Learning Goals

The learning goals describe what I want students to know or be able to do by the end of the semester. I evaluate whether learning goals have been achieved through assignments, written project reports, and in-class presentations. All learning goals are roughly written in a form "after taking this class, the student should be able to ...".

• Summarize and interpret a body of literature on a particular topic; identify gaps in the literature; write a literature review
• Formulate and motivate research questions
• Compare the suitability of different research designs and research methods in different scenarios; explain the relative strengths and weaknesses
• Design empirical studies for different purposes (e.g., evaluating a tool, understanding a phenomenon); choose appropriate methods and defend the choice
• Combine research methods in a mixed-methods design
• Collect and analyze qualitative and quantitative data
• Design interview protocols and user surveys
• Code qualitative data
• Mine data from online repositories
• Run statistical tests and interpret results
• Build, validate, and interpret regression models
• Draw conclusions from empirical data
• Present results verbally and in writing

Logistics and People

Lectures: Thursday 13:00-16:00 EST
Office hour: by appointment

Learning in a Global Pandemic

This course offers a place where students come together to discuss and debate ideas, learn about themselves and each other, and build community. Although we cannot come together in person this semester, the course team will work hard to support you and your learning, and do our best to cultivate a sense of community.
Professor Brandon Bayne of religious studies at the University of North Carolina at Chapel Hill produced a new syllabus for the summer 2020 edition of his course, addressing the pandemic directly. We’ve borrowed his ideas, with gratitude to Professor Bayne:

Nobody signed up for this.

- Not for the sickness, not for the social distancing, not for the sudden end of our collective lives together on campus

- Not for an online class, not for teaching remotely, not for learning from home, not for mastering new technologies, not for varied access to learning materials

The humane option is the best option.

- We are going to prioritize supporting each other as humans

- We are going to prioritize simple solutions that make sense for the most

- We are going to prioritize sharing resources and communicating clearly

We cannot just do the same thing online.

- Some assignments are no longer possible

- Some expectations are no longer reasonable

- Some objectives are no longer valuable

We will foster intellectual nourishment, social connection, and personal accommodation.

- Accessible asynchronous content for diverse access, time zones, and contexts

- Optional synchronous discussion to learn together and combat isolation

We will remain flexible and adjust to the situation.

- Nobody knows where this is going and what we’ll need to adapt

- Everybody needs support and understanding in this unprecedented moment

Important Notes on Information Security Risk and Teaching Remotely from UofT. If you are a citizen of another country, and/or accessing your courses at the University of Toronto from a jurisdiction outside of Canada, please note that you may be subject to the laws of the country in which you are residing, or any country of which you have citizenship. The University of Toronto has a long-established commitment to freedom of expression, with this right enabled by an environment valuing respect, diversity, and inclusion. In your classes, you may be assigned readings, or discuss topics that are against the law in other jurisdictions. I encourage you to become familiar with any local laws that may apply to you and any potential impact on you if course content and information could be considered illegal, controversial, or politically sensitive. If you have any concerns about these issues, please contact your instructor directly to discuss with them.

Schedule

The following schedule describes the current planing status and the covered concepts. It is subject to change and will be updated as the semester progresses, especially to help focus on requested topics or support learning.

Week	Date	Topic	Reading assignments*
1	1/14	Introduction	HW1-Compare methods
2	1/21	Methods & LiteratureReview & Theory	HW2-Role of Theorys
3	1/28	Interviews	HW3-Compare methods
4	2/4	Qualitative Data Analysis	HW4-Grounded Theory
5	2/11	Surveys
[X]	2/15-19	Reading Week
6	2/25	Introduction to Measurement	HW5-QuantitativeAnalysis
7	3/4	T-test & ANOVA ; Experiments 1
8	3/11	Chi-square statistics ; Experiments 2	HW6-Experiments
9	3/18	Case studies & Ethics	HW7-Paper Reading Assignments
10	3/25	Time Series Analysis
11	4/1	Social Network Analysis
12	4/8	Final Presentation

Course Syllabus

The course uses Quercus for announcements, homework submission, grading, , Piazza for questions, discussion, and supplementary documents; slides will be posted both here and on Quercus. See the pdf version of syllabus here.

Expected Course Workload: Student workload should average to about 10 hours per week for ALL course-related activities (reference), including “contact” and/or “viewing” time, assignment/project completion, self-study, etc. In general, 3 hours/week will be spent in class and 7 hours on reading and project. Please feel free to give the course staff feedback on how much time the course is taking for you.

Communication: We make announcements through Quercus, including clarifying homework assignments and other interactions. The instructors and TAs are reachable by email; see above for information on how to contact us.

Textbook: Various readings throughout the semester available online or through the library; we do not have a single text book but rather assemble readings from different sources.

Assignments & Grading

Following the FASE UofT Grading Policy, the evaluation of the course will be based on the following distribution:

• 40% assignments
• 50% research project
  1. 10% initial project description (proposal)
  2. 2% interim report
  3. 8% final presentation
  4. 30% final report
• 10% participation and in-class presentations

Course Policies

Late work policy: Due to heavy reliance on teamwork in this course there are no late days. Exceptions to this policy will be made only in extraordinary circumstances, almost always involving a family or medical emergency---with your academic advisor or the Dean of Student Affairs requesting the exception on your behalf. Accommodations are possible if requested at least 3 days in advance. Please communicate also with your team about timing issues.

Notice of video recording and sharing (Download permissible; re-use prohibited) This course, including your participation, will be recorded on video and will be available to students in the course for viewing remotely and after each session. Course videos and materials belong to your instructor, the University, and/or other source depending on the specific facts of each situation, and are protected by copyright. In this course, you are permitted to download session videos and materials for your own academic use, but you should not copy, share, or use them for any other purpose without the explicit permission of the instructor. For questions about recording and use of videos in which you appear please contact your instructor.

Academic Integrity Policies and Collaboration: The usual policies apply, especially the University Policy on Academic Integrity and the Code of Behaviour on Academic Matters at UofT.

We expect that group members collaborate with one another, but that groups work independently from one another, not exchanging results with other groups. Within groups, we expect that you are honest about your contribution to the group's work. This implies not taking credit for others' work and not covering for team members that have not contributed to the team. Otherwise, our expectations regarding academic honestly and collaboration for group work are the same as for individual work, substituting elevated to the level of "group."

Here are some examples of behavior that are inappropriate:

• Copying or retyping, or referring to, files or parts of files (such as source code, written text, or unit tests) from another person or source (whether in final or draft form, regardless of the permissions set on the associated files) while producing your own. This is true even if your version includes minor modifications such as style or variable name changes or minor logic modifications.
• Getting help that you do not fully understand, and from someone whom you do not acknowledge on your solution.
• Writing, using, or submitting a program that attempts to alter or erase grading information or otherwise compromise security of course resources.
• Lying to course staff.
• Giving copies of work to others, or allowing someone else to copy or refer to your code or written assignment to produce their own, either in draft or final form. This includes making your work publicly available in a way that other students (current or future) can access your solutions, even if others' access is accidental or incidental to your goals. Beware the privacy settings on your open source accounts!
• Coaching others step-by-step without them understanding your help.

If any of your work contains any statement that was not written by you, you must put it in quotes and cite the source. If you are paraphrasing an idea you read elsewhere, you must acknowledge the source. Using existing material without proper citation is plagiarism, a form of cheating. If there is any question about whether the material is permitted, you must get permission in advance. We will be using automated systems to detect software plagiarism.

It is not considered cheating to clarify vague points in the assignments, lectures, lecture notes; to give help or receive help in using the computer systems, compilers, debuggers, profilers, or other facilities; or to discuss ideas at a very high level, without referring to or producing code.

Any violation of this policy is cheating. The minimum penalty for cheating (including plagiarism) will be a zero grade for the whole assignment. Cheating incidents will also be reported through University channels, with possible additional disciplinary action (see the above-linked University Policy on Academic Integrity).

If you have any question about how this policy applies in a particular situation, ask the instructors or TAs for clarification."

Note that the instructors respect honesty in these (and indeed most!) situations.

Inclusivity, Accommodations & Mental Health Support

Inclusivity Statement: You belong here. The University of Toronto commits to all students, faculty and staff that you can learn, work and create in a welcoming, respectful and inclusive environment. In this class, we embrace the broadest range of people and encourage their diverse perspectives. This team environment is how we will innovate and improve our collective academic success. You can read the evidence for this approach here.

We expect each of us to take responsibility for the impact that our language, actions and interactions have on others. Engineering denounces discrimination, harassment and unwelcoming behaviour in all its forms. You have rights under the Ontario Human Rights Code. If you experience or witness any form of harassment or discrimination, including but not limited to, acts of racism, sexism, Islamophobia, anti-Semitism, homophobia, transphobia, ableism and ageism, please tell someone so we can intervene. Engineering takes these reports extremely seriously. You can talk to anyone you feel comfortable approaching, including your professor or TA, an academic advisor, our Assistant Dean, Diversity, Inclusion and Professionalism, the Engineering Equity Diversity & Inclusion Action Group, any staff member or a U of T Equity Office.

You are not alone. Here you can find a list of clubs and groups that support people who identify in many diverse ways. Working together, we can all achieve our full potential.

Syllabus Statement on Accommodations: The University of Toronto supports accommodations for students with diverse learning needs, which may be associated with mental health conditions, learning disabilities, autism spectrum, ADHD, mobility impairments, functional/fine motor impairments, concussion or head injury, blindness and low vision, chronic health conditions, addictions, deafness and hearing loss, communication disorders and/or temporary disabilities, such as fractures and severe sprains, or recovery from an operation. If you have a learning need requiring an accommodation the University of Toronto recommends that students register as soon as possible with Accessibility Services at here.
Phone: 416-978-8060
Email: accessibility.services@utoronto.ca

Mental Health Statement: As a university student, you may experience a range of health and/or mental health challenges that could result in significant barriers to achieving your personal and academic goals. Please note, the University of Toronto and the Faculty of Applied Science & Engineering offer a wide range of free and confidential services that could assist you during these times.
As a U of T Engineering student, you have an Academic Advisor (undergraduate students) or a Graduate Administrator (graduate students) who can support you by advising on personal matters that impact your academics. Other resources that you may find helpful are listed on the U of T Engineering Mental Health & Wellness webpage, and a small selection are also included here:

Accessibility Services & the On-Location Advisor

Graduate Engineering Council of Students’ Mental Wellness Commission

Health & Wellness and the On-Location Health & Wellness Engineering Counsellor

Inclusion & Transition Advisor

U of T Engineering Learning Strategist and Academic Success

My Student Support Program (MySSP)

Registrar’s Office

SKULE Mental Wellness

Scholarships & Financial Aid Office & Advisor

If you find yourself feeling distressed and in need of more immediate support resources, consider reaching out to the counsellors at My Student Support Program (MySSP) or visiting the Feeling Distressed webpage.