After three years of increasing momentum, the SC Reproducibility Initiative makes Artifact Description (AD) Appendices mandatory for all papers submitted to the SC19 Technical Program, but optional for workshop papers and posters. Artifact Evaluation (AE) Appendices are still optional.

AD/AE Appendices will now be auto-generated from author responses to a standard form, embedded in the SC online submission system.

In 2019, the SC Reproducibility Initiative introduces three new tracks/committees, under the SC Reproducibility Chair:

1. AD/AE Appendices
2. Reproducibility Challenge
3. Journal Special Issue

 

New SC Reproducibility Tracks/Committees

AD/AE Appendices

This committee has the following charge:

• Implement a program of actions to mentor authors in the preparation of the appendices. This may include a webinar, blog post, in-person consultation at the conference, expanding the Reproducibility Initiative FAQ, etc.;
• Inspect the appendices of submitted papers, follow artifact URLs to check their contents, and support authors with appendix improvement, if needed.

Reproducibility Challenge

This committee has the following charge:

• Select a paper accepted to the previous conference to be used as source of the Reproducibility Challenge in the Student Cluster Competition (SCC) of the next conference;
• Work with the authors of the selected paper to build the challenge benchmark for the SCC teams;
• Continue working until the time of the conference in crafting the rules and requirements for the challenge, and ensure compatibility with the various hardware used by the SCC teams.

Journal Special Issue

This committee has the following charge:

• Review full papers submitted by the SCC teams for publication in the journal special issue, reporting on the results of the efforts to reproduce the challenge benchmark;
• Work with student teams to assist them in revision of their papers, for a successful final publication of the reports in the special issue.

 

AD/AE Appendix Form

All authors submitting to the SC19 Technical Papers program must complete the AD/AE Appendix form to describe the computational artifacts their results rely on: software, data, and hardware. The form is standard, and embedded in the conference submission system.

View sample AD/AE Appendix Form within the Papers submission form sample under Paper Artifact Description / Article Evaluation (AD/AE) Appendix.

Note: If your paper used no computational artifacts, you will respond “No” to the first question and be done.

Submissions will be double-blind reviewed, and reviewers will have all the information in the AD/AE form except for the URLs pointing to the artifacts. The paper and the appendix, including URLs to artifacts, will be available to the AD/AE Appendices Committee. The committee may provide feedback to authors in a double-open arrangement (authors and committee members are known to each other). The AD/AE Appendices Committee will not share any information with the Program Committee other than to confirm whether artifacts indicated as available are in fact available. “Available” means that the provided URLs properly resolve, and that the author-created artifacts are persistently archived and have a global, unique identifier. Accepted papers with available artifacts will display an ACM badge.

Additional notes:

• The form targets both Artifact Description (required) and Evaluation (optional).
• Availability of software, data, and hardware artifacts must be explicitly indicated.
• Papers should still be self-contained: they should describe the experimental methodology enough for the Program Committee to evaluate the contribution.
• Sample scripts for machine-generated environment data are provided here.
• Authors are encouraged to familiarize themselves early with the AD/AE Appendix requirements. Contact us with your questions.

 

History of the SC Reproducibility Initiative

2015

The SC steering committee approved the reproducibility initiative. Authors of SC15 papers were invited to submit an AD Appendix after the conference: one paper did so, became the source for the SC16 Student Cluster Competition Reproducibility Challenge and the first SC paper to display an ACM badge.

2016

Authors submitting to the SC16 conference could optionally submit an AD Appendix: nine authors submitted one, three were finalists, and one was selected to become the source for the SC17 Student Cluster Competition Reproducibility Challenge.

2017

SC made the AD Appendix a requirement to be considered for the Best Paper or Best Student Paper awards. SC17 also introduced the Computational Results Analysis (CRA) Appendix. 40% of submitted and 50% of accepted papers included an AD appendix; nine submitted papers (six accepted) included a CRA Appendix.

2018

SC extended the option of submitting AD Appendices to Workshops and Posters. The CRA Appendix was renamed Artifact Evaluation (AE) Appendix, and limited to four pages. AD Appendices were limited to 2 pages and remained optional (but required for consideration as Best Paper/Best Student Paper, and also Best Poster/Best Student Poster).

2019

AD Appendices will be mandatory for all submissions. AE Appendices are still optional, and both will be submitted via a standard form in the conference submission system. Three new Technical Program tracks, with their respective committees and chairs, are introduced in support of the SC Reproducibility Initiative.

 

Rationale and Results of Reproducibility at SC

The SC Conference Series is committed to introducing activities that highlight, enhance, and reward participant efforts to improve reproducibility. Each year, authors have engaged more with the Reproducibility Initiative, and the benefits have become more clear to our community.

The AD/AE Appendices improve the quality of SC papers, and increase efficiency of our collective endeavors:

• The AD appendix provides information (details of software, hardware and data) that should be part of any publication with computational results.
• The new AD/AE Appendices form standardizes the items that authors should include.
• If an SC paper used no computational artifacts (software, hardware, or data), the author will click “No” in answer to the first question, and be done.
• In the AE Appendix, authors describe how they verified and validated their computational results, supporting their trustworthiness and providing best-practices descriptions to the community that can be adopted by others who want to improve their own efforts.
• AD/AE Appendices are reviewed in a double-open format by a dedicated committee, independent of the Technical Program committee (which maintains a double-blind process).

 

Engaging the Student Cluster Competition

The SC Reproducibility Challenge committee, now a part of the Technical Program, works with the SC Reproducibility Chair to select a paper from the previous conference to be used as source of the Reproducibility Challenge the next year. This committee works with the authors of the selected paper to develop the challenge benchmark and ensure that SCC teams have the best chance at succeeding.

The SCC teams write extended reports of their results and their efforts with the Reproducibility Challenge, which are submitted for a journal special issue (where they are peer-reviewed before acceptance).

The SC paper selected for the Reproducibility Challenge receives a special recognition at the SC Awards Ceremony.