You will be asked to review the projects of a few of your classmates twice, once for part 3, and once at the end (part 5).

I have several goals for these project reviews:

  • You get feedback that helps you improve your project.
  • You get experience with giving and receiving reviewer feedback, like you would when you submit a paper for publication or review someone’s submitted paper.
  • You practice being on the ‘receiving end’ of reproducible research, i.e. you will have to be able to reproduce someone else’s project so you can properly critique it.

General Logistics

  • By the deadline specified in the Schedule, everyone has to push their project to their project repository.
  • Project reviews happen the following week. For part 3, this is during the semester. For the final, in-depth review, there is dedicated time at the end of the semester.
  • I will post a list indicating which projects everyone is assigned to review to Slack. Watch the announcements channel.
  • You can find links to project repositories for your classmates in the project_related Slack channel.
  • For reviewing, you can either use the fork and pull workflow, or the project owner can make you a collaborator and you can pull/push directly. Fork and pull is maybe a bit safer since there are now 2 reviewers making changes to the repository at the same time. But you can decide on the workflow. Working as a collaborator will be required if the project happens to use a private repository.

Part 3 review

This review is meant to give each other some informal feedback and guidance to ensure everyone is on the right track.

Part 3 review logistics

  • Follow the instructions provided in the repository to run all code and reproduce everything.
  • Use the simple grading rubric shown in the part 3 section of the Project Rubric document.
  • Provide your feedback and your score by filing a GitHub Issue in the project repository. As part of the issue, provide any feedback you think your classmate might find useful to help them improve and be on the right track. Also provide your score as indicated in the rubric.

Part 5 review

After the project has been completed, each of you will review a few of the projects of your classmates, similar to peer review of a manuscript submitted for publication. The part 5 review should be much more detailed and in-depth than the part 3 review.

Part 5 review logistics

Follow the instructions provided in the repository to run all code and reproduce everything. Review the whole project using this review template (right-klick to download). * Write up a detailed review using and filling out the template. Once you are done, place your completed review document into the repository you are reviewing, and send a pull request (or do a direct push) for each project you review.

As you reproduce the project, you will likely make a lot of changes to the repository (e.g., by re-creating figures/tables). If you and another reviewer do that both, you might run into conflicts when sending a pull request. If you encounter problems like this, you can provide the project owner your completed review in some other manner (e.g. through email or Slack) and they’ll add it to the repository. The important part is that at the end of the review process, there should be (at least) 2 completed review documents in the project repository (either the main folder, or make a reviews subfolder for them.)

Review Assessment Rubric

Ok, now it gets maybe a bit complicated. I will review and assess the quality of your reviews. To that end, I’ll use a fairly simple rubric, similar to the ones for the previous project submissions.

Category Description Score
Sufficient Reviews are complete or fairly complete 3
Somewhat insufficient Reviews are somewhat incomplete, lack useful/detailed feedback 2
Insufficient Only one review was submitted, or submissions were very incomplete 1
Absent No reviews were submitted 0

You get a full week to do those reviews, so I expect thorough and good quality work!

Finals steps

  • Use the peer feedback/reviews from your classmates to further improve your project. You can make any further changes you want to make.
  • Push your final project to your GitHub repository by the specified deadline for final grading.
  • I will assess your final project using the same categories in the template used during peer review.