Project Review Information

Author

Andreas Handel

Modified

2024-04-10

Overview

You will be asked to review the projects of a few of your classmates twice, once for part 3, and once at the end (part 5).

I have several goals for these project reviews:

  • You get feedback that helps you improve your project.
  • You get experience with giving and receiving reviewer feedback, like you would when you submit a paper for publication or review someone’s submitted paper.
  • You practice being on the ‘receiving end’ of reproducible research, i.e. you will have to be able to reproduce someone else’s project so you can properly critique it.

General Logistics

  • By the deadline specified in the Schedule, everyone has to push their project to their project repository.
  • I will post a list indicating which projects everyone is assigned to review.
  • Project reviews happen the following week. For part 3, this is during the semester. For the final, in-depth review, there is dedicated time at the end of the semester.
  • You can find links to project repositories for your classmates in the Project-related channel.
  • If your project is a private repository, make sure to add the students assigned as reviewers as collaborators to your project.
  • For reviewing, you can either use the fork and pull workflow, or the project owner can make you a collaborator and you can pull/push directly. Fork and pull is maybe a bit safer since there are now 2 reviewers making changes to the repository at the same time. But you can decide on the workflow. Working as a collaborator will be required if the project happens to use a private repository.

Part 3 review

This review is meant to provide hopefully helpful feedback to each other and to see some projects other than your own to hopefully get further ideas on what to do and how to do things.

Part 3 review logistics

  • Find the projects you are assigned to. If you have been asked to review a private project, make sure you are added as collaborator.
  • Follow the instructions provided in the repository to run all code and reproduce everything.
  • Use the simple grading rubric shown in the part 3 section of the Project Details document.
  • Submit your review by filing one (or several) GitHub Issues in the project repository. Provide any feedback you think your classmate(s) might find useful to help them further improve their project and ensure they are on the right track. Also provide your score as indicated in the rubric.

Part 5 review

After the project has been completed, each of you will review a few of the projects of your classmates, similar to peer review of a manuscript submitted for publication. The part 5 review should be much more detailed and in-depth than the part 3 review.

Part 5 review logistics

Follow the instructions provided in the repository to run all code and reproduce everything. Review the whole project using this review template (right-click to download).

Write up a detailed review using the provided template. Once you are done, place your completed review document into the main folder of the repository you are reviewing, and send a pull request (or do a direct push) for each project you review.

As you reproduce the project, you will likely make a lot of changes to the repository (e.g., by re-creating figures/tables). If you and another reviewer do that both, you might run into conflicts when sending a pull request. If you encounter problems like this, you can provide the project owner your completed review in some other manner (e.g. through email or Discord) and they’ll add it to the repository. The important part is that at the end of the review process, there should be (at least) 2 completed review documents in the project repository (either the main folder, or make a reviews sub-folder for them.) I need to be able to find the reviews for grading purposes.

Review Assessment Rubric

Ok, now it gets maybe a bit complicated. I will review and assess the quality of your reviews. To that end, I’ll use a fairly simple rubric, similar to the ones for the previous project submissions.

Category Description Score
Sufficient Reviews are complete or fairly complete 3
Somewhat insufficient Reviews are somewhat incomplete, lack useful/detailed feedback 2
Insufficient Only one review was submitted, or submissions were very incomplete 1
Absent No reviews were submitted 0

You get a full week to do those reviews, so I expect thorough and good quality work!

Finals steps

  • Use the peer feedback/reviews from your classmates to further improve your project. You can make any further changes you want to make.
  • Push your final project to your GitHub repository by the specified deadline for final grading.
  • I will assess your final project using the same categories in the template used during peer review.