Configuration of the Reviewing Functions

The quality of the presentations is essential for every academic conference and decides about acceptance and reputation of the event. Therefore, it is important to ensure that the submission and reviewing process that you want to employ is correctly configured in ConfTool Pro.

Each conference organizer has specific ideas how to perform the reviewing process. Some of the questions that arise are: How many reviewers are required per submission? Are open, single, or double blind reviews required? Will abstracts be reviewed, full papers, or both? What criteria are used to evaluate papers, and will there be review scores or only written comments or both?

The following manual will explain the configuration options for the reviewing module of ConfTool Pro. In case this is your first conference, you can follow the manual step by step. Experienced organizers can feel free to read specific sections only for detailed explanations.

Please note that several of the settings for the reviewing module are closely connected to the submission module. Thus, make sure that you have understood and configured the submission process appropriately. You can find all relevant information here: Submission Module Configuration.

This manual starts with a number of questions that can help you to identify the strategy behind your reviewing process. It continues with a short overview of the review process from the reviewer's point of view, to illustrate the concepts behind ConfTool Pro. Furthermore, you will find explanations on the most important features and options in a survey of the review module settings as well as the review form configuration. Finally, we have compiled some hints on the final step of the reviewing process: How to assign acceptance statuses to submissions. Please note that all options and settings are also explained online on the settings pages of the system. For less frequently requested options, take a look at the online forum, where you can find a lot of additional information.

Preparation and General Questions about the Requirements

  • How do you want to recruit reviewers? Did they already confirm their participation or do you want to invite them first?
  • How many reviewers are required per submission?
  • What phases does your reviewing process have and how long is each of them?
  • Shall anonymity be unilateral or reciprocal (single- or double-blinded reviewing)?
  • Shall there be no anonymity at all (open review process)?
  • How many submissions are you going to accept?
  • What are the evaluation criteria?
  • How are the criteria weighted?
  • What are reasons for rejection?
  • What are the rules for conflict of interest (COI)?
  • Can reviewers refuse to review a paper?
  • Shall only abstracts or full paper uploads be reviewed?
  • Do you require a two-phase peer reviewing process? (You can find more information in the support forum.)
  • What details of the review results do you want to pass on to the authors? Will it include narrative feedback or scores?
  • Shall authors be able to respond to the review (as narrative only or including a review rating)?
  • Is there a final upload phase, where authors can submit a revised version of their papers that considers the feedback of the reviewers?
  • Do authors get the chance to rate or comment reviews?
Image 1: Sample review form | click on image to enlarge

Overview of the Reviewing Process

Short Overview

The reviewing process in ConfTool is straightforward.

First of all, each reviewer will require a user account, so they can take part in the reviewing process. These accounts allow organizers to assign reviews to reviewers, and reviewers to access the submissions assigned to them.

These user accounts with the user role "reviewer" can be established manually in the back end of the system. Another option is to import users via the ConfTool import function. Simply create an Excel sheet with the first and last name and e-mail addresses of all reviewers and follow the instructions for user import. If the reviewers have not confirmed their participation, you can also invite reviewers by using the invitation function of ConfTool Pro.

Secondly, the received submissions have to be assigned to reviewers. This can be done automatically or manually. After this step, you can enable the reviewing phase.

As a last step, you can inform reviewers that they can start the review process by using the bulk e-mail function.

Reviewer Account

When reviewers log in to their account, they will find a screen which contains at least two main boxes. All reviewing functionalities can be reached in the box with the heading "You are reviewer or member of the program committee (PC member). You currently have the following options" (see image 2).

Image 2: Reviewer account options | click on image to enlarge

Detailed Steps and Options

The review process can also include further steps for reviewers. All these operations are performed online via ConfTool Pro:

  • Reviewers can use "Select Priority Topics" to select their area of expertise. This information will support conference chairs to assign contributions to reviewers with matching interests. It is therefore important for the quality of the review process.
  • During this phase, reviewers can also be allowed to state the maximum number of reviews which they are willing to do (see image 2).
  • In a bidding phase (after the submission deadline has passed), reviewers can "Bid for Contributions". Reviewers will find the list of all submitted contributions of the conference shown in their account. Contributions are listed by matching topics and reviewers can state their preference (much wanted, would like, uncertain, would dislike) or a conflict of interest (see image 2).
  • After these phases are over, the chairs will assign submissions to reviewers, e.g. based on matching topics and biddings.
  • After enabling the reviewing phase, reviewers will then find the submission assigned to them for reviewing in their user accounts. Now, they can "Enter and Edit Reviews". They will find details of the proposals (including uploads, if available), which they can read online or download. Reviewers will be able to enter their evaluation online. Depending on the review form, reviews will be based on review factors and / or textual feedback.
  • It is possible to let reviewers have access to evaluations from other reviewers. Reviewers can furthermore be given access to the online forum after they have submitted their reviews, in order to leave comments and discuss about their own submissions.
  • Furthermore, you have the option to define Meta-Reviewer and Senior Reviewer who always have access to all reviews of those submissions they were assigned to. Additionally they can use the ConfTool system to send messages to all other Reviewers.
  • Members of the Program Committee can have an online discussion via the online forum and vote for or against acceptance for all submissions.
  • You can include a feedback loop for authors. When you enable the rebuttal function, authors will have the chance to reply to all reviews that their submission received.
  • In consequence, reviewers might be asked to revise and refine their evaluations based on the comments of the authors.
  • Authors can also be allowed to rate each review separately on a scale in order to judge the quality of reviews and the work of reviewers.
  • Subsequently, the chairs will decide about acceptance based on all reviews that were received (taking into account the results of discussions and recommendations in the online forum and rebuttals and review ratings, if available).
  • When review results are published, authors of accepted papers might be asked to incorporate the hints and comments from the evaluations and the final result of the program committee in a final upload or camera-ready copy.

 

Survey of the Configuration Options

You will find most configuration options in the box "Parameters for the Reviewing Functions of ConfTool". Additional options can be found when you configure each submission type / track, as they are specific for each submission type / track (see image 3).

Image 3: Reviewing configuration options overview | click on image to enlarge

Let us start with the main options for paper reviewing, which you find here:
Overview => Settings => Main Setting for Paper Reviewing

Here, you can access general settings, such as deadlines, access rights and display options. Many of them are explained in the software itself. Therefore - in the following - we will only take a closer look at the most important of them.

Display Options

In the section "Display Options", you can enter a short text used as "Introductory text for the page 'Enter and Edit Reviews'". This information will be displayed to reviewers when they access the system and will appear above the list of proposals to be evaluated. Often, short texts are more practical. If you have detailed instructions, it could be a good idea to include a link to an external source where they are explained in full.

In the following section "Reviewing phase: PC members and reviewers can access the submitted contributions assigned to them for review" you can activate the reviewing module by selecting the check box for the option "Enable module and select start and end date". Once activated, you can use the option "Set start and end date" to let the software automatically start and close the reviewing phase (see image 4).

Image 4: Display options and reviewing phase | click on image to enlarge

Review Options

Moving on to the section "Review Options", you can first set a "Method for 'Conflict of Interest' Identification". The software will cross-check the details of reviewers and the details given in the submissions. Choose one of the two options. Later on, when you assign reviewers to submissions, a conflict of interest (COI) warning will appear wherever the software has detected similarities according to the COI rule you have chosen (see image 5).

Image 5: Conflict of interest warning | click on image to enlarge

The next two options "Enable Meta-Reviews?" and "Enable Senior Reviews?" are almost identical. When you activate one or both of them, you will get the additional roles of Meta-Reviewer and Senior Reviewer. Users with either of these roles have read access to reviews of other experts assigned to the same submission and can contact them via e-mail and the ConfTool messaging function, e.g. to remind them of deadlines. Furthermore, they can summarize the review results in their own review.

If you want to assign a larger number of reviews to each reviewer, please use the option "Enable Filter for Reviewers", which will enable a function to filter the proposals by submission type / track or open / completed reviews on a reviewer's "Enter and Edit Reviews" page.

In some cases, the option "Finalize Reviews" can come in handy. When you activate it, reviewers will have to finalize their review to submit it, i.e. they cannot update their evaluations after submission.

If you want your reviewers to be able to refuse to work on an evaluation that has been assigned to them, please activate the option "Do Reviewers have the Option to Refuse Reviewing a Submission". The reviewer will have to state a reason why he / she cannot write the review and the chairs / track chairs will be informed about the refusal via e-mail.

You will then find an additional option to manage the reasons for reviewers to refuse an evaluation here:
Overview => Settings => Manage Reasons to Refuse an Evaluation

In case your reviewers will also have access to PDF files uploaded by the authors, the option "Add Watermark to PDF Downloads" - when activated - will automatically create a watermark on the PDF for reviewers that reads "For Peer Review Only".

You can allow reviewers / PC members to select their maximum number of reviews with the option "Selection of No. of Reviews". Define a span for the minimum and maximum number of reviews and also set a default in the option "No. of Reviews per Reviewer". These numbers will be considered during the automatic review assign process (if possible) and be shown to chairs for the manual assignment process. Please activate the phase and set a start and end date for "Reviewers Can Select Their Priority Topics" to enable the option for the selection of the number of reviews (see image 6).

Image 6: Review options | click on image to enlarge

Reviewers Can Select Their Priority Topics

In the next section "Reviewers Can Select Their Priority Topics", you can let reviewers / PC members or all users select their priority topics. This is helpful for matching (potential) reviewers with submissions. Please activate the phase and set a start and end date. Choose an appropriate setting for the option "Priority Topic Selection". Furthermore, you can activate an automatic redirection to the priority topic selection page after user login if you activate the option "Test Topic Selection at Login" (see image 7).

Image 7: Reviewers – priority topics | click on image to enlarge

Bidding: Reviewers Can Bid for Submitted Contributions

For some conferences, it is convenient to enable a phase during which reviewers indicate their preferred papers and conflicts of interest (The options available are: much wanted, would like, uncertain, would dislike, and conflict). You can activate this option in the section "Bidding: Reviewers Can Bid for Submitted Contributions". Again, activate the phase and set a start and end date. Choose the maximum number of submissions that a reviewer can select as "much wanted" to review in the option "Biddings for "Much Wanted" Reviews" (see image 8). The bidding function can also be enabled or disabled for each submission type / track separately. Biddings further simplify the review assignment process and reduce the risks of a conflict of interest.

Image 8: Reviewers – bidding | click on image to enlarge

Enable Rebuttals for Authors

Some conferences allow authors to write a rebuttal as a reply to the evaluation of the reviewers. The rebuttal can refer to all reviews and there are no separate rebuttal forms for each single review. A rebuttal to a submission will appear in the account of all reviewers who were assigned to the submission. The reviewers then might refine their original review. Use the option "Enable Rebuttals for Authors" to activate this feature. Set a maximum number of words using the option "Maximum length of rebuttals". Don't forget to activate the phase and set a start and end date (see image 9). Reviewers and members of the program committee will find the rebuttals in the online forum (if enabled, see below) and in the details of the submissions; admins can also export all contributions with review details including rebuttals as XLS.

Image 9: Rebuttals for authors | click on image to enlarge

Enable Ratings of Reviews by Authors

A similar feature is "Enable Ratings of Reviews by Authors". When you enable it, the authors can rate each review separately. Enable the module and set a start and end date. Reviewers / PC Members will find the review ratings in the online forum (if enabled, see below) and in the details of the submissions; these hints might be helpful to assess the quality of the reviews and the work of the reviewers (see image 10).

Please specify the elements of the review rating form for authors here:
Overview => Settings => Manage Elements of the Review Rating Form for Authors

Create categories and enable or disable them. Then define Min and Max scores in the option "Range of Scores for all Ratings". In the section "Text Fields of the Review Rating Form" you can define if one part of the message created by the authors should be directed to the reviewers and another to the program committee. Finally, you can choose an appropriate setting for the option "Visibility of the Review Rating Results" to decide what part of the reviews rating the reviewer will get to see.

Image 10: Authors – review ratings | click on image to enlarge

Settings for Online Forum for Review Discussion

The section "Settings for Online Forum for Review Discussion" deals with options of an online discussion tool, with which members of the program committee can read all submissions and reviews and create messages for other members with their opinion whether the submission should be accepted, including a personal message (see image 12). To use this module, activate it and set a start and end date. You can choose for each submission type / track separately if the submissions of the track shall be available in the forum. With the option "Access to reviews of co-reviewers? / Access of Reviewers to Online Forum?" you can allow reviewers to get access to reviews from other reviewers and the online forum to exchange messages. You can choose from several display options regarding anonymity using "Show names of message authors". If the uploads are of importance also during the discussions in the online forum, enable the option "Uploaded files available in program committee online forum". You can also let PC members have access to rebuttals and review ratings in the forum when you enable the option "Rebuttals and/or Review Ratings available in program committee online forum". This additional information could help the committee to make the best possible decision (see image 11).

You can also use the online forum for a slightly different purpose: the discussion about the presentations that have already been assigned to sessions in the program via the ConfTool agenda. PC Members can use it to write internal comments about the presentations and make these available to all other PC Members to see. To use this function, please activate the option "Link Conference Agenda with Online Forum". NB: When you use the online forum for this purpose, please disable the options "Enter Votes with Forum Messages" and "Close Discussion after Decision" (set both to "No").

If you activate the option "Enter Votes with Forum Messages", PC members have to state their vote: "Accept", "Maybe accept", "No vote / undetermined", "Maybe reject", "Reject" (see image 12). All votes will be aggregated and visualized here: Overview => Submissions & Reviews => Results of the Reviewing Procedure & Decision About Acceptance. This can facilitate the decision making process (see image 13).

The option "Close Discussion after Decision" can be used to no longer allow any discussions in the online forum on a submission when an acceptance status other than "On Hold" has been chosen. (More specifically: acceptance statuses for which the option "Show Results" is enabled).

Image 11: Review settings online forum | click on image to enlarge
Image 12: PC Member account – enter forum message | click on image to enlarge
Image 13: Reviewing results - forum messages overview | click on image to enlarge

Results of the Reviewing Procedures Are Accessible to Authors

Finally, in the section "Results of the Reviewing Procedures Are Accessible to Authors" you can determine when authors will have access to the review results of submissions for which the acceptance status has been changed from "On Hold" to another status. Enable the module and set a start and end date (see image 14).

Image 14: Review settings - results accessible | click on image to enlarge

Configuration of the Review Forms in ConfTool Pro

You can set further review options and the preferred review form per conference track / submission type (see image below). Please go to:
Overview => Settings => Manage Submission Types and Conference Tracks
to edit your submission type(s).

Scroll down to the section "Review option". With the option "Double Blind Reviews" you can hide authors' names from reviewers, but please note that this will (of course) not remove any author information from the content of the submitted files. Only the file names are anonymized.

"Bid for contributions" can be enabled or disabled for each submission type / track separately. The module has to be enabled and the phase set in (as described above in this manual):
Overview => Settings => Main Setting for Paper Reviewing

The option "Enter and Edit Reviews" allows you to disable the reviewing function completely for a specific submission type / track. When this option is disabled, reviewers will not be able to enter any reviews. This option can be useful when you want to skip the reviewing process and e.g. assign an acceptance status based on other criteria, e.g. in general fits the conference theme or for invited papers.

The "Online Forum for the Program Committee" can also be enabled or disabled for each submission type / track separately. The module has to be enabled and the phase set in (as described above in this manual):
Overview => Settings => Main Setting for Paper Reviewing

With the option "Review Results", you can choose for each submission type / track, if the full review results will be shown to the authors (the corresponding phase must also be enabled and an acceptance status be set) or only the acceptance status (for which the option "Show Results" is enabled) without the comments from the reviewers will appear in the account of the authors.

In the section "Further Settings ", you can define for the option "Class name of review form" which review form shall be used for the corresponding submission type / track. The system provides four standard review forms you can choose from. Below is a list of these four forms.

In case we have developed a customized review form for you, you can find the according "class name" for the review form here, too (see image 15).

Image 15: Name of review form | click on image to enlarge

You can easily test the review forms when you click on "Related Functions" on the overview page of the submission types / tracks (see image 16):
Overview => Settings => Manage Submission Types / Conference Tracks

Image 16: Test review forms | click on image to enlarge

For the "Default" and the "Simple" forms, the weighting factors for each criterion (review category) can be set, too.

Please go to:
Overview => Settings => Factors for Reviews
... to set the weighting factors for each criterion. You can also disable criteria here. Please note that we do not recommend setting the "Overall recommendation" to 0%, as this might be confusing or lead to (logically) inconsistent review results. If you click on "Edit Phrase", you can edit the wordings of each criterion (see image 17). If you do any updates, please test them!

Image 17: Factors for the reviews | click on image to enlarge

Mainly for the "Default" review form, you will find additional options on this page. If you disable the option "Show Scores to Authors", none of the review criteria (including the scores) will appear in the author's review results. The author will only get to see the acceptance status and the comments for the authors (if this option has been enabled).

You can choose for the option "Range of Total Scores" one of two settings: 0-100 will add all weighted scores of the criteria to a maximum of 100, the setting 0-10 has a maximum of 10.

You can enable the option "Familiarity of Expert". With this option enabled, reviewers have to state their familiarity with the topic of the submission during the review process. The value entered here (from 0 "Completely new to me" to 10 "Very familiar") will be considered when the overall score is calculated.

The option "Comments for the authors" is activated and set to "required" per default. This option will create a form for comments from reviewers. These comments will be made available to the authors when the review results are published. This form therefore usually is recommended, as it is the direct feedback with explanatory notes, hints, criticism and praise from reviewers to authors.

You can let reviewers enter the "Contribution of the submission", i.e. a statement about the impact of the submission on scientific progress. The contribution of the submission as entered by the reviewers will also be made available for the author to read.

When you activate any of the last four options "Best paper award", "Recommended as poster", "Major revision required" and "Veto against acceptance", reviewers will see checkboxes which they can simply check if they think that the statement is true. If a reviewer has checked the corresponding box, an icon will appear for the submission on the list "Results of the Reviewing Procedure & Decision About Acceptance" (see image 18):

Image 18: Additional icons - reviewing results list | click on image to enlarge

If you want to, you can adapt the overall scheme for reviewing and use different labels, factors or weighting for the review categories for each of the submission types / tracks. First, please be advised that you can choose different review forms for different tracks – in a lot of cases, choosing a different review form might already be sufficient. Second, you can also create alternative labels for review categories / factors, create new factors (and disable others) or change the weighting for the factors for each submission type / track separately.

Acceptance Status

The acceptance status is mainly needed as final part of the reviewing process. It is usually set by the chair(s) and is based on the recommendations made by the reviewers. The acceptance status defines for instance if a contribution is rejected, accepted or if it needs revision.

However, in a conference with several review phases, the status can also define that an abstract was first accepted ("Accepted - Abstract") but then rejected as a full paper ("Rejected - Full Paper"). Any "Accepted" status might be further divided into submission types, e.g. "Accepted - Oral Presentation", "Accepted - Full Paper" and "Accepted - Poster Presentation". You can also use the acceptance status to mark certain papers: For instance, distinguish between preliminary accepted papers for which authors have not yet paid their registration fees ("Accepted - Unpaid") and papers for which at least one of the authors has registered for participation and paid the fees ("Accepted - Paid").

Please go to:
Overview => Settings => Manage Acceptance Status
Here, you can define different statuses and the options of each status. The system provides several frequently used examples such as "Accepted", "Rejected" and "Reservations". You might have to add more acceptance statuses, e.g. "Accepted as Oral Presentation" or "Abstract Accepted - Upload Full Paper". To define such new statuses, we recommend using an existing status with similar parameters and copying it, so you just need to update the wording.

Hints:

  • "On Hold" is the default status and will not be shown to authors. Choose a status other than "On Hold" to show the review results to authors.
  • More options are available in the expert settings (click on the cogwheel in the bottom navigation bar to activate them)
  • You find more information about setting the acceptance status for submissions here:
    Evaluating the Review Results and Setting the Acceptance Status of Submissions.