Player Reliability

The list team considers the opinions of every victor and player with significant progress when constructing an opinion list. However, opinions may be weighted differently based on player reliability. Players will be asked by a team member to self-assess their reliability for each list opinion on a scale from 0 to 10. Reliability factors that players should consider include but are not limited to the following list:

• Verifications: Verifiers that change gameplay prior to completing the full level tend to have outlier opinions. As such, unless there are little to no victors of a given level, verifiers tend to be omitted from opinion lists.
  
  
• Previous Experience: Players who have completed a level as a hardest demon and/or have not completed other levels around the same difficulty are generally considered less reliable than players with more experience.
  
  
• Level Experience: A player’s personal experience with a particular level may make their opinion less reliable, such as getting “unlucky”, playing with different peripherals, or other factors external to the level itself.
  
  
• Time Between Completions: A direct comparison may be unreliable if there is a large time gap between completions. This could be due to a player’s skill level changing or an opinion becoming less clear over time.
  
  

Player Exclusion

Players that do not cooperate with the list team when providing opinions (e.g. deliberately providing inaccurate information) will not be considered for future list placements or movements. Players may also choose to be excluded from providing list opinions by contacting a member of the list team.

Low Detail Modes

If a level contains a built-in Low Detail Mode that makes it significantly easier, the list team will primarily consider opinions from players that use the LDM. This allows the list placement to correspond to the easiest “official” version of the level.

Opinion List Analysis

When considering potential level placements and movements, players are contacted by the list team for their placement opinion and reliability. An opinion sheet is then used to aggregate the opinions of all reliable players.

When starting direct comparisons, the level currently placed at the mean (excluding outliers) or median placement opinion is usually compared to first. If two levels have a low number of mutual victors, the nearest level with a higher number of mutual victors may be chosen for comparison instead. If a level wins a comparison, it will be compared to levels higher on the list, and vice versa. This continues until a placement near the mean/median is found where the level wins below and loses above in direct comparisons.

Once an opinion sheet is complete, the placement or movement must be approved by multiple members of the list team before any change is made to the demon list.