Ranking, Rating or Allocation: The Best Way to Identify Stakeholder Priorities
While there are a variety of ways to measure community stakeholder priorities, the most popular methods are rating, ranking, and allocation. Examples of the three methods are provided in the table below.
| Method | Example |
| Simple Rating An importance rating is given to each item in a list. The mean rating or sum of the top two categories (e.g., “essential” and “very important”) is then used to order the attributes.
| Example: The City of Carlton is considering a number of different approaches to manage traffic in its downtown. Please rate each how important, if at all, each of the following factors is when choosing the best approach. (Scale: essential, very important, somewhat important, not important.) · Traffic congestion · Vehicle safety · Air quality · Construction costs · Impacts on tourism |
| Simple Ranking* Items from a list are ranked in order of importance or preference.
*Currently not available as a question option on Polco. | Example: The City of Carlton is considering a number of different approaches to manage traffic in its downtown. Please rank each of the following factors in order of importance, with “1” being the most important, “2”being the second most important, etc. · Traffic congestion · Vehicle safety · Air quality · Construction costs · Impacts on tourism Which is the most important? Which is the second most important? And so on. |
| Allocation Allocation is a specific method that can be used to determine the order of competing values by using a points or budget allocation method. Respondents are given a sum of money or a set number of points to allocate across a number of items. Items are then ranked by the amount of money or number of points given to each option. | Example: Imagine you work for a state’s department of transportation and you have a budget of $10 million dollars to split among four program areas. How would you distribute your budget? You can split the funding up however you would like but it should total $10 million across the four program areas.
Improving air quality $_____ Improving vehicle safety $_____ Improving traffic congestion $_____ Increasing transit services $_____
Total $10 million dollars |
For survey questions asked of community stakeholders, one major consideration is the cognitive complexity of the task. Questions that are more complex lead to inaccurate responses and lower response rates. Both ranking and allocation methods are more cognitively complex than simple ratings because they involve trade-offs. Researchers have found that people struggle with any form of ranking or allocation when more than 6 choices are involved, and the number decreases for low literacy populations. Even with as few items as four, these question types pose significantly more cognitive challenges. For these reasons, we often avoid use of these question types.
Instead, in most cases we recommend Simple Ratings as they can help measure the magnitude of difference in priority when compared to Simple Rankings. (Simple Rankings only tell us that Option A is more important than B but residents might rate them almost identically in terms of importance or there may exist a huge gap in priority between the two options.) The most common concern related to Simple Ratings is that every item will wind up as important and/or results may end up as a tie. To reduce the probability of a tie in ratings, we recommend using response scales that are positively skewed toward importance.
| Example of a neutral scale (DO NOT USE IF ALL ITEMS WILL MOSTLY BE IMPORTANT)
Equal number of response options for important and not important | Example of positively skewed scale of importance (USE THESE IF MOST ITEMS WILL BE DEEMED IMPORTANT)
More response options provided to express the level of importance |
| Very important Somewhat important Somewhat unimportant Very unimportant
| Essential Very important Somewhat Important Not very important |
If the items really are equally important to respondents, it is important for policy makers to understand these priorities.