Skill Ratings

The IMS Matchmaker is capable of using player 'skill' values to match players together for more balanced, competitive matches.

To use this feature, your game must use either:

• Use PlayFab authentication and store your player's skill data in PlayFab player data objects, or
• Use AccelByte authentication and store your player's skill data in AccelByte CloudSave data objects.

Te IMS Matchmaker will only read data from your chosen backend; your game server (or backend) will be responsible for updating these player data at the end of each match with new values depending on whether they win or lose.

Overview

The goal of any skill-based matchmaking system is to estimate each player's individual skill level within as few matches as possible, allowing players of like skill to be matched with one another.

Skill-based matchmaking relies on two values being stored and tracked for every user:

• mu - this value describes the measured skill of the player, such that two players with the same mu value are considered to be of equal 'skill'.
• sigma - this value describes the level confidence in the mu value, whereby a small value indicates high confidence, and a large value implies less confidence.

Every time a player wins a game, the mu value typically increases by a constant amount, indicating that their skill appears to be higher than previously measured. Conversely, every time a player loses a game, their mu value is typically decreased by the same amount.

The sigma value typically changes depending on whom the player was competing against. A player that consistently beats lower-skilled players and loses to higher-skilled values will see their sigma value reduce, as it indicates that the mu value is accurate. Conversely, when the player loses to a lower-skilled player, or beats a higher-skilled player, their sigma value will typically increase, indicating a decreased confidence in their mu skill rating.

We recommend using a default value of 30 for mu, and 10 for sigma, and using our rating calculation endpoint to calculate new values for players at the end of each match.

(For more information on skill-based matchmaking, we strongly recommend reading 'Computing Your Skill' by Jeff Moser.)

PlayFab Configuration

Your title should store a PlayFab Read-Only data object for each player, and this object will need to specify two numeric values - a mu (average skill) value, and a sigma (skill confidence) value.

The below Queue configuration assumes you are using a PlayFab data object named Ratings with the following JSON structure:

{
  "ranked": {
    "mu": 30,
    "sigma": 10
  }
}

Queue Configuration

Your Queue will need to specify two additional data blocks in order to first read the rating data (see dataSource), and second process it (see dataPoints).

{
  "name": "projects/speedy-racer-1999/queues/ranked",
  "id": "speedy-racer-1999.ranked",
  "title": "Solo Queue",
  "active": true,
  "simple": {
    "playerSettings": {
      "maxPlayers": 2
    },
    "startSettings": {
      "playersToAllocate": 2
    }
  },
  "dataSource": {
    "playFab": {
      "ranked_rating.mu": {
        "default": 30,
        "readOnlyDataKey": "Ratings",
        "valuePath": "ranked.mu"
      },
      "ranked_rating.sigma": {
        "default": 10,
        "readOnlyDataKey": "Ratings",
        "valuePath": "ranked.sigma"
      }
    }
  },
  "dataPoints": [
    {
      "config": {
        "musigma": {
          "path": "ranked_rating",
          "mu": 30,
          "sigma": 10,
          "beta": 5,
          "maxMuWeight": 2,
          "medMuWeight": 1,
          "ordinalDeviations": 3
        }
      },
      "extension": "musigma",
      "floor": 0.5,
      "name": "Skill",
      "weight": 1
    }
  ]
}

The dataSource.playFab keys pull the ranked_rating.mu and ranked_rating.sigma values from the user's PlayFab Ratings object via keys ranked.mu and ranked.sigma, respectively.

Note: the ranked_rating object keys within dataPoints must be named mu and sigma for skill-based matchmaking to work.

Defaults are also provided for the mu and sigma values - these will be used if the object or keys are missing (for instance, if this is player's first match.)

AccelByte Configuration

Your title should store rating data in a CloudSave AdminPlayerRecord for each player, and this object will need to specify two numeric values - a mu (average skill) value, and a sigma (skill confidence) value.

The below Queue configuration assumes you are using a CloudSave record named Ratings with the following JSON structure:

{
  "ranked": {
    "mu": 30,
    "sigma": 10
  }
}

Queue Configuration

Your Queue will need to specify two additional data blocks in order to first read the rating data (see dataSource), and second process it (see dataPoints).

{
  "name": "projects/speedy-racer-1999/queues/ranked",
  "id": "speedy-racer-1999.ranked",
  "title": "Solo Queue",
  "active": true,
  "simple": {
    "playerSettings": {
      "maxPlayers": 2
    },
    "startSettings": {
      "playersToAllocate": 2
    }
  },
  "dataSource": {
    "accelByte": {
      "ranked_rating.mu": {
        "default": 30,
        "adminPlayerRecordKey": "Ratings",
        "valuePath": "ranked.mu"
      },
      "ranked_rating.sigma": {
        "default": 10,
        "adminPlayerRecordKey": "Ratings",
        "valuePath": "ranked.sigma"
      }
    }
  },
  "dataPoints": [
    {
      "config": {
        "musigma": {
          "path": "ranked_rating",
          "mu": 30,
          "sigma": 10,
          "beta": 5,
          "maxMuWeight": 2,
          "medMuWeight": 1,
          "ordinalDeviations": 3
        }
      },
      "extension": "musigma",
      "floor": 0.5,
      "name": "Skill",
      "weight": 1
    }
  ]
}

The dataSource.accelByte keys pull the ranked_rating.mu and ranked_rating.sigma values from the user's AccelByte AdminPlayerRecord with the name Ratings via the keys ranked.mu and ranked.sigma, respectively.

Note: the ranked_rating object keys within dataPoints must be named mu and sigma for skill-based matchmaking to work.

Defaults are also provided for the mu and sigma values - these will be used if the object or keys are missing (for instance, if this is player's first match.)

Musigma Data Point

The Data Point block configures the Matchmaker to use the ranked_rating.mu and ranked_rating.sigma values provided by the configured Data Source.

Data Points extensions allow us to add behaviour to the matchmaker. In this case, we want to use the musigma extension, which do by using the following parameters:

• name - the name of this data block, we suggest something descriptive like Skill to indicate this block is responsible for skill-based matchmaking, as this will be emitted to the metrics backend.
• extension - enable support for skill-based matchmaking - this must be set to musigma.
• floor - the minimum score that this data point must return for the match to be accepted, and can be reduced to permit looser matching (and therefore shorter queue times) - 0.5 is a good starting value.
• weight - when using multiple data points, this determines how much influence skill-based matching has. (Since we're only defining one data point, a value of 1 is ideal.)
• config - the matching criteria to use, which are described in more detail in Tuning.

Here's an example with recommended values for the extension:

{
  "config": {
    "musigma": {
      "path": "ranked_rating",
      "mu": 30,
      "sigma": 10,
      "beta": 5,
      "maxMuWeight": 2,
      "medMuWeight": 1,
      "ordinalDeviations": 3
    }
  },
  "extension": "musigma",
  "floor": 0.5,
  "name": "Skill",
  "weight": 1
}

Tuning

The default values listed above are a good starting point, but you will want to adjust them through testing.

The musigma block nested within config is where most configuration is done:

• path - where to retrieve the mu and sigma values from - this should be ranked_rating if using the example provided in Queue Configuration
• mu - the default Mu value (skill rating) to use for players if one is not available - we suggest 30
• sigma - the default Sigma value (confidence rating) to use for players if one is not available - we suggest using a third of mu, i.e. 10
• beta - defines the maximum permitted skill disparity (i.e. difference in mu values) to allow - we suggest half of sigma, i.e. 5
• maxMuWeight - the weight to apply to the highest-skilled member of a party - a value above 1 will increase their influence on the party's estimated skill
• medMuWeight - the weight to apply to the median mu value of a party - a value below 1 will decrease their influence on the party's estimated skill
• ordinalDeviations - the modifier affecting how much sigma factors into final skill estimation for a team. Increasing this value will mean the matchmaker treats players with a higher sigma value as less skilled, thereby increasing the confidence that they are at that skill rating or better at the cost of potentially matching them against weaker players. We suggest starting with a value of 3.

We recommend not changing the mu value at all, but you may want to reduce sigma slightly if new players are being put into matches that are too challenging for them.

The beta value can be decreased to make the matchmaker favour players of more similar skill levels (although this may mean players have to queue a little longer), or increased to loosen the criteria (which typically means they queue for less time.) For a more detailed breakdown, refer to Beta Tuning.

If it is more likely for a highly-skilled player to 'carry' their party to victory, you may want to increase the maxMuWeight value above 1, so they have a greater impact on their overall party's estimated rating.

Finally, you may want to increase the ordinalDeviations such that parties with a low overall rating confidence are more likely to be matched with lower-skilled teams. This can help reduce the likelihood of new players (with default - and therefore typically higher - Sigma values indicating lower confidence in their reported skill rating) being repeatedly beaten by higher-skilled teams.

Match Finding

In order to generate matches, we first have to reduce the mu and sigma numbers for each teams into something that can be compared.

First, for each player we rate them within the range of possible skill levels they may be at based on the queue configuration. This is calculated simply by taking their current mu value and subtracting sigma times the queue's ordinalDeviations setting. For example, if a player has the default mu of 30 and a sigma value of 10, using the default ordinalDeviations setting of 3 would result in their final rating being 0 - in other words, the rating at which we are nearly 100% certain they are that skilled (or better.)

As players continue to play matches and their sigma value shrinks, eventually this will result in their adjusted rating being closer to their actual mu value. Players who queue individually are scored this way, while parties work slightly differently.

Players who queue together as a party are rated based on the queue configuration - queues can be tuned so that parties of players will be scored based on their best player (using the maxMuWeight setting) or their median player (using the medMuWeight setting). In the case of teams with an even number of players, the median player will be the average mu of the middle most skilled players. The weights for each setting are relative, so setting a 2-to-1 ratio of max to med weight would result the most skilled player representing 66% of the final base rating for the team, while the median player would account for 33% of the final value. Once a mu value has been estimated for the party, it is reduced by the average of the sigma values for each party member times the ordinalDeviations setting.

Once individual players and parties have been rated, the scoring function determines which players would be a good match for one another by finding individuals and parties whose rating delta is less than the beta setting for the queue. These individuals and parties are organized into teams based on the configured team size in the queue, which are then organized into matches.

Buckets

Typically, you will only match players together if they have skill values that are similar - however, you can configure the Matchmaker to be increasingly tolerant of higher skill disparities as time passes to prevent players from being stuck in the queue forever. This ensures that at busy periods when players of similar skill are more likely to be queuing, they will be matched together, but at less busy periods players can still get matched.

This can be configured using the maxBeta, bucket and bucketDuration keys:

{
  "config": {
    "musigma": {
      "path": "ranked_rating",
      "mu": 30,
      "sigma": 10,
      "maxMuWeight": 2,
      "medMuWeight": 1,
      "ordinalDeviations": 3,
      "maxBeta": 20,
      "buckets": 5,
      "bucketDuration": 10
    }
  },
  "extension": "musigma",
  "floor": 0.5,
  "name": "Skill",
  "weight": 1
}

Note how the beta value is absent and has been replaced by maxBeta, which will dynamically increase the beta value used by the calculation depending on how long a player has been queued for.

• maxBeta - the maximum beta value to scale up to.
• buckets - the number of steps in which the beta value will be scaled as time passes.
• bucketDuration - how long the first bucket will last, defined in seconds - subsequent buckets scale longer against a square exponential series.

In the above example, the 'beta' value used by the calculation (which defines the maximum skill difference permitted between players) will scale up to 20 in 5 separate buckets:

1. First 10 seconds - a beta value of 4
2. Next 40 seconds - a beta value of 8
3. Next 90 seconds - a beta value of 12
4. Next 160 seconds - a beta value of 16
5. Beyond 160 seconds - a beta value of 20

Note how the beta value increases over time, such that the matchmaker will try to match players with a similar skill initially, but widens the search criteria as time passes to increase the chance of the user being matched, albeit with players that may be of higher/lower skill.

Rating Calculation

When a match ends, it is the responsibility of your game server or backend to ensure that the skill ratings of players are updated depending on the outcome of the game and whether they were on the winning/losing team.

The IMS Matchmaker Developer API provides an endpoint that can perform this calculation for you.

You should issue a JSON POST request to the https://prd.matchmaker.os.i8e.io/v1/admin/rate endpoint, with your skill rating settings and a list of teams and players, ranked by their finishing position. For example:

{
  "config": {
    "modelId": "PLACKETT_LUCE",
    "beta": 5,
    "epsilon": 0.001,
    "mu": 30,
    "sigma": 10
  },
  "teams": [
    {
      "rank": 0,
      "team": {
        "teamId": "red",
        "players": [
          { "playerId": "player1", "mu": 35.0, "sigma": 5.1 },
          { "playerId": "player2", "mu": 32.1, "sigma": 2.9 }
        ]
      }
    },
    {
      "rank": 1,
      "team": {
        "teamId": "blue",
        "players": [
          { "playerId": "player3", "mu": 30.5, "sigma": 4.4 },
          { "playerId": "player4", "mu": 29.5, "sigma": 9.4 }
        ]
      }
    }
  ]
}

In the above example, the following parameters can be configured:

• modelId - the type of skill estimation model to use - currently only Plackett-Luce is supported.
• beta - the beta value to apply to the adjustment, which should match the value defined by the data point.
• epsilon - the minimum adjustment to make to sigma values regardless of the outcome of the calculation. This needs to be a small, non-zero value and is necessary to prevent sigma values from zeroing out.
• mu - the default Mu value to use, which should match the value defined by the data point.
• sigma - the default Sigma value to use, which should match the value defined by the data point.

Additionally, the teams block specifies the IDs and ratings of players before the match started, along with their respective ranks when the game ended. In this case, team red (rank 0) beat team blue (rank 1). Had the game resulted in a draw, both teams would have been given the same rank value.

The endpoint will return with a set of new ratings for each player, based on the Plackett-Luce algorithm when applied with the supplied values. You should update your player's data with these values, such that next time they queue for a match, their updated skill values will be used, and they'll be placed in a more appropriate game.

{
  "teams": [
    {
      "teamId": "red",
      "players": [
        {
          "playerId": "player1",
          "mu": 35.703050324698204,
          "sigma": 5.065653319815339
        },
        {
          "playerId": "player2",
          "mu": 32.32732230798585,
          "sigma": 2.8936994946797667
        }
      ]
    },
    {
      "teamId": "blue",
      "players": [
        {
          "playerId": "player3",
          "mu": 29.976699181616407,
          "sigma": 4.360939109491974
        },
        {
          "playerId": "player4",
          "mu": 27.111629116096374,
          "sigma": 9.012856163163935
        }
      ]
    }
  ]
}

Beta Tuning

One of the most important parts of the matchmaking algorithm that needs to be customized to each game is the beta value. Assigning this number correctly will help the score update function converge on player skill faster and more accurately while also helping avoid player assignment to matches they stand little chance of winning.

The beta value should reflect one standard deviation (or approximately an 80% chance of victory) based on player skill. This number can vary a lot based on how much of a factor skill is in your game, or even a particular match type, and really needs to be dialed in as you accumulate match data.

In order to be able to dial this number in, we recommend recording the results of matches along with player information that can be resolved into skill ratings. With this information in hand, we can create a simple scatter plot as follows:

1. For each match, approximate the rating for each team. This should be consistent with the same scale your default musigma values use.
2. Find the delta in rating between the teams.
3. If the favored (higher rated) team won, plot that as a 1 or, if they lost, as a 0.
4. Find the line of best fit for the graph, and check what delta mu the line intercepts 0.8 at - this should be a decent value for your beta.

The 80% chance of victory beta value is what should be used when calling the rating calculator - you may want to use a different beta for the actual datapoint configuration (possibly closer to the 50% value) to create more even matches.

FAQ

How should skill updates work with backfill?

There is no way to take into account a player who was dropped midway through a match when adjusting skill ratings, or new players being introduced. It is likely best initially to not update the skill of players in backfilled matches.

Is there a way to account for lopsided victories?

The basic skill calculation only concerns itself with who won, who lost, and how surprising that result was (based on the relative disparity between player skill). As a result a quick and decisive victory will produce the same results as a protracted and hard-fought one.

If you only use the maximum and median player skills for matchmaking, does that mean that the least skilled player is not considered at all?

For the purpose of estimating the skill of a team yes, but for the purpose of estimating our uncertainty about their skill no. The sigma value of the least skilled players are a factor in the party's average sigma value, which should affect their final rating.

When the matchfinding algorithm rates players and then filters them based on the configured beta, doesn't that mean that you could technically have a match including one player who is -beta and another who is +beta relative to the pivot player?

Yes, in theory this is possible - however, this would give the match a low score in the matchmaking function, and so it would only be made if no other matches were possible.