Against skill based matchmaking?

As a topic we’ve literally been writing about for over a decade, can there actually be any valid argument against using skill-based matchmaking in a competitive game?

I recently watched a couple of youtube videos about this, which discussed an apparently recent ‘debate’ online about it (although it was the first I had heard about it admittedly).

The first video was from Vesper Arcade who covers the reasons that he saw as valid arguments against what he defined as a specific style of “strict” skill based matchmaking vs what he calls “soft” – or in other words, normal or ‘traditional’ skill-based matchmaking:

It’s a long video and Vesper’s actual arguments get VERY muddled with a lot of other issues thrown in unfortunately, so here is my attempt to summarise his position and the points I actually agree with:

  1. Skill based matchmaking CAN be badly implemented in a game
    I could not agree more, after all, a huge amount of this site has been written about this!
  2. SBMM is badly implemented in some games is that it too quickly responds to your very recent ‘results’
    This is what he talks about when he suggests players bounce between higher and lower skill levels too quickly. I agree that is definitely something that could potentially be wrong in a particular game, although I have not experienced it yet myself. Historically in videogames this problem has generally been the other way around with cases where a player’s (perhaps distant) past performance is ‘dragging down’ their current rating too heavily (or even leaving them over-ranked in the case of a player returning to a game after months off for example). Whether real or perceived this is (partly) what some players describe as being “trapped in ELO hell” in some games and why requests for ELO resets etc happen. I generally fall on the opinion that relatively quick adjustments to players current skill levels, and/or things like seasonal resets, are good things though, but I agree this needs to be carefully positioned depending on the game in question and it’s certainly possible to get this wrong.
  3. SBMM is badly implemented in some games in that the skill levels it chooses to match within can be set at a bad ‘strata’ for certain players
    This is what he means when complaining you reach a certain ‘skill level’ of ‘slightly over average’ and suddenly the MM pool widens to include up to the ‘pro players’. Yes I agree this can definitely be done wrong. He likens this to for example if SFV “Gold” rank players could get matched vs “Warlord” rank – which of course SFV does not allow (in it’s Ranked mode at least, except it has many other big issues we’ve covered before), but it’s also something that SFV gets wrong (or, at least has in the past) too in my opinion & experiences.
  4. The wrong ‘results’ are used to determine your skill level
    This is mostly about using various in-game statistics (such as a kill:death ratio) to determine your skill level, and not (just) taking the win/loss result into the equation. I’ve talked about this before too and I do think it’s incredibly risky to ever base a skill calculation on any in-game stats rather than win/loss. In some free-for-all style of games stats maybe more applicable (as they are essentially a leaderboard of results for a match anyway), and there could be some other types of games where using stats to give an adjustment to a players calculated skill in conjunction to win/loss could potentially be useful. But the risks in doing this are extremely high, especially if players are aware of these calculations and have any way to manipulate them, so it is generally better to avoid these ideas and stick to winning and losing to judge player performance.

However NONE of these are arguments against skill based match making itself. They are simply arguing about the best way to do it well, and the ways it can possibly go wrong and be done badly.

So the whole thing about complaining ‘against’ skill based matchmaking is fundamentally wrong and a misnomer (at least for Vesper), and attempts to define it between strict or soft also don’t really make sense to me – to make a clear argument you need to discuss these points about how it can be done well or badly, on a case by case basis, as above. If you don’t, it is just muddying the issue and confusing everyone, and I suspect half the people complaining for or against “SBMM” are not even arguing about the same thing at all and why this has caused such a furore on twitter (it appears) where in depth discussion is so stifled.

Also, his video also confuses additional points about:

  • Uneven playfields in competitive games are a problem (ie: grind to win/unlock, pay to win etc)
    Yup, absolutely. Such designs ruin competitive gaming as we’ve said here since day 1, and also makes SBMM much harder to implement and more confusing (& feel bad) for players, but this is only tangential to the matchmaking itself.
  • No visible (and/or effective) ranking method is a problem, especially for competitive players
    Yup absolutely, again, see half of the things we’ve written about here! Having nothing to go on but pages of stats or a “Kill/Death ratio” to judge your own skill level in a team based FPS game is absolute garbage, no doubt. But again, this has nothing to do with SBMM, it’s just another thing that games can do well, or badly at.
  • Offering players a choice whether they want SBMM or not can be good
    This is definitely a fair point to make, and as he points out, many games do it via some kind of Ranked mode vs Unranked modes of play. However most games still trend towards having some kind of ‘hidden’ SBMM even in ‘unranked’ mode which I also still believe is a good idea. In fact, you just need to look at the way Fantasy Strike offers this for what I would consider an absolutely perfect implementation (one which as discussed before, was exactly what I asked for in FS!). The player can choose to play Casual or Ranked mode firstly. In Ranked, they only ever get matched to other people close to their current ranking (which also resets seasonally, for anyone who feels ‘unfairly’ ranked, or who want to see their Rank with a new team of characters etc). In Casual however, the player can choose if they want to be matched only with players close to their skill or not. Note that even if they choose to potentially be matched outside of their skill range, the game will still try to find the best match it can do in terms of skill (& connection). The game also tracks players skill (purely on win/loss results too) with each character separately, so there is no issue of the matchmaking being wrong because of a player that is really good with one character but weak with another in Fantasy Strike (why oh why can’t other FGs do this?). The most important result of this system is that a match of a pro vs a noob is still extremely unlikely, and what is most important is that it can ONLY happen if both players already agreed they were okay with it, by both having selected “match me with anyone”. FS’s method, to me, is the best of all worlds in offering players their choice of matchmaking.
  • Having ‘daily quests’ and goals outside of “trying to win” messes things up
    Yup, this can and usually is an issue in a competitive game, especially if there are no “unranked” modes as such for players to do ‘alternate play’ of whatever kind.

I next watched Sajam’s video about the same topic. Whilst there are a lot of jokes and “meme” language used here about ‘sweatlords’ and so forth, I do agree with Sajam on just about every actual point he makes. The most important one being exactly what I am saying here – that just because BAD skill based match making is possible, that doesn’t mean skill based matchmaking is bad in itself – he uses the lovely analogy of a restaurant that serves bad fries – it doesn’t mean all fries are bad! 😀 Sajam pretty much lays out a very agonistic view of this ‘issue’ overall that would definitely be at home here at Agoners:

The one very complex point that Vesper does bring up near the end of his video is the (future) concept of “engagement optimised skill based matchmaking” whereby the matchmaking is used to attempt to manipulate the players overall experience of the game to try to prevent win or loss ‘streaks’ for example or giving them “easier matches to start with but getting harder the more they play that day”. I definitely feel very conflicted thinking about this, and I will admit I do not really know how and where I stand on this. I would likely need to experience it first-hand to form a definitive view on such a complex matchmaking issue, however I do think it is extremely ‘risky’ to implement this, and it is definitely the kind of thing (like using stats rather than win/loss to determine skill) or attempts to “force” a 50:50 win/loss rate on players that can easily backfire and actually leave players with a worse experience than if you hadn’t attempted to influence this.


For once I’ve avoided putting reams and reams of in-line links into the text above because there’s just too many previous posts made here that I could link to, as so much of this site over the last decade+ has been dedicated to these discussions. However here’s a few key links if you would like to do “further reading” on my perspective on this:

Competitive Gaming 123 – the fundamentals of all of this!

My ‘competitive gaming review’ of Street Fighter V and how it’s matchmaking & ranking systems are decent but flawed.

My tips article for Fantasy Strike where I also cover some important things about how it’s matchmaking works.

Posts tagged with “matchmaking”.

Posts tagged with “ranking systems” .

Thanks for reading. If you know of any other good videos or articles about skill based match making that you think I may have missed, please do let me know in the comments or on twitter or youtube etc.

6 thoughts on “Against skill based matchmaking?

  1. Got a great discussion with Vesper and Mir going in their youtube comments. I’d like to thank both of those guys for reading and responding 🙂

    Some more videos about this I found related to CoD:

    But there is nothing there to suggest that there’s anything other than a normal good skill based match making system at work there (but it is from 2019, maybe it changed?). Hidden MMR is a fine way to do a skill based matchmaking system!

    And this video:

    Where my comment is that his video is mis-titled! None of the argument is against skill based match making itself. He nails it here: https://youtu.be/KMt5lCqIkik?t=401 (and right at the end too).
    The argument is really about the need for accurate RANKING in a game. This is an absolute given in any proper competitive game, everything else discussed is just symptoms of that problem and it has nothing to do with how the matchmaking works.
    This video should really be titled: “Why not having a ranked mode with accurate skill rankings is a HORRIBLE idea for a competitive game” !
    (It’s also why I would never play a game like CoD unless it got one)

    Like

  2. Interesting read.

    In regard to a forced 50% win rate: there is a tangible benefit to the company doing this. If it can be done in a not so obvious way players will win as much as they lose and are less likely to get so salty at the game that they stop playing altogether. At least in theory. There is no short supply of folks complaining about Blizzard using this style of match-making. Once people realize that the hands of fate (sort of) are determining their games before they’ve even played them it can really take the wind out of their sails. Losing because you got outplayed sucks, but there is a learning opportunity there. Losing because a machine decided you had almost no chance of winning and decided it was time for you to lose isn’t a great experience and winning because of the same system will have you questioning if your victory was hollow or not.

    Personally I don’t see the issue in SBMM, but I think it’s a lot easier to do in 1v1 games versus team games. But I’m also a firm believer in practicing with people who are well above and well below your skill level, so what do I really know? 😛

    Liked by 1 person

    1. Thanks Frosti 😀

      Yup the forced 50% thing I’ve discussed a lot before. It’s not necessarily a “bad” thing overall, but for the same reasons you state, I’ve said it is a “risky” thing as it can be a very negative experience at times if you go too hard on matchmaking to try to force a result. Better to have general skill bracket leagues to match within – exactly like FS does.

      Agree with your practice ideas and again, what I always like the best is to give the players the most choice over their MM settings (whilst still retaining as few queues as possible). Basically FS is exactly what I’d want from a match making system, especially since they implemented =exactly= what I asked for long ago 😀 hahaha
      btw I will say there is still a bit of room to debate some areas of it all in FS btw, but it’s extremely minor details, not the overall concept of how it works.

      Liked by 2 people

  3. Further discussion about Engagement Optimised Match Making (EOMM) with Vesper and Mir was really interesting. They sent me these documents regarding it:

    https://recsys.acm.org/wp-content/uploads/2017/10/RecSys-17-Industry-Presentation-EA.pdf
    http://web.cs.ucla.edu/~yzsun/papers/WWW17Chen_EOMM.pdf

    Note: These are currently only proposals right now though and not what anyone currently understands any game to be using.

    I’m not going to pretend to fully understand all the mathematical modelling there but it all makes sense, however I can immediately identify an important variable they have missed & I believe how this EOMM can backfire because of that variable – which is the players “belief” of what MM model they are playing under.
    ie: their churn rates are for players playing in a system they BELIEVE is giving them fair skill based mm
    and I believe those churn rates, which the EOMM model are based on, will change if the players understand and/or believe they are playing in an EOMM model.

    Mir countered that
    “they leverage the fact that most people don’t know what is going on behind the scenes, and by that I mean they barely even care, if at all, so even if a small percentage notices it’s just a few people compared to the overwhelming majority who will feel happy and buy more DLC”

    Which I agree with, but feel perhaps the internet era of rapid information spread might still affect that in the case of a game applying EOMM and a high % of the player base knowing about it… but Vesper and Mir countered that
    “the only people who care are the dedicated/above average so a small part of the community”
    “also, you underestimate how uninterested the people they’re catering to are”

    Which is a fair if depressing point!

    Like

  4. Here’s another handy reference document about TrueSkill2 that Vesper dug up, which is a newer, modified version of (the familiar to me) TrueSkill even further adapted for team games, especially regarding instances where players play in friend ‘squads’ within the teams, adapts more for quit-outs and offers input from in-game performance stats rather than just win/loss results:

    https://www.microsoft.com/en-us/research/uploads/prod/2018/03/trueskill2.pdf

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.