While I was writing my follow-up to my previous article about competitive gaming design using case examples, I got sidetracked into a very interesting debate on David Sirlin’s Fantasy Strike forums. I’ve linked it here, in case you want to read it all, but I will attempt to summarise the major points of it here.
David Sirlin (and one other poster on the site) were insistent that all Xbox 360 fighting games have functioning true-skill based matchmaking. Here’s a quote from Sirlin:
All these fighting Xbox games have… skill-based matchmaking. They really and seriously do. HDR does. SC5 does. Even shitty SF HF did. As far as I know VF, SF4, and MvC3 do too. That is, they secretly look at the skill-rating (not necessarily the same as the displayed stats, those can be sugar coated) to give the best match they can amongst people currently available to play.
Also backing this is a very old article I managed to dig up about MS Trueskill on XBox Live (although I argued this kind of detail is not the ‘common knowledge’ Sirlin purported it was!).
However the counter to this is my own experience, and the experience of every single person I’ve ever spoken to about it who plays fighting games, is that none of these games have effective skill-based matchmaking. They all give the appearance, and the results, of not even trying to match you against anyone even remotely close to your skill level. Some, like HDR, also fail to match you against an opponent with a decent internet connection to you. To quote myself:
In the 1000s of (ranked) matches I played on HDR I never detected it was managing any form of skill based selection or ping based selection. I’ve always had a seemingly random opponents from all-over the ping range and all-over the skill range of players at all times.
To check I’d not imagined others’ reported experiences I sent out a quick survey question out to all the FG players I could easily access across various online channels to see if any of them thought that any fighting games they played on the XBox gave them decent results with regards to skill-based matchmaking. The results were completely in-line with my view – the only fighting game anyone thought had any kind of functioning skill-based matchmaking was SF4.
So this poses the question to me about how to marry up these two seemingly diametrically opposed facts. Am I, and everyone else I know of, all poor deluded fools who’ve actually been getting great skill-based matchmaking out of our XBox fighting games all along and somehow not realised it? Or is there some other reason why, despite the attempts to matchmake, it isn’t working very well?
Sirlin himself gave two reasons:
1. Small size of online community means there’s no-one of the same skill level available to play.
2. FGs not skill rating people per-character, meaning people are not correctly skill-rated with whichever character they are playing.
And another point was raised that
3. There are many different ways & styles to be at a certain ‘skill level’ in a fighting game that don’t interact well that can lead to lop-sided matches (varying knowledge of character matchups for example).
I completely agree that these points will have a bad effect on skill-based matchmaking, especially the first two. However where I disagree with Sirlin is that these reasons are the be-all and end-all of the problems, and I believe these games still don’t do nearly enough to getting a successful matchmaking system working, because, as I said there:
I’ve always had experiences like getting one of the top few hundred players (with an unplayable 400+ping because they’re in Japan or Mexico) in one match, to getting someone who can’t even do basic moves or block in the next match, even when there’s likely to be 100s of people online.
So I still posit these reasons don’t go far enough to explain the complete disparity of results I, and anyone else I ask to experiment, seem to get. The types of results I see would be the equivalent to a StarCraft II match pitting me against Masters players and Bronze players with the same frequency, or a Halo Reach Arena matching me up against a full team of Onyx players. These things just never happen – so why does the equivalent type of thing happen on fighting games, even allowing for the reasons above by Sirlin. I think that the method of “matchmaking” used in most fighting games actually massively exacerbates and contributes to the problems, rather than alleviating them. I shall attempt to explain why.
Games like StarCraft II and Halo Reach use a system where all players waiting to play are placed in a “pool” and then they are match-made according to various criteria (including skill), and the actual host for the match is determined behind the scenes. This is what I would call actual proper matchmaking. The vast majority of fighting games instead just allow the individual player to decide whether they want to host or search for a match. The hosters manually setup matches, and the searches then manually select a match from a filtered list of results – which is why I’d call this kind of ‘matchmaking’ a “list filter” system. Alternatively a searcher can click ‘Quick match’ to instantly join what would have been one of their filtered list results. This is where I did not realise there was even any ‘filtering’ for skill level of opponent in play, given the results I was getting, whereas Sirlin has said there is. One big problem with the list filter method is that it will always return some matches if it can. It won’t report “zero matches” if there is no-one close to you in skill (and/or connection) it will just give you the best few matches it can find. Quick match is even worse, just putting you into essentially any match at all, even though it’s the best one it can find.
The problem is that the result of all of this means it is incredibly unlikely for two equally skilled players to ever find each other in a match, even with a large pool of players. I don’t even believe it scales well at all with a large community size, because this will just increase the number of searchers, especially those using ‘Quick match’ who will get dumped into any old game as soon as possible. Even Microsoft’s Trueskill documentation covers this situation on their FAQ (emphasis added):
…in off-peak hours or in situations where there are not enough host sessions available, the match quality can suffer and it may happen that you are getting matched with people of much higher/lower TrueSkill.
My point, because of the way the systems are set up within almost all fighting games with simple list-filter ‘matchmaking’, is that there will NEVER be enough host sessions available to filter against unless there is some huge incentive to actually host matches rather than search. Whereas instead, most FGs make it easier and more incentivised to search for matches. Match searchers are quickly grabbing any and all available “better matches” for any particular player (& also players searching get the annoying “that session is no longer available” message repeatedly too).
To explain this with an example, imagine there are 100 players online playing 1v1 StarCraft II matches on your server at a given time. Some will be mid-game, some will be in a queue waiting to play at the same time you are. If there are 10 players pretty close to you in skill and connection out of the 100, as long as one of them is in the match queue at the same time you are (perhaps the queue takes 1 minute?) then you will get matched together. The queue can wait for as long as the developers has deemed necessary to get a good chance at a good match for you; essentially waiting for one of those 10 to be available to play a new game. It could even scale this wait time based around the amount of people online, or it could even wait for ‘as long as it took’ to get a good skill match. As ‘bad’ as this might sound, compare it to the fighting game example where, to get matched against one of those 10 players who is similar to you, they have to be hosting at pretty much the exact same time you are searching (or vice-versa), and they have to have not already been matched with any one of the other 90 players coming free and hosting/searching before you happen to get to them.
I believe this problem, more than any other is the crux of the problems when it comes to matchmaking not “working” in fighting games – or any other game with a list filter matchmaking. This “list filter” system will only work as long as there are plentiful hosts and they remain as open sessions for some time, so it can work for large “lobby” style matches, but not for 1v1 games. In the FG system, host matches are immediately snapped up and filled by a single searcher, and so very few are ever available at any given moment in time. It could work better, if instead of closing matches as soon as they were filled, hosted ‘sessions’ stayed open and allowed 4 skill-matched players into a ‘room’. It could also work better if searches would report no results if there was actually no-one close in skill, rather than responding with the ‘next best’ matches which reduces the chances of similarly skilled players from ever finding each other even further. But really what would be far better would be to use the StarCraft II system and actually let the players build up in a pool before they are matched. Even better would be to allow players more control over how their own matchmaking works, tailoring it to their preferences as to whether they care about skill level or ping connection of opponents for example.
As noted above, the one fighting game that seems to have gotten around this effect successfully, despite using a “list filter” system is Street Fighter 4. Even playing the latest AE2012 version on the PC – with a relatively small community – it still tends to give better results than any other fighting game. The normal host & quick match systems on it are just as bad as other FGs, but the reason it works so much better is due to the “Fight Request” feature.This is essentially a special kind of hosting, with some major differences to the way hosting works in the above “list filter” examples. Firstly, you can continue to play computer opponents while you wait for a match. This is not only a immediate massive improvement for the player to allow them to play the game rather than just sit and wait, it also makes the whole system work better because there is now a good incentive to host, which creates more matches in the host pool. Secondly the fight requesting host player can decide if they want to allow close-skill opponents only, or any skill matches, or only higher skilled players, and they can also decide somewhat on what connections they want to allow by selecting local region only or worldwide matchmaking. Again, more incentives to host due to more control – and importantly, suddenly, if you care about skill matchmaking, there is now the possibility of players NOT being matched who aren’t close in skill. No matter how they go about it, searchers will not get a result of a fight request host who’s asked for close-skill matches unless they are close in skill, and likewise, the fight request player can go on playing the CPU for any amount of time, with no-one ever being matched with them unless someone with close skill searches for them. This is a far better situation especially for a new player, or a player striving to improve, regardless of the size of the community and the other problems with matchmaking in FGs listed above. Of course, it still isn’t the ideal situation – the one I’ve been writing about for years now, where you’d combine true “pool” matchmaking with some kind of fight request features as well, and gaining the best of both systems!