How it works
My Analytics Methodology
From raw win rates to insight accounting for player skill
How we pull the data
We ingest tournament data for all 5+ rounds & 20+ players and all ITC events. We can use a bigger data pool than a typical 40K stats website because we're going to account for what most sites are "proxying" for. Which is roughly "players at GTs bring competitive lists and are good players". We pull apart the actual lists that were played and by who (player, faction, primary detachment, detachment config, unit inclusion, unit config, costed wargear).
How we account for skill
The first thing we account for is player skill - 40K is fundamentally a very skill-based game. Yes, there are dice, yes there are list compositions and faction strength. But it's like comparing how a master League of Legends player playing full AD Garen vs. a bronze player with an optimal build on a skill-expressive champion like Azir and then concluding that Azir is underpowered and Garen is OP. Skill has to be stripped out first and foremost to draw any conclusions. The other thing within this is that certain factions have a higher skill ceiling (e.g. Aeldari/Drukhari) and others have a lower skill ceiling, others over-perform against lower-skill players who don't have their fundamentals locked down (e.g. Knights, Custodes).
How we look at detachments and hero datasheets
Beyond this, there is a structure to list building that's typically followed (outside of the rule of cool). Players look for "hero datasheets" - datasheets that punch above their weight and will typically do well regardless of detachment. Then look for which detachment they perform best in - this was very much the case with Defilers for the end of 10th. It wasn't the detachments that were inherently overpowered (they weren't causing problems before). But instead the raw datasheet power of the Defiler really shone through in certain detachments. We look at these factors first - we look for Primary detachments (3DP, 2DP and "triple 1DP" options) as well as Hero datasheets that really help a faction win.
How we look at detachment configs
Once we've got those, players are now choosing how to configure. They've chosen their detachment and possibly some faction hero datasheets. We then surface the next level in two separate strings (we do this to make our dataset as big as possible). The first one is Detachment config. For a 3DP, this is simple - there are no choices here, move on. For a 2DP, there are two configuration choices - which splash detachment to take and which force disposition. We analyse these two things together and surface the different "detachment configs" and their relative performance.
How we look at unit performance
Secondly is unit performance which we separate into a few options: Hero (covered), Detachment meta picks - they're not globally performing datasheets but they perform well in a specific detachment. Reliable - solid picks that help across the board but don't clear the bar to be called Hero. Neutral - decent inclusion rates, no clear signal either way (list filler/player preferences), Off-meta - they're included quite a bit but they're conclusively detracting from list performance and then niche (low inclusion, not possible to evaluate).
How we build the tier list
This all allows us to say how to optimise a faction (deconfounding for player skill) and there might be multiple builds for a faction - that we will surface. Each of these will be shown in our tier list with their raw win rate (as reported in the data) and our modelled win rate - with equally skilled pilots running high-performing detachments, configurations and lists with high-performing datasheets. This data doesn't exist - or at least it's very thin. How many top 5% players vs. top 5% players with fully optimised lists do we actually see in the data? A tiny amount, an amount so thin that it's statistically meaningless. Through our method, we can use every single data point deconfounded for player skill, faction, detachment, config and unit strength. This is how we then build the tier list, faction analytics and the matchup matrix.
A note on teams data
For predicting actual outcomes (win rates, force disposition strength and matchups) we exclude teams data. The matchup has 'data' we can't see - the pairings done by the captains. We can't see what they intended, crashing one matchup to deliver another. Having a specific tech list in their team that wouldn't have a chance at tournaments but has a really clear role on a team. These things mean the teams data is muddied and we can't use it for outcome prediction. However, we can still use it for unit strength analysis - and that's great because that's the most granular data-hungry layer. We can still draw conclusions about the performance of datasheets within factions and detachments.