Game Deduplication

General Quality and Filtering Criteria

In addition to the algorithmic detection of true duplicates by my Rust scripts, Gigabase applies strict quality filters during creation to ensure the database is relevant for serious training:

Minimum length: Games that end after fewer than 10 half-moves (5 moves per side) are consistently excluded from the database. This reliably filters out abandoned games and disconnections, particularly in online play.
Skill Level Filter (Online): The online database includes only games in which both players have an ELO rating of at least 1800. This ensures a suitable level of play and significantly reduces the file size, thereby improving performance.

How Does Chess Game Deduplication Work?

Imagine you’ve collected a huge library of chess games from all over the place – websites, databases, tournament records. A lot of these games are duplicates: they describe the exact same game, just from different sources.

The PGN Deduplicator is a tool that finds and removes these duplicates – so you end up with a clean collection where every game appears just once.

Why Do Duplicates Exist?

Scenario 1: Different Websites

The same game is published on Chess.com, Lichess.org, and Chess24.com. All three have identical move sequences.

Scenario 2: Different Databases

A championship game is imported into multiple databases:

Chess.com Historical Database (2010)
Lichess Opening Database (2018)
Chessbase Online (2023)

Scenario 3: Incomplete Copies

A game was entered in two different versions:

Version A: Only the first 20 moves (maybe the rest wasn’t documented)
Version B: The complete game with all 40 moves

These aren’t identical, but Version A is actually part of Version B – and should be recognized as a duplicate.

The Three Types of Duplicates

Type 1: Exact Duplicates

Definition: Two games with the exact same sequence of moves.

How is this detected? The system converts all moves into a standard format and compares them. If two games have identical moves, they’re exact duplicates – no matter what notation style was used.

Example:

Game A (Lichess Export):
1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Be3

Game B (ChessBase Version):
1.e4 c5 2.Sf3 d6 3.d4 cxd4 4.Sxd4 Sf6 5.Sc3 a6 6.Le3

Even though different piece symbols show up (Nf3 vs. Sf3, Nxd4 vs. Sxd4), these are exact duplicates – just written in different languages/standards. The system spots this automatically.

What happens during export? The system keeps the longer or higher-quality game as the “master” and marks the others as duplicates. The best information from all copies is merged (e.g., event name, player Elo ratings, source quality).

Type 2: Subsumption Duplicates (Incomplete Copies)

Definition: One game is the beginning of another game.

How is this detected? The system systematically searches for games where the moves at the start match a longer game exactly.

Example:

Game A (from online database, incomplete):
1.d4 d5 2.c4 e6 3.Nc3 Nf6 4.Bg5 Be7 5.e3 O-O
(End – 10 half-moves)

Game B (complete tournament game):
1.d4 d5 2.c4 e6 3.Nc3 Nf6 4.Bg5 Be7 5.e3 O-O 6.Nf3 h6 7.Bh4 b6 8.cxd5 exd5
(32 half-moves total)

Game A is completely contained at the start of Game B – the first 10 moves (20 half-moves) are identical. Therefore, Game A is a subsumption duplicate. Game B is the more complete version and will be used as the master.

Why does this happen?

Different sources have different levels of completeness
Game endings were documented differently
Analysis fragments got saved as separate games

What happens during export? Game B gets exported, Game A is marked as a duplicate.

Type 3: Fuzzy Matching (Similar Games)

Definition: Two games have almost identical moves, but with small differences.

How can this happen?

Different interpretations of ambiguous moves
Different notation standards
Typos in data entry
Slightly different variations

Example:

Game A (Chess.com Version):
1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Bg5 e6 7.f4 Be7 8.Qf3

Game B (Lichess Version, slight variation):
1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 6.Bg5 e6 7.f4 Be7 8.Qe2
(differs at move 8: Qf3 vs. Qe2)

The system spots that these games are very similar and marks them as fuzzy-match duplicates. The differences are tiny (just one move).

What happens during export? The longer or higher-quality game is exported as the master. The similar game can optionally be appended as a variation of the master.

Handling Different Player Name Spellings and Unknown Players

The Problem: Player Name Variations

The same player can be spelled differently in different databases:

Chess.com: "Carlsen, M."
Lichess: "Carlsen, Magnus"
Chessbase: "Magnus Carlsen"

Even though the name is different, these are the same games!

The Solution: ELO Identity Resolver

The system uses three smart strategies:

Phonetic Matching: Similar-sounding names are grouped together (e.g., “Carlsen” → phonetically identical)
FIDE-ID Matching: When a FIDE-ID is available, players are uniquely identified by it
ELO Identity Resolver: Special logic that recognizes players by their ELO rating pattern and finds duplicates even with different names

Practical Example:

Game A: [White "Carlsen, Magnus"] [WhiteElo "2865"] 
        1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 ...

Game B: [White "Carlsen, M."] [WhiteElo "2865"]
        1.e4 c5 2.Nf3 d6 3.d4 cxd4 4.Nxd4 Nf6 5.Nc3 a6 ...

The system spots that:

The names are different but phonetically similar
The Elo ratings match
The moves are identical → This is definitely a Carlsen game

Unknown Players: The Partial Subsumption Problem

The Problem: What if a player isn’t documented?

In PGN format, an unknown player is marked as ?:

[White "?"]
[Black "Kasparov, Garry"]

The Challenge: Two games might have the same opponent (Kasparov), but because of the unknown player, they might not be compared:

Game A: [White "?"] [Black "Kasparov, Garry"] → Moves 1.e4 e5 2.Nf3 Nc6 3.Bb5 ...
Game B: [White "?"] [Black "Kasparov, Garry"] → Moves 1.e4 e5 2.Nf3 Nc6 3.Bb5 a6 4.Ba4 ...

The Solution (Partial Subsumption): The system groups all games with unknown players by their known opponent and compares moves only within these groups. This way, subsumption duplicates are recognized even with unknown players – and Game A is correctly identified as the beginning of Game B.

The Process Step by Step

Phase 1-4: Preparation

The system imports your games and grabs information about players (names, Elo ratings, FIDE IDs) from a database. It uses multiple strategies to identify players correctly:

Phonetic Matching: Similar-sounding names are recognized
FIDE-ID Assignment: Official player IDs connect different name variants
ELO-Pattern Analysis: Players are recognized by their ELO progression/decline
Transliteration: Cyrillic, Hebrew, and other scripts are correctly normalized

Phase 5: Find Exact Duplicates

The system compares all move sequences and finds games with identical moves.

Phase 6: Unify Player IDs

Within duplicate groups, player IDs are made consistent.

Phase 7: Find Subsumption Duplicates

The system searches for games that are the beginning of other games.

Phase 8: Fuzzy Matching

The system searches for games that differ in only a few moves.

Phase 9: Merge Headers

The best available information from all duplicates is combined:

Best event name
Best Elo ratings
Best source documentation

Phase 10: Export

The results are written to a new, deduplicated PGN file.

Master and Duplicates

After the process, each group of duplicates is treated as follows:

Master: The best game is exported (usually the longest or the one with the most information)
Duplicates: The other games are marked as such and can optionally also be exported

What Gets Exported?

You can pick what goes into the final PGN file:

Masters Only (Default): Each duplicate group appears just once
With Variations: Similar games are added as chess variations of the master
Duplicates Only: Export only the games marked as duplicates
Filtered by Strength: Only games by players with Elo ≥ 1800

Quality Assurance

The system plays it safe:

It avoids false positives (games incorrectly marked as duplicates)
It does multiple checks to make sure games really are duplicates
When uncertain, a game gets marked as fuzzy-match rather than a definite duplicate

The Result

After deduplication, you have:

A collection without identical duplicates
All important information from all sources combined
Clean player IDs and Elo ratings
An overview of which games were duplicates

This is particularly valuable for:

Chess research and analysis
Endgame tablebases
Opening repertoires
Training databases
Personal game collections from multiple online platforms
Historical game archives with differently documented player names