The Ultimate SCID Tutorial!! - Train With SCID, Shane's Chess Information Database
Author
ChessTok
Fellow Chess Enthusiast
Table of Contents
Table of Contents
So, your meager winnings from the local chess tourneys ain’t enough to
purchase ChessBase huh? Well lucky for you you don’t have to be a
celebrity the likes of Big Hickey to get the same quality education and
training. Meet SCID! Otherwise known as Shane’s Chess Information
Database. This miraculous piece of open source software has all the bells
and whistles you’ll need to get the job done. The price? A big fat
learning curve along with plenty time to set up and familiarize yourself
with this brilliant and feature-rich tool. Admittedly, a pretty steep price
depending on who you ask. But if you enjoy chess as much as I do this can actually be
kind of fun, and hopefully I can help you with that.
This tutorial will bring you all the way from installation to training for your opening prep in addition to a few bonuses like how to scope your
online opponents and smashing the local competition.
The three main requirements to get up and running with SCID are of course
SCID itself, a decently sized and hopefully representative database of
chess games ideally consisting mostly of master games, and a chess engine. The database
and the engine I’ll be using are Caissabase and Stockfish, respectively.
** If you found any of the following content useful I have sprinkled links to
chess.com throughout this article. If you use them to sign up for a free 7-day trial I get a small kickback. While SCID is free and worth every penny, the convenience you get from Chess.com Premium plans are unparalleled and I highly recommend signing up if you’re looking for an easy way to step your game up.
Caissabase is perhaps the largest curated and regularly maintained free chess database available to the public. To date it consists of ~ 5.6 million games
with all games having at least one player +2000 Elo. You can read more about it on their website. This man is a hero –>
http://caissabase.co.uk/
* If downloading from the original creator’s site please ignore the insecure site warning. For whatever reason this host hasn’t updated their SSL certificates.
Proceed to extract the contents to a directory of your choosing. You’ll need to access this later so remember where it is.
You are of course free to use any chess database you want with SCID including those from Chessbase, however this is not free. But surely there are creative ways to obtain it if you so desire.
I myself had a fairly difficult time finding openly free databases which also had enough games. If you know of any please point me in the right direction. All suggestions are welcome here.
For Windows you can download the mighty Stockfish engine by following the instructions on their official site:
https://stockfishchess.org/
I believe downloading the executable (.exe) file is all that is required and no “installation” is explicitly required via Window’s install wizard. However, for Unix-like systems the binary (essentially an executable command) gets installed. This piece of information will be more relevant during Part 2, the setup of the tutorial, so don’t worry too much about it right now.
For Mac you need a few more steps to get the engine only and not the app, that thing available on the Mac App Store. These are two different things. As per the official stockfish website,
https://stockfishchess.org/download/, you’ll need to install Homebrew, the Mac package manager to do this. It’s easy, so don’t sweat it. I’m here to help. Just open up your Mac’s terminal, Terminal. You can find this using Spotlight. Once open, copy/paste and run the following command in the prompt:
Afterwards, installing Stockfish is a cinch. In the terminal, once again, copy/paste and run the following command:
brew install stockfish
If you’re a Linux Chad 🐧 just use your respective package manager to install the engine:
Ubuntu example:
sudo apt install stockfish
Arch example:
sudo pacman -S stockfish
Again, as with the databases, you are welcome to install any number of engines you want. Stockfish just happens to be the most popular since it also happens to be one of the best.
For SCID itself there are actually two options, SCID and SCID vs. PC. For Windows and Linux either SCID or SCID vs. PC is fine. However for Mac you will need to download
SCID vs. PC. They’re practically identical and you can follow the rest of the tutorial with either installed:
Once you have SCID and Stockfish installed, fire up SCID and you’ll be greeted with something like this:
This can be pretty intimidating at first glance, but there are only a few things you need to get set up since you’ve already
taken care of the database and engine. That is, they both have to be loaded and configured. Piece of cake 🍰
By default SCID loads up the <clipbase>. In their own words:
In addition to the databases you have open, Scid provides a clipbase database, which is just like any other database except that it exists in memory only and has no files for permanent storage.The clipbase is useful as a temporary database, for merging the results of searches on more than one database or for treating the results of a search as a separate database…
You can read more about this if you go to Help–>Index–>Clipbase. In fact, the Help menu is your friend. Use it. You will absolutely need it
if you wish to master SCID.
If you haven’t already done so go ahead and unzip the Caissabase_2022_12_24.zip file first. Then, load in the database as a tree: File–>Open base as tree...
… search for the contents you unzipped earlier. You’re looking for the file ending in “.si4”. Select and open. You’ll notice the new docked window on the right:
What you’re looking at is a frequency distribution of the next move played from the current position displayed in order from most played to least played. This distribution is based on the database you just loaded. Notice that the most frequently played move on move 1. is e4. This is not surprising. Take note of the other statistics provided. I mostly pay attention to “Move”, “Frequency”, and “Score” columns. Score is confusing. If you look at the Help index:
The score is always computed from the White perspective, so 100% means all White wins and 0% means all Black wins. Scores are highlighted for moves that have good (green) or bad (red) results. On average a move should score 53.8% for white, highlighting appears if a move scores more than 3% better or worse than this average and if at least 15 games are contained in the database.
In other words, if it’s black’s turn in the position and the score is over 50% this is a bad move for black. Confusingly, highlighting does change depending on whose turn it is. To be even more
clear Score is absolute (from white’s perspective) and Highlighting is relative (the color, red or green, depends on whose turn it is). Keep this in mind.
Go to Tools–>Configure Engines and click on the New button towards the bottom left of the window that pops up. Name the engine whatever you want. I’ll call mine “Stockfish”. Next, if you’re on Mac Or Linux, fill in the Command box with “/bin/stockfish”. If you’re on Windows add the path to the stockfish executable file that you downloaded and unzipped. This will look something like “C:\Users\<username>\Downloads\stockfish-windows-x86-64-avx2\stockfish\stockfish-windows-x86-64-avx2.exe”.
You’ll also want to configure the engine to utilize your hardware close to its fullest capabilities in order to maximize the analysis depth and speed. Figure out how many CPU cores and how much RAM you have to work with. When running the engine I like to use the max number of threads minus one or two so my computer isn’t painfully slow when multitasking. I’ll then use about half my available RAM. These numbers are pretty arbitrary but the idea is to use as much resources as you can without your computer coughing up a lung.
With the configuration window still open make sure the UCI (Universal Chess Interface) box is checked and click on the Configure UCI engine button. Find the Threads text box. Most modern computer CPUs have hyperthreading, meaning that one core can handle multiple tasks. Usually it’s 2 per core. So, for my example I have 16 physical cores with 2 threads per core giving me 32 threads to work with. So 32 threads minus 1 for the rest of my computer gives me 31 threads for stockfish. This is what I’ll use.
Now go to the Hash memory text box. This is essentially the RAM you’ll dedicate to your engine. I have 64GB of RAM on my machine so I’ll just use 32. SCID assumes memory in Kilobytes so in my case I’ll put in 32000 for the number (32GB ~= 32000KB).
Last important thing is the Multiple variations text box. This is just the number of lines you want your engine to calculate at a time. I try to keep this number fairly low due to the engine being so resource hungry. 3 is a good number for me. This will give you the top 3 engine moves. If you want to find only the best move then just set this option to 1. You’ll notice that when you do this the depth will increase. The depth that an engine can calculate is inversely proportional to the number of lines it’s required to calculate.
Feel free to look up the rest of the engine configuration options from the Help menu. For me, the rest of the defaults are just fine.
After you’ve configured everything and confirmed all your selections you can finally load up the engine: Tools–>Start engine 1. A window with all your configurations will flash for a couple seconds and then the engine will automatically start. If you’ve properly maxed out your hardware you’ll hear your computer whir up like a small jet engine.
You should now have the following window:
SCID has an awesome feature from the Tree Window. Training mode. If you take a look at the bottom lefthand side of the Tree Window you’ll notice a Training checkbox.
When toggled on something very cool happens. First thing you might notice is that the moves listed in the frequency distribution table disappear. Second thing you might notice is that when you make a move on the board SCID makes a move. Third, if you continually try to scroll back further than one move from the move you just made SCID will keep responding with a single move according to the frequency distribution. What’s happening here? Well, exactly that. SCID is playing lines based on the frequency distribution of the database.
The way to use this to your advantage is by first setting up a starting position, say the beginning of an opening whose lines you want to commit to memory such as the Scotch, Scicilian, French, etc. Then toggle Training Mode and decide on a depth you feel comfortable learning to. Try to make the best moves from the distribution, namely those whose score is good for your side, B or W. SCID will respond accordingly. If you start to do really bad or there are no more moves in the distribution, untoggle the training mode and start over. Repeat until you reach your desired depth. You can also untoggle if you ever need to see which move to make because as mentioned previously they’ll be invisible in training mode. You will likely need to do this often in the beginning of training. Training like this will inevitably lead you to memorize the most frequently played lines and all their transpositions which is just as important if not more so. To be clear, SCID won’t just play the same line every time from the opening position but will rather play random lines based on their frequency. For example, if there are two moves available from the database from your current position each played with a frequency of 50% (say it’s 2. …e5 and 2. …c5 respectively) and you come to that position 100 times, well SCID will play 2. … e5 50 times randomly and 2. …c5 50 times randomly. And it’s as such for every subsequent position throughout the line.
Believe it or not there is another mode for training openings. This is accessed through Play–>Training–>Openings, but don’t go there quite yet. This one takes a bit more time to set up as you will need to build a “repertoire database”. But I got your back. Once set up this method is a little more convenient than the other. Openings here are properly isolated from the rest of the database and you will get a warning if you play a move not in the database. This mode also generates a report and keeps a log of stats which you can use to keep track of your progress as you learn. Over all it just makes for a smoother training experience in my opinion. Let’s get started.
* TIP - Before you follow through with the tedious method of building an opening repertoire from Caissabase I’ll mention that you can easily download opening databases from
here. If you choose to do that you can proceed directly from Step 1.3 and select your database you downloaded from the site. Keep in mind that you may still have to whittle down some of those databases to a manageable size.
Create an opening repertoire from Caissabase:
With the Caissabase loaded up create a position on the board for the opening you wish you study and toggle the filter to “show only games that matches the current board position”. From my experience you want to get the new filtered database to be relatively small (under 5k games) compared to the entirety of Caissabase because when you eventually load the repertoire your computer will struggle if it’s too large. For some reason the critical functions of the opening training mode are single threaded and computers tend to choke here. Reduce your opening repertoire database by going a little deeper into the opening you wish to study and taking note of the ratio of games displayed in the database window’s tab. Below is my starting position from a line in the Ruy Lopez. It has ~4.5k games.
Click on the book icon and select the Caissabase if it’s not already selected then right click and select “export all filter games”. Name the file something convenient to remember like the name of the opening.
Open the new pgn database that you just created: File–>Open–><name of your opening database.pgn>
Create an .si5 file database for efficiency: Database–>Copy All Games To New...–>New. Name the database. I like naming it the same as the .pgn file
Now open up the .si5 database you just created. This is the one you will be working with now. You can close the rest of the other databases by right clicking and selecting close (closing the window is not enough).
Click on book icon again in the database window, make sure your new database is selected, and then right click on it and select “Choose database icon…”. Pick either “openings for white”, “openings for black”, or “openings for either”.
Go to Play–>Training–>Openings–> From the window that pops up select “white”, “black”, or “both” (whichever you chose in step 1.6. and click on continue.
After the repertoire database is loaded this little window will pop up. Don’t abort. Just minimize it if it’s in the way.
You’re now ready to begin opening training mode. Just make your first move and SCID will, again, make moves based on the frequency distribution of the loaded database from that position. If you make a move not in the database from that position, SCID will warn you and move your piece back for you so you have another opportunity to play a “correct” response.
* TIP - If you’re ever confused about why your tree/games/database that you thought you loaded aren’t showing up check the Database tab and make sure the right database is selected. Additionally check the Window tab to see if your target window is checked off. Sometimes you’ll be working in a database that you didn’t intend and you’ll go in circles trying to figure out the problem.
In addition to loading up an opening repertoire I always load up an engine and dock it on the bottom to maximize the real estate. I’ll use the tree window, engine window, and also start up SCID in training mode. I’ll just keep the engine running in the background while working through opening lines paying special attention to the tree window and the suggested engine moves. You want to look out for any discrepancies between the frequency/score in the tree window as compared to the evaluation of the engine’s suggested moves. The best moves are not always the most frequently played. Sometimes the engines suggested best moves are not even in the database.
PART IV: Loading Your Own Games and Opponent OPSEC
#
As typical of SCID, there are several ways to load up and/or create a custom database. But for this one, I’ll refer you to a good little series on YouTube by creator Strategery. It’s a little dated but there’s still some great introductory info here:
Now that you know how to create and load up your own database, you might be wishing that you had an easy way to obtain and analyze your entire archive of games from
chess.com or lichess.org. Unfortunately
chess.com does not have an easy way to do this. Lichess.org does have this feature but since more people use
chess.com it’s likely that you and others will have substantially larger archives of games there.
Fortunately you can go here to download the entire database of any user from chess.com and lichess.org in pgn format :–>
openingtree
Now, you don’t have to stick to downloading just your own games or your friends’. You can download games of famous GMs, IMs, FMs, whatever your heart desires.
Once you download a user’s database of your choosing you can load this up into SCID and get to analyzing.
The tree window is extremely insightful in this scenario. Chances are that your typical opponent on
chess.com doesn’t have nearly the opening repertoire compared to a Grandmaster and you can exploit this to study some lines not frequently played by them. I will be writing a future tutorial on exactly how to do this so check back.
One thing I found very confusing with SCID is that options/configurations and windows can transpose into each other. Take for example the engine configuration. You can access the main configurations either from Tools–>Configure EnginesorTools–>Analysis Engine... and in fact the pop-up window looks identical. The only difference is that when you double click on an available engine from the Analysis Engine the selected engine starts up right away. Otherwise the functionality is exactly the same. You can even access the main configurations from the docked window from Tools–>Start Engine 1. In this case the configuration options display for a few seconds before the engine actually starts, but you can also access that same display from the hamburger menu on the bottom right corner of the docked window.
Ultimately you’ll notice that there are typically multiple ways to accomplish the exact same things and there’s no indication within SCID that one particular route leads to the same result. It can be quite frustrating at times and you will often be going through a bit of trial and error . The help menu doesn’t always seem to help much in this regard either.
Another thing I found annoying was that by default the settings/preferences aren’t permanent when changed. I would uncheck that silly “Tip of the Day” box assuming I’d never see it again. But sure enough every time I’d open up SCID there it was. Not sure why the first tip of the day isn’t “how to make settings automatically permanent”. It took me a while to realize that all changes you make have to be manually saved unless SCID is set to automatically save changes. So let’s change that and make it permanent 😉. Open up the Options menu and check on “Auto-save Options on Exit”
Such unexpected odd behavior is littered throughout SCID for whatever reason, but you just have to get used to it. Spend some time playing around with it to get familiar. Don’t be afraid to break anything and remember that the Help menu is your friend.
