Best Polyglot Opening Books

[Stoneface]

Have a folder and put into this some files (PGN file + polyglot.exe).

Open the Prompt of Commands and set the path to this new folder you created.

Example, imagine you made a new folder called Games in C:\YaY_Ultimate_3\Games. And in this Games folder you put the files polyglot.exe and your PGN file.

Then in Prompt you have to type ---> cd C:\YaY_Ultimate_3\Games , it will then look like this

---> C:\YaY_Ultimate_3\Games>

Now you ready to start making your bin books using your PGN file. Just need type the commands in your Prompt. Here are the commands:

1. Simple One (small bin book):

polyglot make-book -pgn MYGAMES.pgn -bin MYBIN.bin -max-ply 100

2. Multiple One (big bin book):

polyglot make-book -min-game 1 -uniform -only-white -pgn MYGAMES.pgn -bin w1.bin
polyglot make-book -min-score 50 -only-white -pgn MYGAMES.pgn -bin w2.bin
polyglot make-book -min-game 1 -uniform -only-black -pgn MYGAMES.pgn -bin b1.bin
polyglot make-book -min-score 40 -only-black -pgn MYGAMES.pgn -bin b2.bin
polyglot merge-book -in1 w1.bin -in2 w2.bin -out w.bin
polyglot merge-book -in1 b1.bin -in2 b2.bin -out b.bin
polyglot merge-book -in1 w.bin -in2 b.bin -out MYBIN.bin

3. Merge Bin Books:

polyglot merge-book -in1 MYBOOK1.bin -in2 MYBOOK2.bin -out NEWMERGEDBOOK.bin

Thank very much Supreme, i'll give it a try this way. I'll let you know if it works my friend.
Have a nice weekend

[Stoneface]

What's the meaning of that min score & does it stil work if i change that?
: min-score 50 -only-white
min-score 40 -only-black

For a small one the first option worked very good, thx my friends.
For information i've 2 pgn files 1with white 611 full games and 1with black 556 full games. So the complete games for white can't go in completely in my book for white cause it's to big, tried it allready.
So i think it's better to use the second option then to make 1big file for each of them i guess. Then i've 2 book of my own games in the past, White & Black! To merge it later that's probably to extremely big for those 1167games. By the way i don't think it will fit together cause i guess it will not give the best moves then if play on either the white or black side.


Thx in advance for the help guys.

[Waldemar]

1000 games are not enough to make a big big book hehehe, You can make an interesting book with more than 50,000 games.

[Stoneface]

1000 games are not enough to make a big big book hehehe, You can make an interesting book with more than 50,000 games.

Good to know my friend. Only that i played not that much games in the past but how it comes then that my games are not complete in it when do the first option? It go only till move 12 in a game of 37 moves when i played one of my games by clicking on the move in the Book-window in SCID. Did i do something wrong then?

Maybe must put the ply number a lot higher then i guess!? I created it with -max-ply 150 and with 239 but it stayed the same, max 12 moves when i did the moves from that particular game of mine. Better to use the 2nd option then I guess, but i don't want to merge my white & black pgn's for the moment. With the first option for small books the percentages were correct here from the moves i played in my games.

Does it make a difference if the number of the -min-score are other ones if use the 2nd option & what they mean or do with the book you create pls? Cause after i did for create my white book:
polyglot make-book -min-game 1 -uniform -only-white -pgn MyWhite.pgn -bin w1.bin (here i had 26470entries)
polyglot make-book -min-score 42 -only-white -pgn MyWhite.pgn -bin w2.bin (here i had 524entries)
polyglot merge-book -in1 w1.bin -in2 w2.bin -out MyWhiteRepertoire.bin (here it said skipped 524entries)
After this i placed "MyWhiteRepertoire" in the scid book file and tried it, but it showed only the first move from 5 different moves, that's all. They also had the same percentage instead the % according the openings i played in my games, all 5had 17% & no second move was evailable. Strange! The result was worse then made it for a small book..


Sorry that i ask many questions, i hope it not bother you guys to much.

Thx anyway though..

[Waldemar]

The commands to make the big book will give you same % to all your moves, different from the only one command to make the small book, where you have different % for each move.

If you use those 7 commands you will have a book like this e4 20% d4 20% c4 20% Nf3 20% b3 20% and so on to all moves in book, if you use that only 1 command for small book you have e4 80% d4 10% c4 5% Nf3 2% b3 2% for example.
I suggest you merge the small book with the big book, it will make a merged book with % from the small one but still big merged book.

BIN book is different from CTG book, when you make a CTG book you can add to book all all moves, but when you make BIN book you will not have all moves that a game has, thats why your book has some moves that end in 12.

[Stoneface]

The commands to make the big book will give you same % to all your moves, different from the only one command to make the small book, where you have different % for each move.

If you use those 7 commands you will have a book like this e4 20% d4 20% c4 20% Nf3 20% b3 20% and so on to all moves in book, if you use that only 1 command for small book you have e4 80% d4 10% c4 5% Nf3 2% b3 2% for example.
I suggest you merge the small book with the big book, it will make a merged book with % from the small one but still big merged book.

BIN book is different from CTG book, when you make a CTG book you can add to book all all moves, but when you make BIN book you will not have all moves that a game has, thats why your book has some moves that end in 12.

Can make with polyglot .ctg book also then?
Yes, but if make a big book the way i explained/did in those 3command lines in previous post(i only used those 3 cause didn't want to merge my white&black for the moment, they're in 2 different pgn files, white pgn has 611full games), then i've only 5different first moves,what is correct in MyWhiteRepertoire, but no second move to see at all after play one of them in scid, so it's not big at all cause it exist only about the 5 different first moves of my white repertoire . That i don't understand.

And about those -min-score for -only-white or black, thus that affect a book much maybe my friend ??
So probably for my 611 white full games i have in one pgn-listfile i can maybe better make it with the option for a smaIl book then & also for my 556 black full games in one pgn-listfile, it'll be only a book of 17kb large then if made with big file-option my book is 415kb but only contains those 5different first moves, strange! I think i've to find a good-complete tutorial to us commands for create .bin & .ctg opening books i guess

[matematiko]

To make a decent bin book you need at least 100,000 good quality games, preferentially engine vs engine games.

No, polyglot can not make CTG books.

The answer to all your questions are in that page I pointed out earlier:

Code: Select all

Book Making
-----------

You can compile a PGN file into a binary book using PolyGlot on the
command line.  At the moment, only a main (random) book is provided.
It is not yet possible to control opening lines manually.  I am
working on it though.

Usage: "polyglot make-book ".

"make-book" options are:

- "-pgn"

Name of the input PGN file.  PolyGlot should support any
standard-conforming file.  Let me know if you encounter a problem.

- "-bin"

Name of the output binary file.  I suggest ".bin" as the extension but
in fact PolyGlot does not care.

- "-max-ply" (default: infinite)

How many plies (half moves) to read for each game.  E.g. if set to
"20", only the first 10 full moves of each game will be scanned.

- "-min-game" (default: 3)

How many times must a move be played to be kept in the book.  In other
words, moves that were played too rarely will be left out.  If you
scan full games "2" seems a minimum, but if you selected lines
manually "1" will make sense.

- "-only-white" *** NEW ***

Save only white moves.  This allows to use different parameters for
white and black books, and merge them into a single file with the
"merge-book" command, see below.

- "-only-black" *** NEW ***

Same for black moves.

- "-uniform" *** NEW ***

By default, a probability is calculated by PolyGlot for each move
depending on how popular it is (how often it was playing in the
provided PGN file) and how much it "scored".  This option bypasses the
default mechanism and affects equal probability to all moves.  This
allows more variety of play.

This option is normally used only with hand-selected lines (e.g. "user
books").

---

Example: "polyglot make-book -pgn games.pgn -bin book.bin -max-ply 30".

Building a book is usually very fast (a few minutes at most).  Note
however that a lot of memory may be required.  To reduce memory usage,
select a ply limit.


Book Merging
------------

*** NEW ***

Usage: "polyglot merge-book -in1  -in2  -out "

Merge two bin files into a single one.   has "priority"; this
means that if a position is present in both input books, data from
 will be ignored for this position.

The two main applications are:

1) combine a white book and a black book (in which case priority does
   not matter)

2) combine a "user book" of manually-selected lines with a broader one
   from a large game set

What follows is an admitedly complicated example of how this can be
used.  DO NOT MAILBOMB ME IF YOU DO NOT UNDERSTAND!

My hope is that at least one advanced user will get what I mean and
writes a better explanation on a web page or forum thread (yes, that's
YOU, thanks by the way) ...

---

Imagine that we've got 4 PGN files as follows:

w1.pgn: fixed white lines, all moves manually checked
w2.pgn: selected games (for random book as with PolyGlot 1.3)

b1.pgn and b2.pgn: same for black

The first step is to build 4 .bin files with appropriate options.
Lines starting with "> " indicate what is typed on the command line.

> polyglot make-book -min-game 1 -uniform -only-white -pgn w1.pgn -bin w1.bin

I added "-uniform" because it allows randomness in the fixed lines
(e.g. d4+e4 at 50%).  It has no effect if lines are deterministic
(only one move for a given position).

"-min-game 1" is characteristic for user books.  All moves are supposed
to be safe so there is no reason to filter them with other heuristics.

> polyglot make-book -min-score 50 -only-white -pgn w2.pgn -bin w2.bin

This shows how min-score can actually be different for white and black
(as with multiple books).  I don't use "max-ply" because "min-game"
default value of 3 will limit depth somewhat.  You are of course free
to use it.

Same for black:

> polyglot make-book -min-game 1 -uniform -only-black -pgn b1.pgn -bin b1.bin
> polyglot make-book -min-score 40 -only-black -pgn b2.pgn -bin b2.bin

At this point we have 4 .bin files.  Notice that different parameters
were used for white and for black (not to mention that different PGN
files can be used).

---

Let's now merge the white books.

> polyglot merge-book -in1 w1.bin -in2 w2.bin -out w.bin

Input files are not symmetrical, "in1" has priority over "in2".

"skipped xxx entries." message from PolyGlot means there were some
position conflicts.  This is normal since we want to overwrite some
random moves with fixed lines instead.

Same for black:

> polyglot merge-book -in1 b1.bin -in2 b2.bin -out b.bin

Now we can finally merge the white and black books.

> polyglot merge-book -in1 w.bin -in2 b.bin -out book.bin

It's important to check that there are no conflicts, otherwise
something went wrong.

Note that this last operation was only made possible thanks to colour
filtering, otherwise nearly all positions would lead to conflicts.
For this reason, it does not make much sense to mix old .bin files
(which contain moves for both colours).

All these command lines might seem numerous and complicated but they
can be put together into batch files.

- "-min-game" (default: 3)---->Translation: This is the number a move had to have been played and that led to a win to be included in the book. Keep in mind that the bigger this number, the stronger the lines stored...but the book gets very small. In the other hand, the smaller this number, the bigger the book becomes but lines are not necessarily strong. Usually 2 or 3 is fine.

-min-score (default: 0.0)
Specifies the minimum score (or weight) this move should have received for it to be included in the book. Do not use this command unless you know what you are doing. And IMHO I do not think it brings much to the table, perhaps in special games or tournaments.

You are going to have to find several thousand more pgn databases and merge them into one (see Databases section in this FORUM and search the internet of course), many databases are corrupted and some games need to be deleted from them. To open and edit huge text file (that's what a pgn file is) you need text editors more powerful than Windows NotePad, we recommend: TextPad(Google).

There are many good tools out there to manipulate/fix pgn databases before they can be used in polyglot (do not worry, if polyglot does not like what it sees it will stop and spit an error).

Bin book making is not an easy task, we like to "initiate" people or give them a "jump start" if you wish, but we also want to see they take off on their own and find the answers in their own, specially for the easy and obvious questions.

Good luck

[Stoneface]

To make a decent bin book you need at least 100,000 good quality games, preferentially engine vs engine games.

No, polyglot can not make CTG books.

The answer to all your questions are in that page I pointed out earlier:

Code: Select all

Book Making
-----------

You can compile a PGN file into a binary book using PolyGlot on the
command line.  At the moment, only a main (random) book is provided.
It is not yet possible to control opening lines manually.  I am
working on it though.

Usage: "polyglot make-book ".

"make-book" options are:

- "-pgn"

Name of the input PGN file.  PolyGlot should support any
standard-conforming file.  Let me know if you encounter a problem.

- "-bin"

Name of the output binary file.  I suggest ".bin" as the extension but
in fact PolyGlot does not care.

- "-max-ply" (default: infinite)

How many plies (half moves) to read for each game.  E.g. if set to
"20", only the first 10 full moves of each game will be scanned.

- "-min-game" (default: 3)

How many times must a move be played to be kept in the book.  In other
words, moves that were played too rarely will be left out.  If you
scan full games "2" seems a minimum, but if you selected lines
manually "1" will make sense.

- "-only-white" *** NEW ***

Save only white moves.  This allows to use different parameters for
white and black books, and merge them into a single file with the
"merge-book" command, see below.

- "-only-black" *** NEW ***

Same for black moves.

- "-uniform" *** NEW ***

By default, a probability is calculated by PolyGlot for each move
depending on how popular it is (how often it was playing in the
provided PGN file) and how much it "scored".  This option bypasses the
default mechanism and affects equal probability to all moves.  This
allows more variety of play.

This option is normally used only with hand-selected lines (e.g. "user
books").

---

Example: "polyglot make-book -pgn games.pgn -bin book.bin -max-ply 30".

Building a book is usually very fast (a few minutes at most).  Note
however that a lot of memory may be required.  To reduce memory usage,
select a ply limit.


Book Merging
------------

*** NEW ***

Usage: "polyglot merge-book -in1  -in2  -out "

Merge two bin files into a single one.   has "priority"; this
means that if a position is present in both input books, data from
 will be ignored for this position.

The two main applications are:

1) combine a white book and a black book (in which case priority does
   not matter)

2) combine a "user book" of manually-selected lines with a broader one
   from a large game set

What follows is an admitedly complicated example of how this can be
used.  DO NOT MAILBOMB ME IF YOU DO NOT UNDERSTAND!

My hope is that at least one advanced user will get what I mean and
writes a better explanation on a web page or forum thread (yes, that's
YOU, thanks by the way) ...

---

Imagine that we've got 4 PGN files as follows:

w1.pgn: fixed white lines, all moves manually checked
w2.pgn: selected games (for random book as with PolyGlot 1.3)

b1.pgn and b2.pgn: same for black

The first step is to build 4 .bin files with appropriate options.
Lines starting with "> " indicate what is typed on the command line.

> polyglot make-book -min-game 1 -uniform -only-white -pgn w1.pgn -bin w1.bin

I added "-uniform" because it allows randomness in the fixed lines
(e.g. d4+e4 at 50%).  It has no effect if lines are deterministic
(only one move for a given position).

"-min-game 1" is characteristic for user books.  All moves are supposed
to be safe so there is no reason to filter them with other heuristics.

> polyglot make-book -min-score 50 -only-white -pgn w2.pgn -bin w2.bin

This shows how min-score can actually be different for white and black
(as with multiple books).  I don't use "max-ply" because "min-game"
default value of 3 will limit depth somewhat.  You are of course free
to use it.

Same for black:

> polyglot make-book -min-game 1 -uniform -only-black -pgn b1.pgn -bin b1.bin
> polyglot make-book -min-score 40 -only-black -pgn b2.pgn -bin b2.bin

At this point we have 4 .bin files.  Notice that different parameters
were used for white and for black (not to mention that different PGN
files can be used).

---

Let's now merge the white books.

> polyglot merge-book -in1 w1.bin -in2 w2.bin -out w.bin

Input files are not symmetrical, "in1" has priority over "in2".

"skipped xxx entries." message from PolyGlot means there were some
position conflicts.  This is normal since we want to overwrite some
random moves with fixed lines instead.

Same for black:

> polyglot merge-book -in1 b1.bin -in2 b2.bin -out b.bin

Now we can finally merge the white and black books.

> polyglot merge-book -in1 w.bin -in2 b.bin -out book.bin

It's important to check that there are no conflicts, otherwise
something went wrong.

Note that this last operation was only made possible thanks to colour
filtering, otherwise nearly all positions would lead to conflicts.
For this reason, it does not make much sense to mix old .bin files
(which contain moves for both colours).

All these command lines might seem numerous and complicated but they
can be put together into batch files.

- "-min-game" (default: 3)---->Translation: This is the number a move had to have been played and that led to a win to be included in the book. Keep in mind that the bigger this number, the stronger the lines stored...but the book gets very small. In the other hand, the smaller this number, the bigger the book becomes but lines are not necessarily strong. Usually 2 or 3 is fine.

-min-score (default: 0.0)
Specifies the minimum score (or weight) this move should have received for it to be included in the book. Do not use this command unless you know what you are doing. And IMHO I do not think it brings much to the table, perhaps in special games or tournaments.

You are going to have to find several thousand more pgn databases and merge them into one (see Databases section in this FORUM and search the internet of course), many databases are corrupted and some games need to be deleted from them. To open and edit huge text file (that's what a pgn file is) you need text editors more powerful than Windows NotePad, we recommend: TextPad(Google).

There are many good tools out there to manipulate/fix pgn databases before they can be used in polyglot (do not worry, if polyglot does not like what it sees it will stop and spit an error).

Bin book making is not an easy task, we like to "initiate" people or give them a "jump start" if you wish, but we also want to see they take off on their own and find the answers in their own, specially for the easy and obvious questions.

Good luck

Thx my friend, this is very good information. I was just reading in your polyglot tutorial here on the site. Now i know what i did wrong. the -min score 42 was defenetly to high i guess then.
Have a nice day further.

[matematiko]

Now that you mentioned it, I never saw the min-score thingy before, it is not in the link I posted, so it makes me think this option only exists in Michel's fork but not in Fonzy's fork.

Anyway, I fail to see what is the usefulness of this. Do not use it, once the book is made you can filter out moves with low score with BookThreshold.

Regards,

[Stoneface]

Now that you mentioned it, I never saw the min-score thingy before, it is not in the link I posted, so it makes me think this option only exists in Michel's fork but not in Fonzy's fork.

Anyway, I fail to see what is the usefulness of this. Do not use it, once the book is made you can filter out moves with low score with BookThreshold.

Regards,

Thank you, i was using it but with 0.0 default, that'll be the same then i guess. I go try it again now. It's a nice challenge for me. Never worked with polyglot and .bin files this way. Maybe it exist allready but i've almost all matches from Kasparov, Karpov & Max Euwe too in white & black pgn also. I'll try to make .bin's from them also, including those of my own official games in the past.

I not gonna use the command "-uniform" now for big files, maybe then it'll make different %percentages of the moves i've played, cause till now i only had different percentages of moves when made a small .bin book.


Kind regards my friends.

[Stoneface]

It worked out very well with my personal pgn games. Also in big books the percentage was good if i not used the "-uniform'" command and in my white, black & merged book all percentages were right till the end of my full games. Thx for the good explanation and my apology for the many questions my friends.
I'll send later .bin files if that's good for you. Take care & have a nice week though.!

Thx, kind regards,
Stoneface

[matematiko]

It worked out very well with my personal pgn games. Also in big books the percentage was good if i not used the "-uniform'" command and in my white, black & merged book all percentages were right till the end of my full games. Thx for the good explanation and my apology for the many questions my friends.
I'll send later .bin files if that's good for you. Take care & have a nice week though.!

Thx, kind regards,
Stoneface

Excellent and welcome to the exciting world of bin book making. Me personally never go back to CTG books. Also, H.G. Muller will in the future modify the evaluation algorithm and maybe the bin books will become even more powerful.

Regards,

[Stoneface]

It worked out very well with my personal pgn games. Also in big books the percentage was good if i not used the "-uniform'" command and in my white, black & merged book all percentages were right till the end of my full games. Thx for the good explanation and my apology for the many questions my friends.
I'll send later .bin files if that's good for you. Take care & have a nice week though.!

Thx, kind regards,
Stoneface

Excellent and welcome to the exciting world of bin book making. Me personally never go back to CTG books. Also, H.G. Muller will in the future modify the evaluation algorithm and maybe the bin books will become even more powerful.

Regards,

Yes, and as soon i understood that the -uniform command made the percentages equal i leave it simply out. I go play a little with it to find out how to make books were white or black wins. Also interesting i guess to create books like that.
I even don't know if there's much difference between .bin & .ctg books hehihheh. Anyway i like to use polyglot .bin books. Only pity that with Houdini in Arena it not play with his .bin book & i'm pretty sure that my settings are correct, cause with stockfish in Arena use his book.bin there.
But i like to play a lot with WinBoard & there everything works well!

[matematiko]

I hate Arena...not because is good or bad...but because they are not open to suggestions....they keep development very tight......those Germans want to control things their own way and I ain't going to be a part of it.....Also their ICS support sucks...!!! They cut ICS support for many version and now that they reinstated it is a disaster....as you know...YaY and YaYp depend on a reliable ICS protocol support.

Regards,

Also.....Arena books can only be created and played with Arena....CTG books and bin books can be used in many GUIs... you are better off without Arena...just IMHO.

Bye

[Prima]

Only pity that with Houdini in Arena it not play with his .bin book & i'm pretty sure that my settings are correct, cause with stockfish in Arena use his book.bin there.

The reason Houdini will not use bin books in Arena GUI or in any other GUI is because Houdini does not natively have its "Own book" UCI option. Therefore, placing any bin book (or any book format) in Houdini's directory will be ignored by Houdini.

On the other hand, Stockfish has its "Own book" UCI option which happens to be a PolyGlot bin format. It's book can be any PolyGlot bin book (re)named as Book.bin, which works with Stockfish under any GUI. Just remember to place both the engine and its book in the same folder or directory.