Check out Symmetric Chess, our featured variant for March, 2024.


[ Help | Earliest Comments | Latest Comments ]
[ List All Subjects of Discussion | Create New Subject of Discussion ]
[ List Latest Comments Only For Pages | Games | Rated Pages | Rated Games | Subjects of Discussion ]

Ratings & Comments

Later Reverse Order EarlierEarliest
Including Piece Values on Rules Pages[Subject Thread] [Add Response]
H. G. Muller wrote on Fri, Mar 15 08:52 PM UTC in reply to Kevin Pacey from 08:01 PM:

I don't keep close tabs on the development of Stockfish. But there are always many forks around, and sooner or later the best of each will be adopted into the official main branch. 2020 as the start of the NNUE mania sounds about right. And there might be hybrid versions around, which still relied in part on hand-crafted terms, added to the NN output to get the total score. I would expect this to have some advantages for terms like Pawn structure; it will be hard for a NN to extract Pawn-Structure info from King-Piece-Square tables. But it seems the latest Stockfish relies entirely on the NN.

It would be funny to test it on positions that it has certainly not seen in its training set, like 3Q vs 7N. It might be at a total loss for what to do. (Not thet the HCE did such a good job on that...)

 


Kevin Pacey wrote on Fri, Mar 15 08:01 PM UTC in reply to H. G. Muller from 06:47 PM:

Well, I knew the fellow might easily be wrong about Komodo. However, previously I had seen Stockfish used a neural network (at least to some extent) starting 2020 - unless that's more false stuff on the internet too (maybe a crusade for truth online could extend beyond chess variants, back to chess itself!?):

https://en.wikipedia.org/wiki/Stockfish_(chess)#:~:text=Starting%20with%20Stockfish%2012%20(2020,leaving%20just%20the%20neural%20network.

edit: it is a similar story as of 2020 for Komodo, apparently:

https://en.wikipedia.org/wiki/Komodo_(chess)#:~:text=On%20November%209%2C%202020%2C%20Komodo,networks%20in%20its%20evaluation%20function.


H. G. Muller wrote on Fri, Mar 15 06:47 PM UTC in reply to Kevin Pacey from 04:33 PM:

'In chess analysis, computer tools like Stockfish, Komodo, and AlphaZero help us know the importance of each chess piece during the game. They use calculations to assign a value to each piece based on factors like mobility, king safety, and board position...'(12 Sep 2023, Tato Shervashidze, Chess Coach...)

It is not only false, but it sounds like total nonsense to me. For one, AlphaZero is not comparable in any respect to Komodo or Stockfish; everything is different, and naming them in one breath already exposes the one who says this as completely ignorant on the subject of computer chess. (Which of course doesn't exclude he is a good Chess coach or has a high rating.)

In the past few years there has been a revolution in chess programming, after it had been converging to a method thought to be optimal for several decades. Initially programs were scoring positions at the leaves of a look-ahead search tree by a static (= not playing out any moves) heuristic that is now called a Hand-Crafted Evaluation. Piece values were a major part of that, often interpolated between 'opening' and 'end-game' values depending on the strength of the material still on board. The positional terms were Piece-Square Tables (accounting for mild general position dependence of piece values, without taking note of the location of other pieces, such as that Knights are poor at edges, and even poorer in corners), mobility (the actual number of moves a piece has in the current position), King safety (the number of squares around the King attacked by opponent pieces, and the value and number of these pieces), Pawn structure (passer advance, isolated / backward and doubled Pawns)

These parameters were never calculated (for orthodox Chess engines), but often were tuned. This was done by taking a large data set (like 500,000) of quiet positions from games with known result, and then tweeking all the bonuses and penalties (including piece values) that were used in the HCE until the calcuated evaluation score correlated best with the game result.

Than came AlphaZero out of nowhere, with everything completely different. It used a neural network for evaluation of positions as well as for guiding the search. This network simulates a brain with millions of cells, in some 40 layers, with tens of millions of connections between them. And they tuned the strength of those connections by having the thing play chess against itself. No one knows what each connection represents, but the result is that it eventually it could very accurately predict the winning probability for a position, apparently paying attention even to subtle strategic condiderations.

After that a hybrid form was invented: NNUE (for Easily Updatable Neural Network; no idea why they spelled it backwards...). This uses a conventional (unguided by any NN) search to calculate ahead, but at the end of each line evaluates by a NN of a peculiar design. It does not use explicit piece values, but calculates something very similar to Piece-Square Tables (which can be seen as a sort of piece values specified by location of the piece, and can simulate a plain piece value by specifying that same value on every square). Except that it does have such a PST for each location of the King. So the value of a piece cannot be dependent only on its absolute location, but also on how it is positioned relative to the King. (Well, this was invented for Shogi, and there proximity to the King is often more important than the intinsic strength of the piece type...). And it doesn't have one such a 64x64 table for each piece type, but 256 of them. And all these 256 values of each piece (on its current location, for the current King location) are than fed into a NN of 5 layers with 32 cells per layer, to combine them, until finally a single number appears at the output. This NN is then trained by tuning all the 256x64x64x6 values in the KPST, and the strength of the 4000 connections in the NN to reproduce the win probability of a huge data set of quiet positions, as good as it can.

This works, but after this no one knows what exactly the NN does. None of the values in the KPST in the optimally trained NN have the slightest resemblance to piece values as we know them. We cannot identify a King-Safety part, or a Pawn-Structure part, or a mobility part. It is just one totally integrated complete mess of totally meaningless multiplier parameters, that magically manage to conspire to give a very accurate prediction for who has the better winning chances in a given position. Stockfish and other strong engines now all use NNUE evaluation, (because they typically gain ~80 Elo compared to their original HCE), and the main development towards higher Elo comes from finding better sets for training it, or playing a little bit with the size of the NN. (Large NN can predict more accurately, but slow doen the engine, so that it cannot look as far ahead.)


Kevin Pacey wrote on Fri, Mar 15 04:33 PM UTC in reply to H. G. Muller from Tue Mar 12 01:16 PM:

Not to say I don't trust your post I'm replying to, H.G., but as they say, 'trust but verify'...

A not-too-old answer I saw when I Googled 'Does Stockfish use piece values', as found on 'Quora':

'In chess analysis, computer tools like Stockfish, Komodo, and AlphaZero help us know the importance of each chess piece during the game. They use calculations to assign a value to each piece based on factors like mobility, king safety, and board position...'(12 Sep 2023, Tato Shervashidze, Chess Coach...)

If that's true, such computers are actively doing 'calculating' of their piece values (rather than relying on e.g. statistical-studies-generated ones that are generalizations), on a position-by-position basis in a given game that they are playing.

That's also rather than by using piece values calculated before the start of any play whatsoever, say in the sort of way Betza tried to calculate fairy piece values (or my own cruder way(s) of estimating such values, i.e. in quick and dirty fashion).


Diceroller is Fire wrote on Wed, Mar 13 08:49 AM UTC:

Just off topic idea: what if Shogi Pawn will have a value of 1?


H. G. Muller wrote on Wed, Mar 13 06:51 AM UTC in reply to Kevin Pacey from Tue Mar 12 07:44 PM:

The problem with Pawns is that they are severely area bound, so that not all Pawns are equivalent, and some of these 'sub-types' cooperate better than others. Bishops in principle suffer from this too, but one seldomly has those on equal shades. (But still: good Bishop and bad Bishop.) So you cannot speak of THE Pawn value; depending on the Pawn constellation it might vary from 0.5 (doubled Rook Pawn), to 2.5 (7th-rank passer).

Kaufman already remarked that a Bishop appears to be better than a Knight when pitted against a Rook, which means it must have been weaker in some other piece combinations to arrive at an equal overall average. But I think common lore has it that Knights are particularly bad if you have Pawns on both wings, or in general, Pawns that are spread out. By requiring that the extra Pawns are connected passers you would more or less ensure that: there must be other Pawns, because in a pure KRKNPP end-game the Rook has no winning chances at all.

Rules involving a Bishop, like Q=R+B+P are always problematic, because it depends on the presence of the other Bishop to complete the pair. And also here the leveling effect starts to kick in, although to a lesser extent than with Q vs 3 minors. But add two Chancellors and Archbishops, and Q < R+B. (So really Q+C+A < C+A+R+B).


Kevin Pacey wrote on Tue, Mar 12 07:44 PM UTC in reply to H. G. Muller from 06:23 PM:

Grandmaster Nigel Short once told me, in so many words, that B+(2 connected passed pawns) generally beats R in an endgame. However, more generally, I have trouble believing N+2 pawns is even = to R, at least in endgames where the pawns are not all part of big healthy pawn island(s), which may be the average case in absolute reality.

The equation Q=R+B+P might seldom exactly hold true in a given chess position. As an observation we discussed long ago, sometimes a mixed bag of units that sticks [defensively] together well holds it own (at the least) vs. a Q, especially if she does not have the initiative (if either side does). However, my intuition tells me that Q is preferable to R+B+P in most cases that could ever arise, i.e. on average (maybe even more so than 2 minors outweigh R+P before an endgame on average), since games tend to open up, and that may favour the Q, for one thing (games often eventually opening up is sometimes given as a reason for thinking B>N on average).

So, a feather in Kaufman's cap here for finding the odd-looking value of the Q compared to R+B+P value. The only issue I have is, Q=R+B+P is such a darn useful/appealing rule of thumb for estimating the value of a Q in quick and dirty fashion, even in chess variants - such a fashion can serve players on CVP's GC while more accurate values are waiting to be found for the ever expanding number of variants played here.


H. G. Muller wrote on Tue, Mar 12 06:23 PM UTC in reply to Kevin Pacey from 05:43 PM:

Well, the values that Kaufman found were B=N=3.25, R=5 and Q=9.75. So also there 2 minor > R+P (6.5 vs 6), minor > 3P (3.25 vs 3), 2R > Q (10 vs 9.75). Only 3 minor = Q. Except of course that this ignores the B-pair bonus; 3 minors is bound to involve at least one Bishop, and if that broke the pair... So in almost all cases 3 minors > Q.

You can also see the onset of the leveling effect in the Q-vs-3 case: it is not only bad in the presence of extra Bishops (making sure the Q is opposed by a pair), but also in the presence of extra Rooks. These Rooks would suffer much more from the presence of three opponent minors than they suffer from the presence of an opponent Queen. (But this of course transcends the simple theory of piece values.) So the conclusion would be that he only case where you have equality is Q vs NNB plus Pawns. This could very well be correct without being in contradiction with the claim that 2 minors are in general stronger.

BTW, in his article Kaufman already is skeptical about the Q value he found, and said that he personally would prefer a value 9.50.

If you don't recognize teh B-pair as a separate term, then it is of course no miracle that you find the Bishop on average to be stronger. Because i a large part of the cases it will be part of a pair.


Kevin Pacey wrote on Tue, Mar 12 05:43 PM UTC in reply to H. G. Muller from 01:16 PM:

Chess piece values in beginner books (N=B=3, R=5, Q=9) are in fact little white lies to merely simplify their lives (other, unrelated, common white lies also exist - some are the fault of books simply being very old, and/or by poor authors). As you get more experienced/read advanced books, you are told/discover to generally not trade 2 minor pieces for R and P, at least not before the endgame. Similarly, you are told/discover to generally not trade 3 minor pieces for a Q. Also, don't trade a minor piece for 3 pawns too early in a game, as a rule of thumb.

World Champion Euwe, for example, had a set of piece values that tried to take all that into account, yet stay fairly true to the crude but simple to recall beginner values. His values were N=B=3.5, R=5.5 and Q=10 (noting that one thing beginner values get right is 2R=Q+P). Some of the problems of assigning piece values go away if you worry more about satisfying the advanced equations for 2 for 1 and 3 for 1 trades when thinking about such possibilities during a game (or as part of an algorithm).

Euwe did not bother to give a B any different value than N numerically, although he examined single B vs. single N cases in chapter(s) in a Middlegame Book volume (with co-author Kramer). Various grandmasters have historically given a B as having a [tiny] edge in value over a knight - some didn't pin themselves down, and wrote something like N=3, B=3+, the '+' presumably being a small fraction. Since I prefer Q=B+R+P=10, I have B=3.5 to keep that equation tidy, and have N=3.49 completely arbitrarily in my own mind (but generally leave it as 3.5 when writing a set of values, for the sake of simplicity).

To my mind, anyway, there may be a way I haven't mentioned until now to establish close to an absolute true value difference between B and N, if any, if enough decisive 2700+ games can ever be included in a database. For the wins and losses comparison, if you can somehow establish that having the B or the N was The decisive reason for the game's result, after an initial small error or two by the loser, that's the kind of decisive game that really matters. Yes, that raises the number of games you would need in such a database even way more. That's a theory, though again something impractical at present.


H. G. Muller wrote on Tue, Mar 12 01:16 PM UTC in reply to Kevin Pacey from 11:44 AM:

"Well-defined value" was used there in the sence of "universally valid for everyone that uses them". (Which does not exclude that there are people that do not use them at all, because they have better means for judging positions. Stockfish no longer uses piece values... It evaluates positions entirely through means of a trained neural net.)  If that would be the case, it would not be of any special interest to specifically investigate their value for high-rated players; any reasonable player would do. I already said it was not clear to me what exactly you wanted to say there, but I perceive this interest in high ratings as somewhat inconsistent. Either it would be the same as always, and thus not specially interesting, or the piece values would not be universal but dependent on rating, and the whole issue of piece values would not be very relevant. It seems there is no gain either way, so why bother?


Kevin Pacey wrote on Tue, Mar 12 11:44 AM UTC in reply to H. G. Muller from Sun Mar 10 09:34 PM:

I may be wrong, but I thought your first two paragraphs in this post of yours I'm replying to indicated that you thought pieces have a 'well-defined value'. Call me mistaken for thinking you meant that there is an absolute truth.


H. G. Muller wrote on Tue, Mar 12 11:29 AM UTC in reply to Kevin Pacey from 12:26 AM:

Systematic errors can never be estimated. There is no limit to how inaccurate a method of measurement can be. The only recourse is to be sure you design the method as good as you can. But what you mention is a statistical error, not a systematic one. Of course the weaker side can win, in any match with a finite number of games, by a fluke. Conventional statistics tells you how large the probability for that is. The probability to be off by 2 standard deviations or more (in either direction) is about 5%. To be off by 3 about 0.27%. It quickly tails off, but to make the standard deviation twice smaller you need 4 times as many games.

So it depends on how much weaker the weak side is. To demonstrate with only a one-in-a-million probablity for a fluke that a Queen is stronger than a Pawn wouldn't require very many games. The 20-0 result that you would almost certainly get would only have a one-in-a-million probability when the Queen was not better, but equal. OTOH, to show that a certain material imbalance provides a 1% better result with 95% 'confidence' (i.e. only 5% chance it is a fluke), you will need 6400 games (40%/sqrt(6400) = 40%/80 = 0.5%, so a 51% outcome is two standard deviations away from equality).

My aim is usually to determine piece values with a standard deviation of about 0.1 Pawn. Since Pawn odds typically causes a 65-70% result, 0.1 Pawn would result in 1.5-2% excess score, and 400-700 games would achieve that (40%/sqrt(400) = 2%). I consider it questionable whether it makes sense to strive for more accurate values, because piece values in themselves are already averages over various material combinations, and the actual material that is present might affect them by more than 0.1 Pawn.

I am not sure what you want to say in your first paragraph. You still argue like there would be an 'absolute truth' in piece values. But there isn't. The only thing that is absolute is the distance to checkmate. Piece values are only a heuristic used by fallible players who cannot calculate far enough ahead to see the checkmate. (Together with other heuristics for judging positional aspects.) If the checkmate is beyond your horizon you go for the material you think is strongest (i.e. gives the best prospects for winning), and hope for the best. If material gain is beyond the horizon you go for the position with the current material that you consider best. Above a certain level of play piece values become meaningless, and positions will be judged by other criteria than what material is present. And below that level they cannot be 'absolute truth', because it is not the ultimate level.

I never claimed that statistics of computer-generated games provide uncontestable proof of piece values. But they provide evidence. If a program that human players rated around 2000 Elo have difficulty beating in orthodox Chess hardly does better with a Chancellor as Queen replacement than as with an Archbishop (say 54%), it seems very unlikely that the Archbishop would be two Pawns less valuable. As that same engine would have very little trouble to convert other uncontested 2-Pawn advantages (such as R vs N, or 2N+P vs R) to a 90% score. It would require pretty strong evidence to the contrary to dismiss that as irrelevant, plus an explanation for why the program systematically blundered that advantage away. But there doesn't seem to be any such evidence at all. That a high-rated player thinks it is different is not evidence, especially if the rating is only based on games where neither A nor C participate. That the average number of moves on an empty board of A is smaller than that of C is not evidence, as it was never proven that piece values only depend on average mobility. (And counter examples on which everyone would agree can easily be given.) That A is a compound of pieces that are known to be weaker than the pieces C is a compound of is no evidence, as it was never proven that the value of a piece is equal to the sum of its compounds. (The Queen is an accepted counter-example.)

As to the draw margin: I usually took that as 1.5 Pawn, but that is very close to 4/3, and my only reason to pick it was that it is somewhere between 1 and 2 Pawns. And advantage of 1 Pawn is often not enough, 2 usually is. But 'decisive' is a relative notion. At lower levels games with a two-Pawn advantage can still be lost. GMs would probably not stand much chance against Stockfish if they were allowed to start with a two-Pawn advantage. At high levels a Pawn advantage was reported by Kaufmann to be equivalent to a 200-Elo rating advantage.


Kevin Pacey wrote on Tue, Mar 12 12:26 AM UTC in reply to H. G. Muller from Mon Mar 11 09:59 PM:

Well, 2700+ play could be of interest one day (that is, may matter, in spite of being dismissed as if that level of play should be treated in that type of [different, i.e. dismissive] way no matter what), if someone were to imply they know the actual difference, if any, between a B and a N on average for 8x8 [chess]. Computer study results, perhaps at times implying they have established the truth of piece values, are already posted all over the internet, not just here on CVP, where readers/players are all presently presumed to be sub-2700. Even thus, readers of CVP might care out of curiosity alone to know the ultimate truth, however established, even if they do not benefit by it very often, if ever, in their own games, unless they become [near-]2700+ themselves one day. A lot of people finding it interesting to know that 8x8 checkers has been weakly solved in modern times is somewhat comparable, perhaps.

That explanation of margins of error doesn't mention a thing or two that might go wrong if any assumptions are made at any step, such as assuming the presumed materially weaker side will never win the most games in a study [especially if only a few hundred games] by a possible fluke, even if unlikely.

Then, there is my own hypothesis that the larger in value/(more powerful) a piece is, the greater a certain margin of error might need to be within a study.

Perhaps unrelated(? - cannot recall if we discussed ever), 4 [uncompensated]tempi (worth 1/3rd of a pawn each in an open position), or 4/3rds of a pawn might be a normal minimum decisive edge, at least that's in line with an old-school rule of thumb I saw in an old book 'Point Count Chess', where a pawn is 3 points and 4 points ahead is supposed decisive (again, 1 pawn = 3 tempi in an open position is an old rule from even longer ago).


H. G. Muller wrote on Mon, Mar 11 09:59 PM UTC in reply to Kevin Pacey from 07:11 PM:

yet you're going right ahead yourself and saying 2700+ play is different

No, I said that if it was different the 2700+ result would not be of interest, while if they are the same it would be stupid to measure it at 2700+ while it is orders of magnitude easier around 2000. Whether less accurate play would give different results has to be tested. Below some level the games will no longer have any reality value, e.g. you could not expect correct values from a random mover.

So what you do is investigate how results for a few test cases depend on Time Control, starting at a TC where the engine plays at the level you are aiming for, and then reducing the time (and thus the level of play) to see where that starts to matter. With Fairy-Max as engine there turned out to be no change in results until the TC dropped below 40 moves/min. Examining the games also showed the cause of that: many games that could be easily won ended in draws because it was no longer searching deep enough to see that its passers could promote. So I conducted the tests at 40 moves/2min, where the play did not appear to suffer from any unnatural behavior.

You make it sound like it is my fault that you make so many false statements that need correcting...

Betza was actually write: the hand that wields the piece can have an effect on the empirical value. This is why I preferred to do the tests with Fairy-Max, which is basically a knowledge-less engine, which would treat all pieces on an equal basis. If you would use an engine that has advanced knowledge for, say, how to best position a piece w.r.t. the Pawn chain for some pieces and not for others, it would become an unfair comparison. ANd you can definitely make a piece worth less by encouraging very bad handling. E.g. if I would give a large positional bonus for having Knights in the corners, knights would become almost useless at low search depth. It would never use them. If you tell it a Queen is worth less than a Pawn, the side that starts with a Queen instead of a Rook would lose badly, as it would quickly trade Q for P and be R vs P behind.

The point is that the detrimental behavior that is encouraged here can never be stopped by the opponent. Small misconceptions tend to cancel out. E.g. if you twould have told the engine that a Bishop pair is worth less than a pair of Knights, the player with the Knights would avoid trading the Knights for Bishops, which is not much more difficult than avoiding the reverse trades, as the values are close. So it won't affect how often the imbalance will be traded away, and while it lasts, the Bishops will do more damage than the Knights, because the Bishop pair in truth is stronger. But there is no way you can prevent the opponent sacrificing his Queen for a Pawn, even if you have the misconception that the Pawn was worth more.

Note that large search depth tends to correct strategic misconceptions, because it brings the tactical consequences of strategic mistakes within the horizon. Wrecking your Pawn structure will eventually lead to forced loss of a Pawn, so the engine would avoid a wrecked Pawn structure even if it has no clue how to evaluate Pawn structures. Just because it doesn't want to lose a Pawn.

Statistical margins of error is high-school stuff. For N independent games the typical deviation of the result from the true probability will be square-root of N times the typical deviation of a single game result from the average. (Which is about 0.5, because not all games end in a 0 or 1 score.) So the typical deviation of the score percentage in a test of N games is 40%/sqrt(N). Having to calculate a square root isn't really advanced mathematics.


Kevin Pacey wrote on Mon, Mar 11 07:11 PM UTC in reply to H. G. Muller from 06:11 AM:

I'll agree using 2700+ level play for studies is impractical at this time. Earlier I almost did tell you to re-check something you posted saying my implying that 2700+ level play being different from 2300+ level play meant that I'd thought there were no absolute piece values - yet you're going right ahead yourself and saying 2700+ play is different (besides impractical to use for studies). I thought you'd just had some sort of an automatic reaction to try and say everything I write is wrong, and that you wrote inconsistently in place(s) unknowingly. I do not know if you were still trying to be fair. In any case, I get your overall drift, certainly as of this last post of yours.

One person who earlier wrote somewhere that 'the person with the hand that holds the piece' affects it's value was Betza, if you wish to argue with that, too. I personally believe the true piece values (for average case) should be absolute (however I think we might never be able to know them for sure). I still had a couple of other things about studies that I thought were suspect (margins of error, initial setup/armies chosen, as I wrote a bit earlier) that you didn't address, but I now recall we discussed those long ago here on CVP - it's just that I never was fully convinced.


H. G. Muller wrote on Mon, Mar 11 06:11 AM UTC in reply to Kevin Pacey from Sun Mar 10 10:29 PM:

That is a completely wrong impression. After a short period since their creation, during which all commonly used features are implemented, the bugs have been ironed out, and the evaluation parameters have been tuned, further progress requires originality, and becomes very slow, typically in very small steps of 1-5 Elo. Alpha Zero was a unique revolution, using a completele different algorithm for finding moves, which up to that point had never been used, and was actually designed for playing Go. Using neural nets for evaluation in conventional engines (NNUE) was a somewhat smaller revolution, imported from Shogi, which typically causes an 80 Elo jump in strength for all engines that started to use it.

There are currently no ideas on how you could make quantum computers play chess. Quantum computers are not generally faster computers than those we have now. They are completely different beasts, being able to do some parallellizeable tasks very fast by doing them simultaneously. Using parallelism in chess has always been very problematic. I haven't exactly monitored progress in quantum computing, but I would be surprised if they could already multiply two large numbers.

By now it should be clear that the idea of using 2700+ games is a complete bust: 

  1. it measures the wrong thing. We don't want piece values for super GM's, but for use in our own games.
  2. it does it in a very inefficient way, because of the high draw rate, and draws telling you nothing.

So even if you believe/would have proved piece values are independent of player strength, it would be very stupid to do the measurement at 2700+ level, taking 40 times as many games, each requiring 1000 times longer thinking than when you would have done it at the level you are aiming for. If you are smart you do exactly the opposit, measuring at the lowest level (= highest speed) you can afford without altering the results.

Oh, and to answer an earlier question I overlooked: I typically test for Elo-dependence of the results by playing some 800 games at each time control, varying the latter by a factor 10. 800 games gives a statistical error in the result of equivalent to some 10 Elo.


Kevin Pacey wrote on Sun, Mar 10 10:29 PM UTC in reply to H. G. Muller from 10:23 PM:

My impression was that periodically there is a leap in the strength of one engine someone is working on (e.g. AlphaZero), and then it outperforms other engines, say in 100 game matches, at least for a while, until the next such cycle begins.

edit: Chess.com is quoted by Google as saying AlphaZero lost 8 games to a version of Stockfish in a recent match, out of 1000 games, causing the loss of the match. Not many decisive results this time around, but wins are still possible at such a lofty level 'at the top' as we have right now, given enough games are played.

edit2: I haven't kept track of the progress quantum computing has been making, but that could lead to stronger engines and perhaps even open the door to solving chess.


H. G. Muller wrote on Sun, Mar 10 10:23 PM UTC in reply to Kevin Pacey from 10:02 PM:

The problem is that an N vs B imbalance is so small that you would be in the draw zone, and if there aren't sufficiently many errors, or not a sufficiently large one, there wouldn't be any checkmate. A study like Kaufman's, where you analyze statistics in games starting from the FIDE start position, is no longer possible if the level of play gets too high. All 300,000 games would be draws, and most imbalances would not occur in any of the games, because they would be winning advantages, and the perfect players would never allow them to develop. Current top engines already suffer from this problem; the developers cannot determine what is an improvement, because the weakest version is already so good that it doesn't make sufficiently many or large errors to ever lose. If you play from the start position with balanced openings. You need a special book that only plays very poor opening lines, that bring one of the players on the brink of losing. Then it becomes interesting to see which version has the better chances to hold the draw or not.

High level of play is really detrimental for this kind of study, which is all about detecting how much error you need to swing the result.

And then there is still the problem that if it would make a difference, it is the high-level play that is utterly irrelevant to the readers here. No one here is 2700+ Elo. The only thing of interest here is whether the reader would do better with a Bishop or with a Knight.


Kevin Pacey wrote on Sun, Mar 10 10:02 PM UTC in reply to H. G. Muller from 09:34 PM:

If you want a definition of near-perfect play, that still allows for the possibility of a (small) error or two, a very long game (that is well-played) that is a win for one side comes to mind.

You could decide to solve chess and it could be useful for determining piece values - just tag the number of moves a given 'game' of near-perfect chess takes to play until checkmate, and also keep track if B vs. N is involved. Optionally, you could have an engine assessing (albeit not perfectly accurately) who has the advantage (and how much) at every move. This of course is all not practically possible, in today's world at least.


H. G. Muller wrote on Sun, Mar 10 09:34 PM UTC in reply to Kevin Pacey from 06:49 PM:

Well, I looked up the exact numbers, and indeed hish threshold for including games in the Kaufman study was FM level (2300) for both players. That left him with 300,000 games out of an original 925,000. So what? Are FIDE masters in your eyes such poor players that the games they produce don't even vaguely resemble a serious chess game? And do you understand the consequences of such a claim being true? If B=N is only true for FIDE masters, and not for 2700+ super-GMs, then there is no such a thing as THE piece values; apparently they would depend on the level of play. There would not be any 'absolute truth'. So which value in that case would you think is more relevant for the readers of this website? The values that correctly predict who is closer to winning in games of players around 1900 Elo, or those for super-GMs?

Your method to cast doubt on the Kaufman study is tantamount to denying that pieces have a well-defined value in the first place. You don't seem to have much support in that area, though. Virtually all chess courses for beginners teach values that are very similar to the Kaufman values. I have never seen a book that says "Just start assuming all pieces are equally valuable for now, and when you have learned to win games that way, you will be ready to value the Queen a bit more". If players of around 1000 Elo would not be taught the 1:3:3:5:9 rule, they would probably never be able to acquire a higher rating.

The nice thing about computer studies is that you can actually test such issues. You can make the look-ahead so shallow and full with oversights that it does play at the level of a beginner, and still measure how much better or worse it does with a Knight instead of a Bishop. And how much the rating would suffer from using a certain set of erroneous piece values to guide its tactical decisions. And whether that is more or less than it would suffer when you improve the reliability of the search to make it a 1500 Elo player.

It would also be no problem at all to generate 300,000 games between 3000+ engines. It doesn't require slow time control to play at that level, as engines lose only little Elo when you make them move faster. (About 30 Elo per halving the time, so giving them 4 sec instead of an hour per game only takes some 300 points of their rating. So you can generate thousands of games per hour, and then just let the computer run for a week. This is how engines like Leela-Chess Zero train themselves. A recent Stockfish patch was accepted after 161,000 self-play games showed that it led to an improvement of 1 Elo...

And in contrast to what you believe, solving chess would not tell you anything about piece values. Solved chess positions (like we have if end-game tables if there are only a few pieces on the board) are evaluated by their distance to mate, irrespective of what material there is on the board. Piece values are a concept for estimating win probability in games of fallible players. With perfect play there is no probability, but a 100% certainty that the game-theoretical result will be reached. In perfect play there is no difference between drawn positions that are 'nearly won' or 'nearly lost'. Both are draws, and a perfect player cannot distinguish them without assuming there is some chance that the opponent will make an error. Then it becomes important if only a tiny error would bring him in a lost position, or that it needs a gross blunder or twenty small errors. And again, in perfect play there is no such thing as a small or a large error; all errors are equal, as they all cost 0.5 point, or they would not be errors at all.

So you don't seem to realize the importance of errors. The whole elo model is constructed on the assumption that players make (small) errors that weaken their position compared to the optimal move with an appreciable  probablity, and only seldomly play the very best move. So that the advantage performs a random walk along the score scale. Statistical theory teaches us that the sum total of all these 'micro-errors' during the game has a Gaussian probability distribution by the time you reach the end, and that a difference in the average error/move rate implied by the ratings of the players determines how much luck the weaker player needs to overcome the systematic drift in favor of the stronger player, and consequently how often he would still manage to draw or win. Nearly equivalent pieces can only be assigned a different value because it requires a smaller error to blunder the draw away for the side with the weaker piece than it does for the side with the stronger piece. So that when the players tend to make equally large errors on average (i.e. are equally strong), it becomes less likely for the player with the strong piece to lose than for the player with the weak piece. Without the players making any error, the game would always stay a draw, and there would be no way to determine which piece was stronger.


Kevin Pacey wrote on Sun, Mar 10 06:49 PM UTC in reply to H. G. Muller from 04:13 PM:

I guess I have to get into the specifics I personally still don't trust about computer studies, again.

First, Kaufman's type of study: saying that B=N based on large number of games stats (I only vaguely recall, but many of the players in his database may have been sub-grandmaster level - GMs are relative adults compared to 2300 players playing wargames in a sandbox vs. each other). If you want to establish the absolute truth of if B=N, solving chess from the setup and then doing some sort of a database wins/losses count for [near-]'perfect' play would be best, but that is impossible on earth right now (perfect play, if it does not result in a draw, would probably favour White).

Today's best chess engines might be used to generate, say, a 3000+ vs. 3000+ engine vs. engine database if enough games could be played over time to very statistically matter - that would be arguably second best, but even then there may be some element of doubt to the result being the truth that might be hard to assign exact probability to, perhaps (maybe even a professional statistician who is also a GM could throw up his hands and say, we simply cannot say). In any case, the time it takes to make such a database makes it impractical for now, yet again.

Coming to the type of study used for fairy chess piece values, I don't know how margin(s) of error for such a study can be confidently established, for one thing. Next, more seriously, on my mind is the exact setup and armies used in a given study. For Chess960, I saw somewhere long ago online that someone figured after their own type of study that certain setups are roughly equal, while others favour White more than in orthodox chess, say up to 0.4 pawns worth over Black (you might find this somewhere on the internet, to check me). Consider also that that's just for armies that are equal in strength exactly, being identical as in chess. You may give both sides equally White and Black, but the setup and armies vary per study, and I'd guess it's hard to always be exhaustively fair to every possible setup/army, given time constraints.

Finally, you wrote earlier that errors tend to cancel each other out with lower level play (say 2300+ vs. 2300+ engines, as opposed to 2700+ vs. 2700+), It would be very good to know how many games and studies (even roughly) you base that conclusion on, if you still recall. Also, does the cancellation ever significantly favour one side or the other very much with any given [sort of] study? I think the strength of the engine(s) used just might be the most underestimated/large factor causing possible undetected error with this type of study (and sub-GM play within Kaufman's database study, as I alluded to above).


H. G. Muller wrote on Sun, Mar 10 04:13 PM UTC in reply to Kevin Pacey from 03:26 PM:

Sure, methods can be wrong, and therefore have to be validated as well. This holds more for true computer studies using engines, than for selecting positions from a game database and counting those. The claim that piece A doesn't have a larger value B if good players cannot beat each other when they have A instead of B more often than not is not really a method. It is the definition of value. So counting the number of wins is by definition a good method. The only thing that might require validation is whether the person having applied this method is able to count. But there is a point where healthy skepsis becomes paranoia, and this seems far over the edge.

Extracting similar statistics from computer-generated games has much larger potential for being in error. Is the level of play good enough to produce realistic games? How sensitive are the result statistics to misconceptions that the engines might have had? It would be expected of someone publishing results from a new method to have investigated those issues. And the method applied to the orthodox pieces should of course reproduce the classical values.

For the self-play method to derive empirical piece values I have of course investigated all that before I started to trust any results. I played games with Pawn odds at many different time controls, as well as with some selected imbalances (such as BB-vs-NN) to see if the number of excess wins was the same fraction of the Pawn-odds advantage. (It was.) And whether the results for a B-N imbalance were different for using an engine that thought B>N as for using one that thought N>B. (They weren't.)

New methods don't become valid because more people apply them; if they all do the same wrong thing they will all confirm each other's faulty results. You validate them by recognizing their potential for error, and then test whether they suffer from this.


Kevin Pacey wrote on Sun, Mar 10 03:26 PM UTC in reply to H. G. Muller from 08:07 AM:

Your second paragraph may be bang on, except it could be a circular argument to say a body of evidence has been found by chess studies, yet it is the methodology of those very studies that might be viewed as unproven.


H. G. Muller wrote on Sun, Mar 10 08:07 AM UTC in reply to Kevin Pacey from Sat Mar 9 09:48 PM:

I am not sure what you try to demonstrate with this example. Obviously something that has never been tested in any way, but just pulled out of the hat of the one who suggests it, should be considered of questionable value, and should be accompanied by a warning. As Dr. Nunn does, for untested opening lines. And as I do, for untested piece values. That is an entirely different situation than mistrusting someone who reports results of an elaborate investigation, just because he is the only one so far that has done such an investigation. It is the difference of someone being a murder suspect merely because he has no alibi, or having an eye witness that testifies under oath he saw him do it. That seems a pretty big difference. And we are talking here about publication of results that are in principle verifiable, as they were accompanied by a description of the method obtaining them, which others could repeat. That is like a murder in front of an audience, where you so far only had one of the spectators testify. I don't think that the police in that case would postpone the arrest until other witnesses were located and interviewed. But they would not arrest all the people that have no alibi.

And piece values are a lot like opening lines. It is trivial to propose them, as an educated guess, but completely non-obvious what would be the result of actually playing that opening line or using these piece values to guide your play. It is important to know if they are merely proposed as a possibilty, or whether evidence of any kind has been collected that they actually work.


Kevin Pacey wrote on Sat, Mar 9 09:48 PM UTC in reply to H. G. Muller from 06:12 PM:

Re: "It was not checked by anyone, so it must be wrong" is not really a valid line of reasoning"... (H.G. wrote)

Something being not yet worthy of trust is a shade different than being said to be wrong (or proven to be).

In 'Secrets of Practical Chess', GM Dr.[of math] John Nunn wrote of little- or un-tested sequences of play (in over-the-board games of strong players) in the opening phase of a game, that are recommended by chess authors, not to trust them to be in your chess opening repertoire (especially if you must rely on just such sequence(s) to keep your repertoire from going under, I'd add). Meaning, I suppose, treat them like rubbish until proven otherwise. Or, let someone else be the Guinea Pig - probably the advice was especially meant for players well below GM level.

That was back in the 1990s, when commercially available computer engines were mostly still relatively weak, though. Nowadays maybe you can count on what you come up with at home using a chess engine as to be virtually golden.


25 comments displayed

Later Reverse Order EarlierEarliest

Permalink to the exact comments currently displayed.