lichess.org
Donate

A Harness for LLM Chess: Board State and Context Handling

Wow man ! That's so cool ! Can you estimate their elo ? Maybe you could make bots on lichess people could play against. Or use it as an analyst after a game to explain the moves and why they were good/bad.

Amazing project anyway

Wow man ! That's so cool ! Can you estimate their elo ? Maybe you could make bots on lichess people could play against. Or use it as an analyst after a game to explain the moves and why they were good/bad. Amazing project anyway

@McGambler said ^

Wow man ! That's so cool ! Can you estimate their elo ? Maybe you could make bots on lichess people could play against. Or use it as an analyst after a game to explain the moves and why they were good/bad.

Amazing project anyway

Thanks! Yeah I think setting up a Lichess bot for a few models to get their rating would make sense as a follow up - I thought about it before but stopped because I can't imagine how the token usage would explode lol.

@McGambler said [^](/forum/redirect/post/1hjJuqwU) > Wow man ! That's so cool ! Can you estimate their elo ? Maybe you could make bots on lichess people could play against. Or use it as an analyst after a game to explain the moves and why they were good/bad. > > Amazing project anyway Thanks! Yeah I think setting up a Lichess bot for a few models to get their rating would make sense as a follow up - I thought about it before but stopped because I can't imagine how the token usage would explode lol.

@Fingolfin7 said ^

Thanks! Yeah I think setting up a Lichess bot for a few models to get their rating would make sense as a follow up - I thought about it before but stopped because I can't imagine how the token usage would explode lol.

I think with the right models it can get quite cheap. ChessAgine provides high quality free llm analyses for all the games you import. In any case you can always set a limit ahah

It's open source so you can also look at their prompts to compare it to yours

@Fingolfin7 said [^](/forum/redirect/post/TAELDPeY) > > Thanks! Yeah I think setting up a Lichess bot for a few models to get their rating would make sense as a follow up - I thought about it before but stopped because I can't imagine how the token usage would explode lol. I think with the right models it can get quite cheap. [ChessAgine](https://www.chessagine.com/game) provides high quality free llm analyses for all the games you import. In any case you can always set a limit ahah It's open source so you can also look at their prompts to compare it to yours

It would be nice to try to teach these LLM models on chess books! For example, Capablanca's book - The Manual of Chess, and so on. So that the LLM model explains each and why it is done! For example, there is another book by Andre Filidor - Analysis, that's what you need to teach.

It would be nice to try to teach these LLM models on chess books! For example, Capablanca's book - The Manual of Chess, and so on. So that the LLM model explains each and why it is done! For example, there is another book by Andre Filidor - Analysis, that's what you need to teach.

do you have a use case for this relating to explaining moves or positions? like a "coach" role?

do you have a use case for this relating to explaining moves or positions? like a "coach" role?

My question is: why do this at all? How does the LLM help here? An LLM seems to me to be fundamentally the wrong tool for the job, and teaching them to play chess is kind of like modding a toaster so it can cook pasta. You can do it, but why?

My question is: why do this at all? How does the LLM help here? An LLM seems to me to be fundamentally the wrong tool for the job, and teaching them to play chess is kind of like modding a toaster so it can cook pasta. You can do it, but why?

@Presiident said ^

do you have a use case for this relating to explaining moves or positions? like a "coach" role?

It's doable but the issue is that the LLMs aren't actually all that good at chess so I wouldn't trust their analysis all that much. Unless we connect them with a traditional engine to do the analysis and then the model could do its best to "explain" what Stockfish is explaining? But even then its not too different than just having an okay-ish player explain things to you...

@Presiident said [^](/forum/redirect/post/txUetqPY) > do you have a use case for this relating to explaining moves or positions? like a "coach" role? It's doable but the issue is that the LLMs aren't actually all that good at chess so I wouldn't trust their analysis all that much. Unless we connect them with a traditional engine to do the analysis and then the model could do its best to "explain" what Stockfish is explaining? But even then its not too different than just having an okay-ish player explain things to you...

@Gardenirr said ^

My question is: why do this at all? How does the LLM help here? An LLM seems to me to be fundamentally the wrong tool for the job, and teaching them to play chess is kind of like modding a toaster so it can cook pasta. You can do it, but why?

there's not much practical utility apart from just wanting to see how these models think through positions. The models are supposed to be general thinking machines (not necessarily saying that they are now, but that's what the labs are trying to make). Treat it like a semi-benchmark for testing how good LLMs are at reasoning outside of the standard benchmarks the large labs are optimising for.

I think chess is a good general test of thinking quality because it requires a bunch of different skills like long-term planning and visual/spatial understanding. So it's an interesting way to see how well the models "think".

@Gardenirr said [^](/forum/redirect/post/SxcLFpDg) > My question is: why do this at all? How does the LLM help here? An LLM seems to me to be fundamentally the wrong tool for the job, and teaching them to play chess is kind of like modding a toaster so it can cook pasta. You can do it, but why? there's not much practical utility apart from just wanting to see how these models think through positions. The models are supposed to be general thinking machines (not necessarily saying that they are now, but that's what the labs are trying to make). Treat it like a semi-benchmark for testing how good LLMs are at reasoning outside of the standard benchmarks the large labs are optimising for. I think chess is a good general test of thinking quality because it requires a bunch of different skills like long-term planning and visual/spatial understanding. So it's an interesting way to see how well the models "think".