STUCI technical implementation

[zappa_engine]

(edited note) I didn't want to set any expectations so this is why this post is in a semi-private forum instead of STUCI.

Okay I've pitched my idea in another thread. At this point, it's all just ideas and vaporware. But there's a lot of concept behind it and how it would all come together. This is more of a details thread on the guts of the program.

There are 3 parts to the program, the "client" plug-in's, the "server" plug-in's and the core "manager".

The program has already been through some modularization in the demo code. This is the first step and it will separate the client functions from the server functions, allowing the manager to call each the server or client multiple times if wanted, or not at all. The core is stand-alone. Calls to and from clients and servers will be threaded.

CORE: The core's basic input is the keyboard input. It acts as the "default" client. You load STUCI, and you're staring at the core prompt command line. It manages clients and servers. At this point, STUCI does implement the keyboard interface now. The core also controls basic functions like create table, chat stuffs, player lists, etc, but it does this in a neutral way to any protocol (see server now).

CLIENTS: I noted the core has a default client, but you can also connect an external "ICS" client to STUCI. This changes the default basic input to the ICS client. Here's a common misconception, when you're typing in the ICS window you're not typing directly at the core. You're typing at the client which then sends the information to the core. So, yes, the input is the ICS client. There is a mechanism to switch what inputs the core will process. Connecting an ICS client should automatically switch control of the input from the keyboard to ICS client. If there are no clients setup, there will be a display to allow setup of clients. Those client setups are saved and can be re-used. If a client is disconnected, the input control will kick back keyboard (core control). There can be only one client connected at a time, as opposed to the server which can have many. It's either the ICS client or the "default core" client. There's also UCI but that is actually an extension of the default core client to use an engine to play.

SERVERS: The server can defined by how the protocol will connect and communicate. It used to be that YtoICS managed all this, but with modularization this job is separated from the core. The neutral functions that core has, are actually empty functions. The protocol code will fill out these functions to adapt to the server protocol used. What's interesting to note about Yahoo's server protocol is once you receive a cookie, you don't have to keep receiving another one to log in again. You can simply reuse that cookie to log in. You only have to log in once as long as the cookie never expires. In this way, you can automate log ins. And by that, I mean you can save user/server log in setup files. file:"zappa_engine.yahoo.server" if I gave you this file and you used it, you could log in as me without knowing my password. Interesting stuff, huh?

Well, if you have any further questions about how it will be done, ask away.

[zappa_engine]

Well, the default core client is now functional in a basic version. This is allowing me to hot-plug clients. That is I'm switching between the default core client and ICS with no major problems.

Next step is multi-login, though in a limited functionality. I'm not going to sync the net connections just yet. I'm just going to create 2 or more connections to different rooms and allow switching between them. Their data streams will remain separate though (which could cause some problems as core doesn't know how to handle multiple network inputs). So it's just going to be a chat connection. I'm not syncing two user lists together yet. They will eventually be managed through an array/hash type.

[dene]

What's interesting to note about Yahoo's server protocol is once you receive a cookie, you don't have to keep receiving another one to log in again. You can simply reuse that cookie to log in. You only have to log in once as long as the cookie never expires. In this way, you can automate log ins. And by that, I mean you can save user/server log in setup files. file:"zappa_engine.yahoo.server" if I gave you this file and you used it, you could log in as me without knowing my password. Interesting stuff, huh?

Very Interesting.

Well, the default core client is now functional in a basic version. This is allowing me to hot-plug clients. That is I'm switching between the default core client and ICS with no major problems.

Damn you work quick

[zappa_engine]

Okay, here's a technical update for those who want to know how it works. This picture shows me starting in core client mode, issuing a command to switch to ICS client mode (where network sync is lost), and switching back to core client mode (no sync lost but it retained the original network data lost when switching from core to ics mode in the first place) so showing sync issues of past.

Client hot-plugging is working, although with issues. I will explain the quirks and how they were solved or will be solved.
Let's start with an update to what's new with STUCI now.

Issue #1: The core default client has been put into place. This client takes input data from the server, and data input from the user (keyboard). Because of the blocking/wait nature of grabbing input from the keyboard, the networking updates don't come through until after the user presses enter.
The solution has already been solved. STUCI has started to use threads (http://search.cpan.org/~rgarcia/perl-5. ... thrtut.pod). What is being threaded out is the network input management from the server, while the main thread still handles the keyboard input.

Issue #2:Threading in itself causes some issues. Let me explain, the variables used by the main program are copied over to the new thread. They are not linked to the same data in the main program. As the new thread updates information, it's only updated in that thread. At this moment, the updated information isn't shared yet. Switching between ICS client and core client will cause things to become unsynced. Notice in the picture above, it's trying to process data for evenchrisjones and shrik[???] and it can't because things have become unsynced. There will need to be the creation of a shared variable to store the pertinent information to pass back from the threaded network updates to the main program again. It doesn't need to store every network transaction or anything, just the user data, the games data, anything that doesn't require a response to be issued (as we switch from core client to ics client, we don't want to flood update responses back to server!). I don't think anything needs a response actually. Well, if you have suggestions or comments, let me know. The obvious solution here would be to share the user and games and any other needed variables right? Except they are hash format with complex structures inside them that can't be shared. I believe a network buffer cache needs to be stored. By the way, this issue only happens when switching from core default to ics client (because of threading and separate memory model). It doesn't happen switching from ics client back to core.

[matematiko]

Hello zappa_engine,

I've been going thru some of the code you posted and this de-modularizations looks very promissing and its an excellent idea. My limited "perl mind" does not allows me to understand everything but good enough to grasp the concept behind. Probably I'll put some of these ideas into action some day. I partucalry think I do not have the brains to accompish something like that, I am more interested on been able to launch the script in core+server mode(alaways the samer server Yahoo! of course) without the need of winboard or any other GUI/engines. This of ocurse is something for my personnal needs and does not have anything to do with public releases. So, hopefully the study of your code will allow me to do something I tried before but I couldnt: Launch my script and join the Yahoo server right from the command prompt. Be able to chat, create tables and stuff with no GUI no engine no nothing, just my prompt, Yahoo.....why do I need to do this?---well, that's another story he he----
Anyway, you kow I am busy also getting my own flash version to work, but if you need any help testing something I will make time to do so. If this project of your is ever completed, I probably woudnt even have to use YaY anymore he he and if I can help with testing, that will be an honor.

Take care bud,

[zappa_engine]

core+server

Okay, I sent you some code but it is my test builds. You'll need to look inside the main() function and comment out the line $client = 'ics';
Also you'll want to run (stuci-080613-grab-cookie.pl) just once to grab the saved login data. You only need to do this once. As far as I can tell, I haven't had my cookies expire in 3 days so far! The main code is set to automatically use this data and I'm not sending you my saved login data for you to test I think when you look inside the code, there's a lot of toggle flags for options scattered through out. It is a test build after all! Let me know of your suggestions or comments

Switching between core client and ics client... within the fics% prompt, type "ev set_client_core" or "ev test_start" (loads ICS client).

Oh! One more thing... you can look through the code and comment out the $debug variable, or you can type debug in the fics% prompt to turn off all those networking messages.

[matematiko]

Thanks for the explanation and the code samples, I'll take a look when I get home. Anyway, I am afraid I didnt explain my self correctly.

You know when you launch YtoICS in unix, the script does not waits for the winboard connection, it logs in to Yahoo, there is not an ICS Interaction window and yet you can issue commands thru the keyboard. This is exactly what I want to do in Windows. In the past I was able to do this, connect to Yahoo without the need of an ICS server GUI, but due to the infite loop that is checking for server/winboard commands the keyboard is never released to the user? Is this confusing?

I suspect this is more a functionality difference between Linux and Window sbut I hope there is a work around in Windows.

It is not my intention att all to ever run STUCI(or my script) in core only mode. The idea is to use the core plus the ics emulation client without WInboard, Arena or whatever. Does it make sense now I hope.....thanks !!!!!

[zappa_engine]

You know when you launch YtoICS in unix, the script does not waits for the winboard connection, it logs in to Yahoo, there is not an ICS Interaction window and yet you can issue commands thru the keyboard.

Here's a common misconception, when you're typing in the ICS window you're not typing directly at the core. You're typing at the client which then sends the information to the core. So, yes, the input is the ICS client.

I'm not familiar with that YtoICS branch that does that but I've only looked at the latest YtoICS code. You might be tricked into thinking it does that because xboard's ics console output and input is merged into the shell console but the communication is still with xboard, so it doesn't lock up the keyboard.

In the past I was able to do this, connect to Yahoo without the need of an ICS server GUI, but due to the infite loop that is checking for server/winboard commands the keyboard is never released to the user? Is this confusing?

Nope, you're making sense. I encountered this issue trying to manage input from user without sending the information through the Winboard client. The keyboard input in the while was causing a 'block' issue which was not allowing the server inputs to process until after the user entered a command. This is where I learned some threading techniques to split the program into two processes while in core client. The main process still handles the keyboard input and yes, the blocking still happens, but here's the nice part. Server network data is processed in parallel on another process (thread). Notice on the core display, it will say "(thread) Mapping server inputs to operations..." which means it is starting that parallel thread "sub handle_serverops()". This is where data is copied but syncing of data between the main process and the server handling thread goes out of balance. I haven't implemented the re-syncing once the user decides to switch back to ICS client mode (the thread ends, sync information is send back to parent, ics client processes sync data, and then resumes handling network in a single main process).

So yes, this code will accomplish what you requested. But at the moment, it is set to automatically load with an ICS client to match YtoICS behavior. You just need to edit the main() function to comment out the $client='ics' variable and it will default to loading in core client mode (what you wanted).

addendum: I noticed the code I sent you still has the original core client implementations that was not threaded. It is in a function called set_client_core2(). It shows what I mean by keyboard blocking server network inputs. If you rename the set_client_core() function to something else and rename set_client_core2() to set_client_core() you'll see the effects of non-threading client.

addendum2: With the function set_client_core2 replacing set_client_core, you can also notice the affect of network timeouts if you change that variable up to 1 second, you'll see it slows down input! As such I realized setting it to bare minimum timeout doesn't affect processing data and speeds up input dramatically. Well, not dramatically if you don't notice the extra 1/20th of a second. That's where I implemented the minor performance boost.

[zappa_engine]

The details about handling multiple connections.

There is a list array @connection that contains a list of all network connections by yport value (unique key). These values correspond to a hash (associative array) of the actual network connections. That way any split processes will have access to checking and updating the network connections $net{'yport'}=individual $net(IO::Socket) through the @connection array by cross checking.

[matematiko]

These are some snapshots I took just a minute ago while I was logged in to Advance Lounge with my ID table.stealer using a unix shell account and accessing this account with Putty.exe .

In this first picture I issued the "who" command and received some data:

who.JPG

In the second one I issued the "games command and received again some data:

games.JPG

In the third picture I joined and unobserve a table, then joined another table and got booted.

commands.JPG

Based on the messages issued by the script, looks like some how the "iprint" messages are sent to the prompt window some how.

I'm not familiar with that YtoICS branch that does that but I've only looked at the latest YtoICS code. You might be tricked into thinking it does that because xboard's ics console output and input is merged into the shell console but the communication is still with xboard, so it doesn't lock up the keyboard.

I do not understand your statement; the way I launched my script was: prompt> perl ytoics.pl

I did not associated this script with xboard at all, and xboard is not running in the background either, so I don't understand how xboard comes into the picture.

Anyway, in case it was not clear, this is exactly what I am trying to accomplish on my Windows machine.

Thanks.

[zappa_engine]

Based on the messages issued by the script, looks like some how the "iprint" messages are sent to the prompt window some how.
It could be. With the latest version of YtoICS though, output is piped through the client which then prints it to STDOUT.

I did not associated this script with xboard at all, and xboard is not running in the background either, so I don't understand how xboard comes into the picture.
You might have a different version of YtoICS I haven't seen. Maybe it is based on YtoICS older than 0.0.4 CVS right? Sorry I was a relative new-comer to auto-chess back then, so my sources were somewhat of the latest release. The latest release waits for inputs from both the client and the server. It looks like your ytoics doesn't even wait for a client. Maybe I can get a copy of that source to see how it managed with keyboard blocking. Quite possibly keyboard blocking was the issue in Windows environment but not in Linux environment and I developed the solution for Windows environment (as well as an alternate solution in Linux).

Anyway, in case it was not clear, this is exactly what I am trying to accomplish on my Windows machine.
Yes, the current implementation will do that

[zappa_engine]

Okay I got the code. The very old YtoICS didn't listen for client input. (This change to client input was done from rev 1.16 -> 1.17)

http://ytoics.cvs.sourceforge.net/ytoic ... arkup[code] 394 my $sel=new IO::Select(\*STDIN, $net);
395 my($buffer,$in);
396 while (1) {
397 foreach my $i ($sel->can_read) {
398 if ($i==$net) {
399 &handleop;
400 } else {
401 sysread($i,$in,1) or exit;
402 if ($in eq "\n") {
403 handlecommand $buffer;
404 $buffer='';
405 } elsif ($in ne "\r") {
406 $buffer.=$in;
407 }
408 }
409 }
410 }
[/code]
I believed I tried this and there was keyboard blocking for Windows. I guess that doesn't happen in *NIX environments. I will cut/paste this code in some time to test it though.

Okay it's very similar to set_client_core2 code I tried. So yes, I did implement the alternate solution (threads) that works on Windows.
The note for the change log on revision 1.17: added -p option (which fixes Windows combatibility issues) -- this is the support for ICS client because probably, he encountered the keyboard blocking issue which made YtoICS incompatible with Windows. I'm glad to say it is compatible without the ICS client now (via core client).

On another note, an important function there is the IO::Select statement. It enables hot-plugging clients and servers. Translating it to human-readable form, it means I want to SELECT the \*STDIN keyboard (AKA client) and also select the $NET socket(AKA server) and capture any input data from them and post results to $SEL.

[matematiko]

Yes, all my scripts are based on version 0.0.4 and not 0.0.4-CVS (with the exepction of the one I am currently working on). Thanks for taking the time to looking into this.

Oh....I forgot to mention, how do I launch a YtoICS from a unix shell account if the shell has not graphics support? Well, since CAPTCHA was implemented, I create the applet.html as usual in my Windows machine, FTPit to my shell account via WinSCP and viola, this confirms your statement that everyone can log in to Yahoo Chess server if they have your cookies he he.

[zappa_engine]

I was thinking... why do you want to load core+server but no client/engine? Are you not allowed to run UCI binaries? I was thinking of creating a specialized UCI server. You can then load the UCI engine on any machine that lets you as a server. When you connect to it, you can send commands to it and receive responses over the network! So then STUCI core could connect to the UCI server and pretend it's a client. Or maybe I should make the UCI client which connects to STUCI.

[UCI Client on machine #1] <--network link--> [STUCI Classic on machine #2 ] <--network link--> [Yahoo Server]

Just taking the hot-plug concept to another level... oh boy. Harness the power of a super-computer UCI server over a network! My laptop is no longer my bottleneck. STUCI handling book locally, UCI server handling thinking, STUCI Classic core, and Yahoo Server. Will someone host a UCI server for me?

[matematiko]

I do not know if this help, but I can run your UCI server in my unix shell account or of course you can set up your own, very easy as you know.

Why do I want you load core+server without client? I rather tell you about this in MSN Messenger.