Insecure

It may not seem that way, but I suffer from a severe lack of confidence occasionally. It's an irrational kind, the one that you might develop when, as a child, you never have received love.

I doubt myself in regard to MoyoGo sometimes. I am sitting behind those huge flatpanels (NASA would be envious) thinking: "Why nobody buys it?" - ignoring that I sold a copy less than 24 hours ago - and I feel a mild sense of despair. This process repeats itself every day, and yet, on average, since the first release I have sold almost one copy per day.

When I have no sales for a day or so, I start to think: "This is because the product sucks, I suck, life in general sucks and everybody hates me" and I ferociously start improving MoyoGo or the website. I did that yesterday, when I spent 8 hours trying to crash MoyoGo and fixing the underlying bugs. When I got out of the "anti-depression bath" this afternoon, I saw that I had sold a copy to someone in the Netherlands.

Better something that is imperfectly there, than nothing at all. Like when Luba told me in the abzhizhitije in Dnepropetrovsk: "Your Russian is getting worse and worse every day - are you drunk or on drugs?".

This is my blog so I can talk about anything I want - even when it's unrelated to Go programming. Well, in a way this story is, for it has contributed to shaping me.

One day I was in Orsa, Sweden. I was the most sophisticated homeless on the planet, travelling semi non-stop for many years. Autumn was approaching and foolish me, I had hoped to work in the bear park there, Grönklitts Bjørnparken. I figured they would at least give me board & lodging.

Travelling light, I crashed out in the woods, about 100 m from a logging track. Someone who had given me a lift had told me that the area was so heavily bear-infested that they wouldn't let their kids play in the garden unsupervised. I tied my mosquito net to a branch, draped my plastic foil over it and "cocooned" up, safe against the terrifying giant killer mosquitoes that managed to sting through Meindl's buffalo leather mountain boots.

I fell asleep. Suddenly - it was pitch dark - I woke up because I was pushed against the tree by something. This "something" was rather strong and kept pushing very hard, like it intended to push me right through the tree trunk.

This caused a fight-or-flight reflex in me, but I could do neither. Straightjacketed by my mummy sleeping bag, enveloped by mosquito gauze and plastic foil (tactically arranged to allow fresh air in), there was no way I could do either. "Elk or bear" went through my head, and I let out a horrific scream, summoning up all I could muster to make it as loud, long and intimidating as possible. Immediately the pushing stopped and I heard something cracking the branches around my lair.

I lay there, pounding heart, deciding what to do next. Get up and find a safer place? I was sleepdrunk and, with that creature around, too scared to get up and walk around in the forest in pitch dark (obsessed by weight-reduction of my gear, I did not carry a flashlight anymore). Then I thought: "If it fled, it was probably frightened and won't return - besides, bears don't kill people who are just sleeping".

I fell asleep and slept like a rose until morning. They told me later that it was 100% sure that it was a bear. Elk don't walk around the forest at night pushing people, they sleep.

I am not sure how this has affected me & the development of MoyoGo, but I'm sure it has, somehow ;-)

Back to the Drawing Board!

In the best sense of the word.

Yesterday I started with the design of a TsumeGo module. Pattern recognition - although limited in its capabilities - is excellent for Fuseki and Joseki, Tesuji and "good shape", the basis for a high-end Go program. What's still missing is a tactical module.

I hear from kyu- and dan-level players that MoyoGo gives them ideas for moves they wouldn't have considered before, and because MoyoGo doesn't "dumbly" suggest moves but shows the situation in games in which those moves were played, a player can make an informed judgment whether that (semi)-local move really is interesting in his own global position.

As to the usefullness of a pattern-recognition system to suggest strong Go moves in general: Of course such a system can never be (highly) "tactical" and the more "thinking ahead" required, the less useful any pattern-based system, but it excels at the opening stages of the game, yielding pro-level Fuseki and Joseki suggestions and their continuations, all the way to midgame.

Of course, players want to analyze "deeper" too, they want to get 100% proven answers as to whether chains can be captured or connected.

If anyone doubted my commitment to providing such functionality (as a free update), have a look at a few of the crates of ordners I have with computer Go abstracts.. Some titles: "Tree Search", "Secure Territories", "Adversarial planning", "Life & Death", "Eyes" and "Combinatorial Game Theory". Most are filled to the brim with publications, and I have read (but not always understood ;-) them all.

I started collecting them five years ago. It is so important to get the design of a TsumeGo module straight, that I did not want to embark on a "quick & dirty" implementation. So I decided to fund the development of the world's strongest TsumeGo module with the world's best (only?) Go pattern expert system. That is the right way to do it, because the biggest moves are in the opening and a TsumeGo module can't achieve pro-level strength in the opening. MoyoGo's pattern system can, and often does. I start with the opening and work my way forward :)

Whilst putting the finishing touch on the Fuseki/Joseki information system - in this phase it is important to listen to the "early adopters" - it becomes time to work on many small feature requests and minor bugfixes, and parallel to that, on the TsumeGo module. The goal is to design a system that is able to handle problems that can't be handled (fast enough) by GoTools or MadLab, but that will be a tough nut to crack. I started to read Steve Pavlina's articles on personal development to prepare myself for this intimidating yet intriguing challenge.

Cool variation lists

Someone said I should stop bickering on rec.games.go and instead work, and that's what I've been doing :)

The Fuseki- and pattern modules have gotten a "variations list", which shows the statistics of the Fuseki continuations resp. possible "pattern moves".

Moyo Go starts to become a real "Go Information System", because it doesn't just show you boring statistics, it also shows the names of the patterns and the Fuseki! And you can add new ones. And look them up on Sensei's Library.

Those (not so) Funny Blocks

You might wonder why you see those funny squares (or garbled text) instead of Chinese, Korean or Japanese players' names, and this is why: You should check Start - Settings - Control Panel - Regional and Language Options - Languages - Install Files for East Asian Languages.

This is a bug by Microsoft. I ask Windoze to show me some Chinese, and Windoze refuses. Why haven't I simply included a Unicode font? Well I haven't gotten round to finding an affordable, high-quality font that includes all CJK codepoints. Redistributing Arial Unicode MS is prohibited.

Fortunately, anyone in China, Japan or Korea will have those fonts installed by default, but if you want to see those funny blocks transformed into their real oriental splendor, you'll need your Windows CD..

Ten Bucks More

I just raised the price of the product by ten USD. The new Fuseki module, albeit not polished yet, warrants it.

Moyo Go has instant Fuseki-search, since today. And it doesn't matter how many games have to be searched. At least on my PC, Moyo Go sports 0.1 second Fuseki search of any position below 40 moves, even when the list of results is hundreds of thousands of games. Moyo Go Studio sets the new standard of no-wait search! (So fuseki matching is on be default).

And.. Next week I'll upload new databases that have Fuseki up to 40 moves. Because, (surprisingly to me as a weak player), there are plenty of games with identical moves all the way up to move #30, and they branch at move 31, etc.

I want to thank the customers who've bought Moyo Go last month, when my website was taken down and Sensei's Library removed my ads. Because - due to Lymerage - I am out of a job and I don't get any benefits. But: Even though I'm not selling much now - I sold thirty copies that month, which pays my rent for a while. These people invested in the new Fuseki module, and I am paying interest by raising the price by ten dollars.

The new Fuseki module will get fancier soon, with statistics on the variations and tooltips with statistics on their annotation. After that I might get around implementing a few dozen smaller features that have been pending for a while.

AGA denies Moyo Go's existence

AGA review: BiGo Assistant
(Reviewed by Philip Waldron)

"I have always found the most useful programs to be databases that allow me to examine how professionals have played in various situations. The latest program in this category is the BiGo Assistant."

Hm.. BiGo is at least a decade old.. The latest program in this category is MoyoGo Studio! Funny how MoyoGo seems to be such a threat that its very existence has to be denied.. Two people have submitted reviews to AGA, both have been rejected..

"Finding common fuseki and joseki patterns usually takes just a few seconds, but other searches took me a full five minutes."

No wonder AGA refuses to review MoyoGo.. AGA makes money peddling MasterGo, a competitor to both BiGo and MoyoGo Studio.. MoyoGo does not need any noticeable time for any searches..

Setup time < 4 minutes! :)

I did it, Moyo Go now installs its hefty 1.8 GB from DVD in less than four minutes (the speed depends on the harddisk, I have a 15,000 RPM SCSI HDD, I don't think there is a faster one but I would be surprised if it would take longer than ten minutes now, on the slowest disks around.

I am uploading new database updates as well - they will install in less than a couple of minutes now! I really had to do something about those outrageous install times because the new Fuseki module required very many new indices, and putting those indices in separate files was disastrous in terms of install speed..

The size of the database updates will be reduced by 10% as well. The system - although very snappy already - seems to search even faster now, perhaps because the system does not have to do a FAT search, every time it needs to do an indexed-based search. It will have cached the sector data of the aggregated index file already.

Another advantage of such a fast installation procedure is the possibility to test every DVD that gets produced.

Updates to Moyo Go will be downwards compatible with the old index format, so it is not neccessary to download new databases (except when the auto-fuseki generation process screws up for some reason).

Setup Time :(

The disadvantage of having 75,000 index files is that it takes ages to install Moyo Go!

Speed of installation is not dependent on the CPU speed because the files are not compressed (the DVD has enough space anyway). With so many little files to transfer, installation speed depends mainly on (always relatively slow) harddisk seek time, and if you thought it was agonizing to sit through a Windows install, wait until you install Moyo Go :)

Small consolation for existing customers: I am going to fix this! Simply by throwing Higher Technology™ at it. Instead of using all those small index files, I will use one huge index file and the software will know where to look. This means that searching will still go as fast as always, but at install time, there are 150,000 less HDD head movements to make.

I am very eager to fix this - install time should go from more than an hour to just a few minutes!

Fuseki Module Rollout!

The Fuseki module is ready! At least its basic functionality. I'm very excited - it's lightning fast and I managed to make an automatic update (to be released soon, after more testing) that builds its own Fuseki databases, so it's not neccessary to download new ones. Although that might be neccessary anyway for some customers, because building Fuseki indices for the huge CyberKiwon collection might prove too hefty for machines with 512 MB RAM. The latest CyberKiwon databases are split into "both at least" 4d, 5d, 6d and 7d volumes, to avoid such problems in the future.

Architect of #1 Go Program Executed

Excerpts from an article by Sonni Effron and published in the record edition of the LA Times, Aug. 19, 1999. Provided under "fair use" (original article is almost three times as long).

"...Silver Igo beat 40 other programs from around the world to win the FOST cup, run by an artificial-intelligence group, in Tokyo in August 1998.

But soon after, the Chinese author of one of the most successful go programs, Chen Zhixing, accused the Silver Igo program of having plagiarized his "Handtalk." Chen... ...demanded the return of the FOST cup title.

Masahiro Okasaki, who was the chief judge for the tournament, said Silver Igo was not a direct copy of Handtalk, although like many programs, it imitated the ideas of the powerful Chinese program.

The legalities of software plagiarism remain vague in Japan, and the North Koreans were allowed to keep their title. But in a bit of intrigue around the same time, the Silver Star head was executed in North Korea, reportedly for political reasons having nothing to do with the go program, according to Okasaki.

...Chen, in a telephone interview, said he has had no response to his allegations.

Meanwhile, Silver Star has been shut down, and the 22 or 23 people working in the games section have been moved to North Korea's largest computer enterprise, called KCC, according to Silver Star Japan's Yamamoto."

Fuseki Debugged

It's 04:00 AM and I wanted to share this. I found the remaining bug and soon the next auto-update will have an extra tab with a list of all matching Fuseki positions of all enabled databases. This does not take any noticeable time. It's pretty cool to see a list of tens of thousands of matching positions appear and change instantaneously as you move through a position!

The things left to do is re-building all databases. I have changed so many things (added indices etc.) that I want to be able to have the latest versions downloadable before I release the "Fuseki"-update. But the auto-update will be able to build its own Fuseki database very quickly, so there is no urgency in downloading the new databases. It's just that I want to be able to have a "backup" solution just in case something doesn't work.

Rebuilding all databases should be completed in a day, running on 4 Opteron cores. The Fuseki part takes mere minutes, it's the patterns that take so much time.

The other thing is to add "variation statistics" to the Fuseki module. Just a list of matching games and winning percentage/average rank isn't enough, we also want to see the variations and their winning percentage and average rank.

I hope to implement this in the time that the new databases are building.

Shipping of new orders will be delayed until the Fuseki system works (by the weekend, heaven permitting) and existing customers will have to wait until the next auto-update, which will build its own Fuseki database. This update should be ready before Sunday.

'Been Vexed: Hex to Bin Binned!

There is a devious bug in the new Fuseki module that likely has something to do with 64-bit bitmasks and shifts performed on them so I need to verify & document what's going on.

But one of MS's updates (or perhaps x64 itself) "improved" the Calculator app. so much, that it became useless to me. I often need to convert 64-bit binary to hex, and vice versa.

MS managed to break that functionality by deciding to display large binary numbers in scientific notation, from now on. Now, when I convert a hex number I get things like 1.^+101100 instead of a nice string of bits. There is no workaround: Yet another MS app has been upgraded to death - my scientific calculator can't display 64-digit binary numbers either so I will have to look for some freeware that can perform this trick for me.

I am worried about the path Microsoft has chosen. I am dependent on Microsoft, my software does not run on other platforms than theirs. But every single update of every single piece of software Microsoft has released recently is worse. Another example: Their latest Media Player can't pause the movie any more by pressing the spacebar, or by any other method I can find. If a digital electronics engineer with 25 years of software development experience can't figure this out, not many others will. MS's problem is that they employ an army of codemonkeys and they have to be kept working, apparently regardless of the merits of their work. "New" is not always "better".

Paranoid Programmer

Some say that being paranoid is another word for having all the facts, and that might be true when no mental illness is involved.

When there is a real threat, improvisation may be neccessary to deal with it.

I think all good programmers become a little paranoid eventually, or perhaps it's the other way round.

Fuseki Module: Status Report

I discovered a few wrong bits in all new database updates (the ones containing Fuseki data), so I removed them from my site. My apologies to people who downloaded them already. Because they contain some new sorting fields they will ultimately all have to be redone and re-uploaded, but that will not affect the Fuseki module.

I had to remove them (temporarily) anyway, because I was getting rather scary bandwidth bills of 22,000 USD (even though they were a mistake, I prefer not to exceed the monthly maximum).

A much better solution would be to build the new databases on the user's machine, if that would not take too much processing time. This appears to be feasible. Extracting the Fuseki is very fast, but sorting the index files could need some more work (can be twice as fast with a better sorting algorithm). So the next update will build its own Fuseki index files for 410,000 games, without the need to download new databases!

The main part of the Fuseki module is working: It displays a list of matching games with average rank & winning percentage. I designed the system to take almost no time for the search. Which means that when you have a position with a black stone on a 4-4 point, there will be a list of a few hundred thousand games, and you won't notice that this took any time, on a reasonably modern PC. Now I just have to do the boring stuff like column-size/position persistence, column sorting, finishing & debugging of the GUI integration etc. And of course the part that creates the Fuseki index files on the user's PC.

But I also want it to display all variations and their winning percentage and average player rank, so that will take quite some work. I might do that at a slightly later stage, as an additional auto-update.

I want to have it well-rounded and tested rigorously, so please bear with me.

A Strengthening of the Force

Lots of visitors from Japan today! (Image is from Google Analytics.)

22,000 USD downloading bill

Update: I don't have to pay anything! They had set the monthly limit to 6 GB, while in fact it's much more. They assured me I will not pay more for exceeded bandwidth either, they'll warn me first.

Dear Frank de Groot,

This letter informs you of how much was accrued/charged for the following operations:
Login: frank
Account ID: 2056
Plan: VIP
Order: #2046-2513-1

Monthly fee for VIP plan (1/11/06 - 2/11/06) $25.00
Monthly fee for extra 44,000 MB above free 6,000 MB of disk quota (1/11/06 - 2/11/06) $22,000.00

TOTAL: $22,025.00
YOUR CURRENT BALANCE: ($21,750.00)

Dizzy

I am working on the last part of the Fuseki module (the harvested Fuseki data is already downloadable in the form of the latest database downloads, and the current auto-update of Moyo Go adds Fuseki data to both the old and the new databases). I have been dizzy all day due to change of antibiotics, so things go slow. The light may also be a factor, I put in those energy-saving types that generate a lot of light. To fight winter depression here in Oslo. I worked a bit on the website. The ads that Sensei's Library refused are now here & there, some in this blog.

If people want to support me, feel free to put such an ad with a link to my site. That might offset the balance a little, as I will not get linked by GoBase and neither does Sensei's Library allow me to put ads there. AGA won't publish a review and no dealer wants to retail it. Perhaps it will help me fight back, at least get a decent Google pagerank.

I was thinking about the future genetic programming part of Moyo Go. Of course, merely evolving domain knowledge is not enough. This has to be done efficiently. So I found the solution for that: A Go-specific scripting language in which complex concepts can be represented with short "genomes", and where every possible genome encodes for a "legal" piece of Go-related executable code. I took ladder-finding as a testcase for this idea. Would it be possible to evolve ladder-finding code out of "nothing"? Yup! Easily, in fact, with this method.

Hoshi vs. Flower-point

My friend Young H. Rhie sent me this:

"You can see nine points in the go board. They are called Hoshi points.
Hoshi means "star" in Japanese. Koreans call them Hwajeom. Hwajeom means "flower point".

The word Hoshi tells us that the go board itself represents the Cosmos. The center-most point is called Ten-gen meaning "The center of the Cosmos".

The Hoshi points of old Korean go boards are not shaped like circular points of current go boards.

There used be shapes of 17 or 25 lotus flowers or ume flowers. 17 points are made for pre-setup points for Sunjang Baduk (old style Korean Baduk).

So Koreans called them flower points.
Though we do not see the flowers on the board, we still call them flower points."

Evolvable Domain Knowledge

Some regulars at rec.games.go think I am a stark raving, barking mad insane lunatic nutcase.

Because when I said I was going to focus on a "genetical" approach to comp. Go, one of its residents said that it was (I paraphrase) "a bad idea to try to evolve a flower, because a flower does not play Go well". It sounded as if he thought Darwin was crazy too.

This comes from someone who says he used to be paid for trying to write a Go program, and he failed. Now he likes to tell folks that making a strong Go program is "impossible".

I checked out some GA pages, and what I found strengthened my resolve: Several recently patented inventions (antennæ, methods of solving equations, digital and analog circuits etc.) have been SURPASSED by GA's, in relatively short amounts of running time. Even better: those GA's are being optimized with a factor 100 as we speak, while Moore's Law keeps on going (we'll see massively parallel computing being mainstream in about ten years).

When I explained our dear friend on rec.games.go that I obviously did not intend to test fitness on flowers but would steer evolution towards correctly solving Go problems or finding Pro moves, he said (I paraphrase) "AHA! Now you're more sensible!".

According to him and others, genetic evolution of a Go program would take "longer than the age of the universe", yada, yada and more of the usual "armchair expert" opinions.

As if I would start with 1 + 1 = 2 and try to let it evolve into a Go program..

Why would I do that? Of course, I would START with the cream of the crop of existing domain knowledge, like a good area estimator, an efficient search algorithm, some fast string property pattern matching code, my own 37 million pattern library, etc. String it together with evolvable weighing factors and make this domain knowledge evolvable. That is the secret to evolving a strong Go program. You have to and tell it how to approximately find the important stuff first. The GA does the rest. The task of the programmer is to find the important domain knowledge and encode this in an "evolvable" way. This code needs to be able to execute very fast, so good coding skills are required otherwise it will indeed take a long time. Optimization techniques will be crucial.

So, instead of endlessly tweaking a carefully weighed hierarchy of elaborate domain knowledge algorithms, you roughly code them up and haphazzardly string them together, but you spend extraordinary attention, effort and time on the efficient evolvability of those modules. It it totally unimportant to get things properly implemented, the only thing that counts is that they are able to evolve towards a good enough solution.

Not Invented Here Syndrome

In my previous post, I mentioned that Moyo Go uses lots of third party components (Moyo Go's sources are almost one million lines of code, less than a tenth written by me).

The advantage is the ability to develop tremendously powerful applications in a hundreth of the time that would be required if I had to develop all those components from scratch - the disadvantage being the infamous "Not Invented Here Syndrome". This well-known syndrome - bugs in Other People's Code - can be ameliorated by Having The Source, which I always do. For those who are interested, I have source licences for:

- The toolbar
- The database system
- The docking system
- The Rich Text editor
- The memory manager
- The splitter used to show/hide the clock
- The HTML renderer
- The Exception tracking/emailing module
- The superfast graphics ibrary

Some of these licences are in the hundreds of USD, some cost peanuts.

In addition to non-freeware libraries, I use drop-in replacement libraries that speed up certain functions, Unicode libraries, a Base64 library, Communication protocol libs, visual component libs, a theming lib, a DirectX lib, a treeview lib, an XML lib and much, much more.

On top of those 3rd-party sources come lots of (mostly paid for) applications like an EXE compressor, an installer, an ISO creator, a memory leak checker and a profiler. Those applications speed up development/deployment. Not to forget Delphi itself. Again, some of these applications are free or cost peanuts, some are many hundreds, or even a few thousand USD (Delphi Enterprise).

Whatever you do when you develop software, ALWAYS make sure you have the sources for EVERYTHING, to be well-armed against Not Invented Here syndrome.

I will switch to an installer to which I will have the source as well, because basically nothing Just Works™, including my 600 USD installer, Ghost Install. I have NEVER seen an installer that Just Works™. This includes Wise and InstallShield. They occasionally REFUSE to create a Setup EXE, or they do, but that EXE does not install a working program (Moyo Go installs around seventyfive thousand files filled with semi-random data, and that excludes the four hundred thousand SGF files included with the product, as they are zipped up in a flat-file SQL databases).

Their tech guys said that it's very possible that their compressor crashes. "Try a lesser compression level", they said. Indeed, that worked. Until it didn't. Reverting to the best compression fixed it again, until I changed a file in the setup, and then it didn't any more, etc.

Developing software is a constant struggle against one's own and other people's bugs, and whatever you do, don't become a victim of Not Invented Here Syndrome..

New Memory Manager

I run Moyo Go Studio like one would run a large software development project: After careful deliberation, I buy whatever is best, if I can't develop it myself. I have spent thousands of USD on third-party components, and today I bought a 150 USD licence for a memory manager, after finally fixing the last (knock on wood) bug in the multiprocessing game import wizard. It turned out I could double the import speed on machines with 4 CPU's or more, if I used a memory manager that was especially designed for SMP.

Origin of the word "Ko"

Young H. Rhie sent me a little gem about the origin of the term Ko, and here's my version of it:

"Originally, Ko is a Buddhist term, the original term is Kalpa. Kalpa describes a long timespan - from the beginning of the cosmos till its destruction.

Another description of Kalpa:

There is a stone measuring 10 x 10 x 10 km. Every 100 years, a fairy's dress gently brushes it. It will take "Kalpa" for the stone to be eroded fully away by her soft garment.

If there would not be a Ko rule in Go, it could take Kalpa before the game finishes.

The Korean term for Ko is Pae. Pae means "Struggle to win". The Korean term is more descriptive than the Japanese and Chinese terms, but the Japanese and Chinese terms for Ko are more philosophical."

Eureka! (continued)

The past 24 hours I have been feverishly trying to find a bug in the multiprocessing code (usually, the last 5% of anything non-trivial I make takes 50% of the time).

Even worse, usually I manage to build horrendously complex things without the slightest problem, only to be stumped by an obscure bug in a seemingly insignificant line of code.

The weird thing was that with 4 CPU's harvesting patterns, substantially less patterns were found than with 1 CPU, in spite of using Critical Sections around global data. For the rest, everything worked fine and dandy.

I finally concluded it must be due to some use of global data that I had missed, which turned out to be true. I confess to occasionally using global data with the aim of simplifying the architecture (DoTheSimplestThingThatCanPossiblyWork), but with the advent of multiprocessing, this is even less of a good idea than it already was :)

Eureka!

The multiprocessing Fuseki & Pattern module is 99% ready and all CPU's are working to extract data from a collection of SGF files! This image is what I see when four lists of SGF files are simultaneously "Fuseki-harvested". The more processor cores in a customers' machine, the faster the game import process will run. This is a major breakthough in scaleability, something that will provide Moyo Go with a significant competitive advantage in the near future, compared to non-parallel processing competitors.