Yazoo ---> Online Help Docs ---> Embedding C/C++ into Yazoo: A Neural Network Tutorial

The Anagrambler

Of course, we would still have to run a few more tests to really convince ourselves that NN.zoo is working. In particular, we should try re-initializing the network, and still need to take the run() method out for a spin. As luck would have it the rest of the script turns out to work just fine. So at this point we can close NN.zoo, go back at the command prompt and put our networks to use.

The types of network and learning algorithm we are using are very well suited to the task of memorizing a set of output patterns in association with different inputs. In this example, we'll use our network to unscramble anagrams. There are a number of ways we could imagine doing this, the most obvious being to tell the network how many of each letter is found in the anagram, then read off the word directly from the output. That is, the value of each output neuron could encode a letter by its activity level -- anything less than 0.05 might represent an `a', 0.05 thru 0.1 would be a `b', and so on. But the problem with this approach is that it will spit out garbage (with letters that weren't even in the anagram) unless the network can be trained to very high tolerances. So we'll use a more indirect approach that shows a more graceful performance curve.

Our strategy will be to associate letter order with letter count. The input will be the number of each letter in the anagram (so there will be 26 inputs). The output will, hopefully, be the ordering of the letters relative to an alphabetical ordering of the letters. For example, the output `32415' for the input `ortob' would imply the ordering 3-2-4-1-5 of the characters `b-o-o-r-t', which gives `robot'. Different words require different output sizes, so we should set the number of output neurons to the maximum word length we expect to use. In our example 6 output neurons will be adequate.

To begin with, it's probably good practice to define a working variable that is separate from the template variable, as the latter was really intended to be a class. Ordinarily we might just want to do it from the command prompt, as in Hob :: neural_network. However, in this case we actually want to use the inheritance operator to give it a bunch of routines to read and re-spell words, which makes for a lengthy definition for the command prompt. So let's put it in a separate file instead: say, "Anagrambler.zoo", in the same directory as Yazoo.


    foreach :: {
   
       code
       
       for (c1::ulong) in [1, top(args[1])]
           args(args[1][c1], c1)
       endf
    }
   
    Hob :: neural_network : {
   
       SetupNN :: {
   
           ltr :: string
           params :: { step_size :: learning_rate :: double }
           
           code
           
           params = { .5, .1 }
           if trap( (params<<args)() ) /= passed
               printl("Error: optional params are step_size, learning_rate")
               return
           endif
   
           the_word = args[1]
           foreach(NN_in ;
               ltr =! extract(alph, args[2], args[2])
               args[1] = find(the_word, ltr, 0)   )
   
           NN_out[^size(the_word)]
           NN_out[*].letter =! the_word
       }
   
       ask :: Hob.SetupNN : {
   
           out_str :: string
   
           code
           
           run(NN_in, params.step_size)
           
           sort(NN_out, 2)
   
           NN_out[^outputs_num]
           NN_out[*].order = activity[inputs_num+2, inputs_num+outputs_num+1]
           if size(the_word) < outputs_num, NN_out[^size(the_word)], endif
           
           sort(NN_out, 1)
           out_str =! NN_out[*].letter
   
           print(out_str)
       }
   
       teach :: Hob.SetupNN : {
           c1 :: ulong
           
           code
   
           foreach(NN_out ; args[1].order = args[2]/size(the_word))
           sort(NN_out, 2)
   
           NN_out[^outputs_num]
   
           for c1 in [1, args[2]]
               run(NN_in, NN_out[*].order, params.step_size, params.learning_rate)
           endf
       }
    }
   
   
    the_word :: string
    NN_in[26] :: double
    NN_out[Hob.outputs_num] :: { order :: double, letter :: ubyte }
   
    alph := "abcdefghijklmnopqrstuvwxyz"
   
   

Just know that it would have been possible to do all this from the command line, just inconvenient. With direct command-prompt entry, lines must be separated by commas, with line-breaks only if we use the line-continue marker `&': e.g.


    > foreach :: { ; for (c1::ulong) in [1, top(args[1])], &
    args(args[1][c1], c1), endf }
    ...                 | etc.
   

At last, we're ready to put our newly-constructed brain to the task of unscrambling anagrams. We begin by booting up Yazoo, then running the two zoo source files.


    > run("NN.zoo")
   
    > run("Anagrambler.zoo")
   

Next we'll need to initialize the little brain. Let's decide to work with words of 6 or fewer characters---we can always change this---and we'll expect a 26-character alphabet. So we enter the following line at the command prompt.


    > Hob.init(26, 6, 0)
   

Once everything is loaded into memory, we can try


    > Hob.ask("lleoh")
   
    ehllo
   

Hardly a surprise; we haven't taught it its first word yet.


    > Hob.teach("hello", 10)   | 10 = # training cycles -- mandatory argument
   
    > Hob.ask("lleoh")
   
    hello
   

Cute. So this concludes our Yazoo demonstration. Of course we've barely investigated Hob's intelligence, which is pretty dim (though subtle in surprising ways). A great virtue of the interactive command prompt is that the cycles of investigation are very short. After hello, the author taught Hob yazoo and then tomato (10 cycles apiece), after which the network had gotten foggy on both hello and yazoo. Then a follow-up 5 cycles of `yazoo'-teaching, and Hob was for some strange reason able to recite all three words perfectly. It all took less than a minute.

Another thing to play around with is the learning rate and/or step size for evolving the neurons' activities. If runtime becomes an issue, will we be able to trade a higher learning rate for some training runs? It's simple to script.


    > Hob.teach("hello", 5; learning_rate = that*2)
   

Finally, we could look into build a bigger brain. Increasing the number of output neurons would allow Hob to memorize longer words. Adding hidden neurons would increase both its memory capacity and the depth of its calculations. To explore these possibilities we would just re-initialize the network (Hob.init(...)) with different parameters and repeat the training.

That is the last that we shall say about neural networks -- this is, after all, a Yazoo help file. The tutorial concluded, the remainder of this chapter will explain, rather than show, how in the general case one embeds C or C++ code into Yazoo.


Prev: The Yazoo wrapper: writing and debugging   Next: Rules for embedding C/C++ code


Last update: July 28, 2013

Get Yazoo scripting language at SourceForge.net. Fast, secure and Free Open Source software downloads