Hi all,
Since the last update, we have mostly been researching new encoders again. In particular, we are interested in encoders that perform well with recurrent versions of AOgmaNeo. Recurrent branches seem to have better compression on many tasks (but not all), so it will likely be an optional thing to enable.
Two new encoders are looking promising, the more general one being a topology-preserving encoder, and the more narrow one being a "fast weights" encoder.
The topology-preserving encoder is once again based on self-organizing maps, much like our previous experiments in this area. However, this time it minimizes reconstruction error in one pass as opposed to several iterations. This both performs better in terms of encoding quality and also runs a fair bit faster. Right now this encoder is a suitable candidate to replace the old ESR encoder still used in AOgmaNeo at this time: It seems to either perform better or the same on just about every task, and has the added benefit of having a topology-preserving encoding (which can be used for certain invariances).
The second encoder we experimented with is one based on the concept of "fast weights" (paper). Our use is a bit different from that paper - instead of using "fast weights" as a memory mechanism primarily, we wanted to see if we can make an encoder that "forgets on purpose", but in a predictable way. The idea is that if an encoder always forgets in the same way for a certain stream of inputs, the forgetting itself is a form of permanent memory. This idea seems to work well on only a few tasks (on which it outperforms all other encoders though). Until we can figure out what is harming its generality, this encoders is currently not likely to replace ESR.
Demo-wise, this last month has been slow, but we performed a couple of experiments on the Lorcan Mini robot. There are still some issues with out-of-distribution inputs there, but it's getting close to a new video.
We have pushed a new update to AOgmaNeo and PyAOgmaNeo, but it doesn't contain the new encoder just yet (it's just some fixes and a couple of other things). Soon!
Until next time!
We recently published a proof-of-concept demonstration of something called Unsupervised Behavioral Learning (UBL), something we have been researching before and finally got in a good enough condition to do some navigation tasks.
So, we made a little T-maze "rat" demo, where a tiny robot (the "rat") must figure out how to navigate in the T-maze given various "goal states". UBL is a method that allows us to specify arbitrary goal states and the rat will attempt to match them as well as it can. It can be trained completely passively, so we trained it by driving around the maze semi-randomly, and then testing various goal states.
If you want to try it out: The branch "goal4_nodestride" was used to make the video.
Here is the video of that:
Also, we helped out James Bruton in one of his amazing projects on YouTube with AOgmaNeo!
Recently I tried to use the "GAN Theft Auto" sample dataset from YouTuber Sentdex to make my own simulated GTA stretch of road.
Here is his GitHub repository: GANTheftAuto and his original video: video
Of course, instead of using Nvidia's "GameGAN" framework, I use AOgmaNeo!
Using AOgmaNeo for this task has one huge advantage - it runs much, much faster. Sentdex trained on an Nvidia DGX A100 workstation, which is currently one of the most powerful workstations Nvidia makes. With AOgmaNeo however, one only needs a regular desktop CPU.
Here is a link to my results (using AOgmaNeo).
It's not as detailed as that of Sentdex, since I only have the sample dataset and I am also not using upscaling. However, I am surprised it works this well given the vast difference in compute needed.
The code for training uses the Python bindings to AOgmaNeo (PyAOgmaNeo), and is available here.
So this last month there were two major developments for AOgmaNeo.
First, there is a new interactive demo hosted on the Ogma website that you can try. I compiled AOgmaNeo with Emscripten (WebAssembly), and it runs pretty nicely in the browser. The demo is a real robot that was then turned into an web demo by learning a "simulator" for it with AOgmaNeo (simply by observing its response to motor commands through a camera). It's a fun demo that showcases the world-modeling capabilities of AOgmaNeo, give it a try! I called it "real2sim", a reversal of the more common "sim2real" paradigm in machine learning. It still uses the dual-encoder setup discussed in previous blog posts, but we now find that that is not needed, leading to the next point.
Second, new findings show that the dual-encoder setup is actually not strictly necessary. With the inclusion of a new "importance" setting for each IO layer, one can now manually set how much the hierarchy pays attention to certain inputs. This was previously done automatically by the error-driven encoder, but a simple manual setting is more general and can perform tasks the error-driven encoder could not. So, I have released a new version of AOgmaNeo that goes back to the faster single-encoder setup, but included a new function called "setImportance" (there is also "getImportance") that allows the user to control how important an input is to the encoders.
Thinking about what's next, I think I may want to change the allocator used in AOgmaNeo. The memory it uses is static (minus some minor things in the interface), and is heap-allocated when init is called. I feel like I can just use a memory arena here, although it will probably not make much difference it feels like a cleaner solution. I am also working on a new branch that introduces a new algorithmic optimization I call "the topology optimization". This new optimization also has the side benefit of being more cache friendly, due to the simpler memory access pattern.
That's all for now!
Since the last post we have been performing tons of experiments with various improvements to the dual-encoder setup.
Most didn't work, but some made it into the upcoming version of AOgmaNeo.
Importantly, there is now a guide in the AOgmaNeo repository, that provides a brief overview of what AOgmaNeo is and what it does.
It doesn't have code usage yet though, instead it is describes the algorithm. For code usage, the examples are still the main resource at the moment.
We also trained reinforcement learning (RL) agents in the DonkeyCar simulator (Website), which drove around the track quite nicely. Here is the "imagination" of the RL agent creating its own little simulation of the environment:
Finally, we made a Ludum Dare 48 entry that uses AOgmaNeo to control enemies. There wasn't enough time to really get it working well (the creature generation often created immovable creatures), but it was fun regardless! Ludum Dare Link