Today and the previous few days have brought an interesting problem. When given a choice of options, where some choices are better than others, how do you try and find the optimum one? This is at its heart a mathematical problem above all else but what makes it interesting is its applicability to everyday life and the variety of approaches you can take to solving it.
Imagine that you want to find the best ice-cream flavor combination possible (or at least what you think is the best). Let's say you have 20 different flavors to choose from and you get to pick 2. But that's not all an ice-cream is, what about the right cone? Or a cup? That adds maybe 4 different possibilities of let's say 3 cones or getting a cup. Finally, you have to choose the sprinkles and that adds at least 5 more possibilities. This gives you a LOT of choices in total (if my math is correct: 3800).
So how do you go about choosing which one is best. My personal favorite method would be to try every combination because I love ice-cream, but as far as my waist-line is concerned, that may not be the best option. Another option might be to start with what your favorites are, for me that would be an easy cookie dough and chocolate brownie and then I can change up the sprinkles and cone how I see fit. The problem with this is that my assumption that my favorite would include cookie dough and chocolate brownie may be flawed (although I personally doubt it). Perhaps there exists a more delicious sorbet combination or perhaps there are two other flavors that go together better, like maybe just cookie dough and cookie dough.
This is the problem that I am currently attempting to solve, or at least, get sort of right when finding the best neural network combination. The difficulty is with something as complicated as AI, finding an optimum point is a little like searching a city block that someone has scattered a variety of lightly radioactive objects in and you're goal is to try and find the most radioactive one with only a Geiger-counter. It's possible, but not exactly easy. Oh and there's always the chance that the whole time the Geiger-counter doesn't actually work because you didn't assemble it properly.
Luckily, mathematics offers some solutions! Unluckily, a lot of them are really complicated and would involve such a radical change in the search algorithm that you would likely break it and not know it until you had already sunk hours or days into the search and I have seen that happen too many times to bother risking it, for now. What does make the problem easier though is that AI structure, unlike ice-cream, (I know, it's shocking that these two things might differ in some way) exist in dimensions, not categories. Flavors are discrete categories and while to some extent you could plot them on an axis of % chocolate or sugar content, in general there is not a consistent relationship between them that would allow you to analyze ice cream numerically and yield useful results. Although as I am writing this I have started to realize that this statement might not be entirely correct and I may have to conduct some experiments as to how the % cookie dough impacts overall flavor. But, returning to neural networks, variables like scanning windows, breadth and depth of layers can all be varied along an axis allowing for one to cheat in how they examine these variables.
If somebody wanted you to find what percentage of cookie dough in ice-cream made it the tastiest (my bet is 50%), then you wouldn't test down to the molecule, you would maybe add in one cup of cookie dough, then two, then three and when you found the best of that sample, you could narrow it down and test only in the region that yielded the best results. If that region were 3 cups, then the next round of testing might be 2.5, 2.75, 3, 3.25, and 3.5 cups. Using this method you can narrow results down after initially getting a ball-park guess and this is exactly the algorithm that I have adopted where it tests broad categories first and then, when I analyze those results, I'll narrow it down further until I find "The chosen one". This network may not be the absolute best possible, but it'll likely be pretty good. And when that happens, you can bet I'll eat a lot of ice-cream.
