Genetic Algorithms Within Evolving Web Search

I’m not here to discuss how major search engines like Google work. This post was born out of the idea to tie the way we, as regular users, utilize search engines in our daily lives to a systematic approach. This way, we can easily make use of the knowledge already available on this topic.

While “Evolving Web Search” may sound a bit complicated, it’s actually not so daunting. You might already be searching this way without even realizing it. I previously wrote about how to conduct a literature search, and since that method is applicable in any context, I am translating it here:

“Suppose I’ve been given a problem to solve. I select a few keywords from the problem and search for them on Google. From the results I get, I pick out new keywords and search with those. I keep repeating this process until I find something useful.”

How does this method align with genetic algorithms? The core of genetic algorithms is closely tied to the evolution of our civilization and animal species. You’ve probably heard the phrase “survival of the fittest.” It means that nature does not favor the weak offspring. As a result, weak creatures are weeded out in every generation, while superior species continue to survive. Now, can you see the resemblance to evolving web search?

A genetic algorithm is a systematic process that mimics the above strategy. There are two excellent articles on Wikipedia regarding this topic, titled Genetic Algorithm and Genetic Programming.

By now, you’ve certainly realized that this web search approach is essentially an application of the genetic algorithm. For your convenience, I’m summarizing the steps of evolving web search here again:

1. Select some keywords related to your topic, even if they seem a bit random.
2. Use these keywords to search.
3. From your search results, choose the best ones. From these good results, select a few more keywords.
4. Use these new strong keywords to search again. Repeat until you find exactly what you’re looking for.
5. If, after searching for quite a while, you still find nothing, stop.

Like genetic algorithms, this approach has a few drawbacks. If your initial keywords are not good, you might never find what you’re looking for. On the other hand, if you never like any of the results, you could keep searching forever and never find anything. Conversely, if you like all the results, you’ll eventually accumulate too much unnecessary information.

You may be surprised at how quickly this method can work. As far as I can recall, I have never needed more than three rounds of searching to find what I need. What I often do is pick a few results after searching a couple of times and start working with them, whether they’re good or bad. After working with this information for a while, if I feel I’ve gained a good understanding, I break down the problem into sub-problems, select some new keywords, and start searching again.

If you’re aware of the potential issues mentioned above, evolving web search could be a valuable tool along your journey.

So tell me, how do you search the web?

(Translation of a post originally published on my English blog)