The Spider & The Algorithm
The Carrot and the Stick
Understanding this natural market preference for the organic result, Google embarked on a graduated incentive program to guide (others might say force) website managers to adopt “best practices” in the content and architecture of web marketing strategy. Thus, the infamous “Google algorithm” became known (refined with Google Panda in 2011) and if you wanted your website to be ranked well organically by the “spider”, you had to keep changing and improving things with each new set of prescriptive commands or risk a painful drop in rankings. As it became more widely accepted and understood this bucking bronco was here to stay, the “SEO specialist” was born.
The Spider
The spider or “crawler” (of which there are many) is an ever-evolving mathematical “cyber mechanism” that systematically re-scans the entire World Wide Web (or specific countries) indexing and assessing each page of every single website it touches. When the Google spider indexes a site, it employs the latest algorithm to measure and ultimately compare the level of compliance of that particular website with the current and historical recommendations Google has publicly announced to all. Since the first generation spider literally took months to index the entire World Wide Web (today it takes only hours), the changes made to websites back then took a while to gain traction—and when they did—it was often already time for Google’s next “recommended” change(s).
Steady Evolution
Besides proper keyword use, the first big push in these early days promoted the adoption of “backlinks”. The greater the number of websites which linked to your website (implying that you were a trusted and established web presence), the more points you were awarded by that ever-crawling spider. When the spider determined that the desired proportion of websites had finally achieved sufficient numbers of backlinks, the algorithm was then changed to award fewer points for backlinks and more for company YouTube videos. This cycle continued to encourage best practices adoption of social media, Google+, Google Maps listings, Google reviews, Google Street View, and so on with the latest push mandating “responsive” websites that work properly on smartphones and tablets for mobile searches or your Google rankings will tank no matter how good your previous compliance to date.
300 Year Plan
Google has a 300 year plan to organize the world’s information so it knows where we’re headed to compete effectively in the future World Wide Web. The way they’ll make it happen will be to keep prodding us along by constantly adjusting our rankings while measuring incremental compliance with what it asks/tells us to do. For non-technical business managers stuck in traditional advertising modes it’s a genuine nightmare. For competent and incompetent web marketers alike, Google has become the greatest make-work project of all time.