Colinkri is the new indexing tool that is application called from within SEO Autopilot. Here we take a look at what the tool is, and how it works as well as some results.
SEO Autopilot Update 1.0.3.7 “Colinkri”
SEO Autopilot now includes an API call to Colinkri.
Colinkri is a web-based application that coerces Google into crawling web pages and the links on those web pages.
It works very well, but there are two questions that are asked every time the topic of crawling new pages comes up
- What is the different between crawling a web page and indexing links and content on a web page?
- How does Colinkri “force” Google to crawl URL’s with my new links on?
Those are excellent questions. Here’s a video that might help.
In short. Crawling is a visit from the Google bot. Google has, at any one time, it is not known how many iterations of the Google bot are on the web at any one time. It is safe to say the answer is at least in the millions.
These bots (and there are many versions of the bots for different content, different languages and different configurations of web sites) read the HTML and CSS of a page. They look for content, links, META data and other vital aspects of web function.
However, during a crawl, this is all they may do – because crawling is not indexing.
To be indexed Google must find a spot for you somewhere. Under a keyword or key phrase or set of keywords or key phrases, where your new content should go. If it can’t find a spot in its index, or it cannot work out what your content is about -0 it probably won’t get indexed.
Crawling does not equal indexing – but it is a vital first step
Colinkri does not have direct control over Google bots. Of course, it doesn’t. But the developers of this crawling and indexing tool understand what triggers google to visit pages.
The developers use something called the “frontier” The links on the last page that was crawled that are yet to be visited and crawled themselves.
So, imagine if you had a set of hundreds or thousands of URL’s that you KNEW Google crawled regularly. Perhaps every few minutes?
Well established pages, ones with authority and some level of profile that Google visited often
The challenge would then be to add links to your new pages (that c0ontain your links and content) to these highly visible, often crawled pages “on the fly” so that these frontier pages are always pointing to your latest links. Always preparing for the next time the Google bot arrives to direct it towards you (the Colinkri users) latest content and links.
The Colinkri frontier pages being properly maintained. Not appearing to be pages full of spam that Google eventually decides is not with its time to crawl is a challenge.
How the developers overcome obstacles like this is not fully known. Maybe they rotate pages? Perhaps they have so many highly crawled pages that they dilute what might appear to be link spam to an acceptable level.
There is also the possibility of adding the links to often crawled XML sitemaps. This is a trickier process as these maps are designed to give an overview of the site, they are actually on rather than third party sites or links. However, it is feasible to imagine new pages being added to established sites. These pages would contain the links to be indexed and little else, and they could be removed or replaced periodically.
Whatever the exact method – it operates along lines like this, and more importantly -it works.
My testing has shown that Colinkri increases the crawl rate on a random set of new content pages and links from around 35% in the first 14 days to almost 90% in the first 14 days.
It increases the indexing rate from under 28% to over 80% in that same time period.
That’s a huge win. That’s almost a 3-fold increase in link visibility, and Colinkri is incredibly easy to use and works automatically with new links inside SEO Autopilot
You got it. And thanks – I appreciate the comment and the sentiment. 🙂
I never really thought of indexing like that. It does require continuous growth to work. The more links, the more Google needs to consider those links. So crawling might see the links, it might even count the links in terms of page rank but getting the links actually INDEXED means there has to be a space someone on Google for them. Someone somewhere – when they type in a query, one of the URL’s Google lists, even if it’s the last one out of 10,000 – has to have the URL with your link on it. That’s the difference between crawled and indexed. So obvious, yet I never considered it.
Thanks again Scritty. Top explanation of a process that always slightly confused me
Thanks Dave. You got it. 🙂
Does the indexer run after all the tasks are complete, or does it do it immediately as it goes? I’ve been running SEO Autopilot all morning and it hasn’t run any URLs through Colinkri yet so I was just wondering at what point are the links sent through Colinkri. The API is connected I’ve double verified. Thanks!
There is always a delay. I use Colinkri outside SEO autopilot so the application data isn’t masked by the transparent connection to SEO – AP. Typically it takes 6-24 hours to crawl and those that are going to be indexed are usually to be found 2-3 days later with stragglers coming through after 7 days.