Month: February 2022

Sphider can index using a sitemap — PROVIDED it is a traditional sitemap of url’s and not a sitemap directory listing additional sitemaps (which contain the url’s). This is popular on larger websites.

Well, we have been playing with a mod that can change that! Initial tests show that just might actually work! We have found one instance that can mess up the process and have disarmed it. The question is, are there other instances that can derail us? Only extensive testing will tell.

We will post the mod in the Sphider Help Forum, but will also provide it here.

In spiderfuncs.php, find the function getSiteMap(). Modify the function with the bold code as follows:

function getSiteMap($input_file)
{
$links = '';
$sitemap = simplexml_load_file($input_file);
if ($sitemap != '') {
$links = array ();
foreach ($sitemap as $url) {
// START MOD PART 1
// For some reason, wlwmanifest.xml interfers with the recursion
// Therefore, let's ignore it
if (preg_match("/wlwmanifest\.xml$/i", $url->loc)) {
continue;
}
if (preg_match("/\.xml$/i", $url->loc)) {
$submap = $url->loc;
foreach ($submap as $input2) {
$sitemap2 = simplexml_load_file($input2);
if ($sitemap2 != '') {
foreach ($sitemap2 as $url2) {
$links[] = ($url2->loc);
}
}
}
} else {
// END MOD PART 1
$links[] =($url->loc);
// START MOD PART 2
}
// END MOD PART 2
}
$links = explode(",", (implode(",", $links)));
}
return $links;
}

Let us know if you try this, and ESPECIALLY if there are issues!

IS Sphider obsolete!?

NO! At least not yet. Sphider is on the road to obsolescence, but it’s not there just yet. Before going into further detail, I wish to point out a few things.

The intended use for Sphider is for a website to have an internal search feature for that particular site. Sphider is, and never was, intended to be a personal Google, Bing, Yahoo, Yandex, or any other search engine. Yes, it is capable of indexing more than a single website, but even there it is intended for indexing perhaps a family of related sites.

Next, keep in mind that Sphider first debuted when the web was a much simpler place. Websites consisted of a series of files and sub-directories (some might refer to them as folders). A website had a home page, often named “index.html”, or “index.php”, or “index.aspx”.  There might be a directory named “products” and files in that directory like “product1.htm” and “product2.htm”. You would access these pages from a browser with something like ‘http://bigfactory/products/product1.htm”. For many websites today, this is still a valid scenario. Maybe “https” has largely replaced “http”, but it is still the same concept.

The reality, though, is that the web is changing. Take this blog, for example. It uses WordPress, and in a pretty basic, almost primitive, way. There are quite a number of pages. There is even a contact page, which judging from what appears at the top of the browser, is located in a directory named “contact”. But you know what? There is no such directory! There is an “index.php”, but it doesn’t contain anything like what you see on the home page of this blog. Since this blog is not very complex in the way it is laid out, Sphider can index it, although the results are rather messy! That is okay, since WordPress has it’s own search functionality if the user wishes to implement it.

You will notice that the “downloads” page of this blog has a url of “https://www.blog.worldspaceflight.com/downloads/”. There is no name in the traditional sense, no page extension (htm, php, etc.). It looks like it is a directory, so the default would be “index.php” or something? NOPE!

This isn’t just a WordPress thing. This is the future of the internet. As time goes by, more and more websites are going to become like this. Cpanel settings, htaccess settings, iframes, api’s, server configurations… These all are evolving.

So what does all this have to do with Sphider? Sphider uses old technology, technology which is still in large use today. But that use is diminishing. Sphider is going to try to index some websites and immediately end with a “Relocation: 301” message and never get a step further. So why can’t Sphider simply follow the 301 and start indexing that page? Because it is a 301 that only Sphider can see. It isn’t a REAL 301. There is no redirect header, no redirect in htaccess (Apache servers). This is all in configuration. Sphider needs a file name, and increasing there just simply is no file name. It’s a modern website using features Sphider is not equipped to handle.

So is Sphider dead? No. Is Sphider dying? No, Sphider is not dying, but the universe in which it works is definitely shrinking. As websites evolve, the number of websites able to utilize Sphider is going to decrease.

So what about Sphider now? What is its future?

I don’t see any feature changes or additions in the future. Sphider will continue to be supported and updated to keep up with the technology it does use. Sphider works with PHP 8.1 and MySQL 8. As PHP evolves, Sphider will keep pace. The same goes for MySQL. Sphider will keep up. If any hidden flaws are found in the code, it will be corrected. If security issues are detected, we will attempt to address them.

As the web evolves, there may come a time in five or ten years, when Sphider becomes a quality buggy whip in a Tesla world. Even then you will still be able to find it residing in some antique software repository. But it isn’t quite ready to hang it all up just yet.