Month: June 2019

Sphider comes with the ability to backup and restore your database. How well this works depends on not only the size of the database, but on your MySQL settings. The restore could restore a single record at a time, but this would be time consuming. It would be reliable, but for a larger database you could probably spend the weekend at the shore while it ran. So, to speed things up, the restore process works on blocks of records. However, this increase in speed comes with a cost. If a block or records is too big, the restore will fail. There is a way to prevent this.

First off, check to see if you might have an issue. From a command prompt:
mysqld –help –verbose –pid-file

In the results, look for “max_allowed_packet”. If the value is less than 33554432 (32M), you might have an issue. Values of 67108864 (64M) or greater and you should be good to go.  The 64M is recommended, although larger won’t harm a thing! The value can be up to a maximum of 1G (1073741824).

If you need to increase the value of “max_allowed_packet”, there are two ways of doing so. The first is a permanent fix. Edit my.cnf (my.ini in Windows). In the “[mysqld]” or “[client]” section,  put in “max_allowed_packet=64M”. If the line doesn’t exist, add it. Then restart the mysql service.

The second method is temporary, existing until the next time the service is restarted. Run this simple query:
SET GLOBAL max_allowed_packet=67108864

Of course, in either instance, entering larger numbers will do no harm. More importantly, you can have confidence that the backup and restore procedures will work properly.

 

The Sphider database, like most databases, has relationships between various tables. Unlike most databases, however, these relationships have never been defined within the database! Sure, tables have had their keys, but there were no foreign key constraints. All relationships between tables and the consequences of record deletions or modifications have been handled strictly by the Sphider code. MySQL can handle this more efficiently than any Sphider code ever could.

At this point, adding foreign key constraints is really a straight forward task, since for the typical user, the tables all have data. It would be easy for new installations, but the presence of data complicates things. What has been done is to develop a method to back up the database, destroy and recreate all the tables (with foreign key constraints), and then repopulate the tables with the backed up data. The existing backup procedure was not an option because when a restore is run from that, the database would revert to one without the constraints. Then, Sphider code can be REDUCED in size by a couple hundred lines to eliminate code that handles relationship changes that MySQL can do more efficiently.

The process of building a process to back up the data, recreate the database with constraints, and restore the data has been completed. Now it has to be thoroughly tested. Early tests are positive, but we want to be sure. Once we have a high level of confidence in the process, Sphider 3.3.0-MB will be released. The intent is that the ONLY changes in 3.3.0 will be the database structure and associated code related to structure.

Look for Sphider 3.3.0-MB towards the end of July.