How to speed up drupal's slow batch processing
Note committed on 10 August 2011, tagged drupal, drupal 7
I have a task to import pretty huge drupal 5 site into drupal 7. I decided to create a specific import module that would import old drupal site content one by one. Perfect candidate to perform this job seemed to be a drupal's Batch API.
What I've later discovered is that for huge tasks like mine (import several thousands of items) is very slow, probably because of overhead that is created by drupal bootstrap. The thing is that batch api creates a list of functions and parameters for those functions upfront and then executes them one by one. Each step includes the drupal bootstrap with all the regular page processing, and that's what making the overall batch processing slow. I've measured my import function call lasts 0.8s..1s and the overhead is about 5s, for almost every call.
If only it could do series of my import function calls in one session.
Digging the batch API I found this line (file includes/batch.inc, function _batch_process()):
if ($batch['progressive'] && timer_read('batch_processing') > 1000) {
// Record elapsed wall clock time.
$current_set['elapsed'] = round((microtime(TRUE) - $current_set['start']) * 1000, 2);
break;
}
After I changed that 1000 milliseconds in the 'if' condition to something like 30000 ms, I found that the process of importing became ~5 times higher! The source code is pretty self explanatory: if the process takes longer than 1 second, it breaks the execution and waits for another call. I will probably reset this value back when the site goes into production, but for now it is what I was looking for.
======= Add new comment =======
Imagefield: automatically transfer file on selection without clicking upload button
Note committed on 03 April 2011, tagged drupal, drupal 6, js
I wanted file to be uploaded automatically after selecting it, without having to press the Upload button:
Here my solution to this problem, which seems to be working better than the one in the filefield issue queue on d.o:
$(context, '.filefield-upload input.form-file').change(function(context) {
$(context.target).parent().find('input.form-submit:not(.au)').addClass("au").mousedown();
});
}
======= Add new comment =======
Enhancing the drupal statistics: ip geolocation
Note committed on 20 August 2007, tagged drupal, module
Drupal statistics logs every access to a page. It is kind-of useful when the user is logged in, but there are times when you want to see where the user physically comes from. For this situation you can just go into "details" and see his ip address. Copy and paste it into some kind of online ip2location service and see the results.
This code allows you to shorten that cycle to just one click.
Here we will enhance the statistics page, providing link to a ip2location free service
open the statistics.module and find the statistics_recent_hits function. The changes in code are marked bold
....
$sql = 'SELECT a.aid, a.path, a.title, a.uid, u.name, a.timestamp<b>, a.hostname</b> FROM {accesslog} a LEFT JOIN {users} u ON u.uid = a.uid' . tablesort_sql($header);
$result = pager_query($sql, 30);
while ($log = db_fetch_object($result)) {
$rows[] = array(
array('data' => format_date($log->timestamp, 'small'), 'class' => 'nowrap'),
_statistics_format_item($log->title, $log->path),
theme('username', $log)<strong>.", ". l(check_plain($log->hostname), "http://www.ip2location.com/$log->hostname")</strong>,
l(t('details'), "admin/logs/access/$log->aid"));
}
...
after that you will have clickable ip address in the username column.
======= 6 comments =======