NDL2DB History
21.8.2011 v2.11
* fixed db functions inclusion
16.8.2011 v2.10
* php 5.3 compatibility fixes
* configurator2: sub class open fixed
* replaced ereg() with preg_match()
28.4.2010 v2.03-RC1
* fixed php5 errors and warnings (KvE)
- replaced eregi_replace("..." by prep_replace("/.../i"
- replace eregi("..." by preg_match("/.../i"
- Where necessary / in expressions escaped by changing to \/
- comment out register_long_arrays and magic_quotes_gpc in php.ini
26.9.2008 v2.02-beta01
* statistics: NL_LINE reads is the same as total reads.
corrected counting.
25.09.2008 v2.02-beta01
* func_nf1: $st = $state||'Online'; line 765 seems to be that doesnt works.
state Down hasnt been inserted into the db. Keyword parsing and State
parsing now fixed.
* record hashes: nl_node includes keword and nl_line includes state
02.06.2008 v2.01-alpha10
* Once more NL_ENTRY insert errors:
- Warning: record allways exists in NL_ENTRY: 127, 714056.
- ERROR: insert record failed NL_ENTRY: insert into nl_entry
(id_nodelist, id_nl_line) VALUES (127,684034)
- ERROR: SQL error: (1062) Duplicate entry '127-714094' for key 1
Problem occures when running 2 tasks and 2 tasks works
on the same nodelist:
# 01:43:46 [1] nodelist import (1) of: 'nodelist.006' (1995)
# 03:20:54 [1] Warning: One or more record insert errors.
Possible DB inconsistency.
# 01:44:53 [0] nodelist RE-import (1) of: 'nodelist.006' (1995)
# 03:21:57 [0] Warning: One or more record insert errors.
Possible DB inconsistency.
The workqueue update routine now checks the sql_affectedrows()
after changing the state of the affected nodelist to prevent
duplicate execution on the same nodelist at the same time.
State: fixed.
* nl_line dupes: id_nl_line 766420 + 766421
2x 358895:74521f51f7adb69789351bbce29f17db
same id_nl_node, but only one id_nl_line in nl_entry.
procedure nl_line produces dupe entries (about 10%, 5800 and more
within half a million nl_line records), that needs to
checked and eliminated (dupeelim maintenance) -
results in a table restructure: nl_line index set to unique
multiple inserts ends in an sql insert error that can be
detected. State: fixed.
28.04.2008 v2.01-alpha09
* Performance counters analyze results, that minimum time that is
needed to read hash array values compares to the maximum readin
time that is needed to readin a db record.
This results in code rewrite by removing hash arrays
(reduced memory usage, reduced access time).
27.04.2008 v2.01-alpha08
* runtime conflicts inserting new records and identifying
the IDs of the inserted records. Relates also to the
nl_entry dupes. Search actual records by their hash key. Fixed.
* hash statistics: add read hash from disc counter
26.04.2008 v2.01-alpha07
* Undefined variable: sres1 func_nf1.php on line 529
Fixed.
* Undefined offset: 10339 in func_nf1.php on line 529
under some conditions. Fixed by isset().
25.04.2008 v2.01-alpha06
* changed table structure: nl_wqueue, add #nodelist lines
+ if multiple same sort order level exists, added addtl.
sort order by year, day
25.04.2008 v2.01-alpha05
* modified sord order to 1 for single update to workqueue.
22.04.2008 v2.01-alpha05
* multitasking import: new import and re-import at the same time
occures. A timing problem in search next workqueue entry
and set it to under construction. Recheck state on updating
record.
21.04.2008 v2.01-alpha05
* Re-import of nodelist creates 2nd record in nodelist
with new id_nodelist. Nodelist-found flag name mistyped. Fixed.
20.04.2008 v2.01-alpha05
* using strict in array_search() to prevent
insert into db errors.
* on insert into db failures skip following
db inserts for a single nodelist line because
related keys are missing and inserts fails too.
+ added a FORCE import switch
+ added type 4 insert: insert a single nodelist to DB
w/o workqueue handling
12.04.2008 v2.01-alpha04
* Notice: Use of undefined constant PROG - assumed 'PROG' in
func_lib line 462
func_lib and constants inclusion needed before starting logfile()
* Notice: Undefined offset: 7497 in ndl2db line 402
now checking array index with isset(). Fixed.
* import_file: unable to identify '...'
packeridentifier() => current(), next(), key() array functions
doesn't work under php 4.4.1
replaced by foreach() loop. Fixed.
10.04.2008 v2.01-alpha04
+ Problem: execute workqueue deletes all records in workqueue
no log message why records are deleted.
The machine has rebooted, the drive mapping hasn't been
reestablished. Missing source drive prevents the script
to import the source files from within the
workqueue handle loop, so the records gets deleted
w/o any further infos.
Added global check that the source path is accessible
otherwise the script terminates. Fixed.
09.04.2008 v2.01-alpha4
* multiple task run produces nl_entry sql insert with
duplicate id_nodelist, id_nl_line pairs. As of a result,
the array_push()/array_slice()/array_splice() combination
handled thru an bulk array of 5000 elements hasn't been
inserted and updated. About 20 of 5000 pairs was
dupes. Now inserting each single id_nodelist, id_nl_line
pair with checking if pair is allways in the db. Fixed.
08.04.2008 v2.01-alpha04
* type 2+3: check if allways in workqueue results 0 forever
as a wrong resultset were used that produced dupe entries. fixed.
- removed inclusion for unused library func_dli
- wqorder: removed inclusion for unused library func_dli
07.04.2008 v2.01-alpha03
+ add SID to log lines to identify session# in multitasking
environments
* type 2+3: search or add nodelists to workqueue
added check in nfmt1() if nodelist allways in workqueue
07.04.2008 v2.01-alpha02
* create flag dir if not exists
+ add install.txt after some failures and lost records
in a production nl_wqueue
06.04.2008 v2.01-alpha01
+ added wqorder.php Workqueue ordering script
05.04.2008 v2.01-alpha01
+ added workqueue handling
- scan to workqueue
- execute from workqueue (*default)
- add single file to workqueue
+ identify type 1 nodelist 4d (new format, with zones)
or nodelist 2d (old format, w/o zones)
* simplier process indicator
+ added session handling and process termination handling
to interrupt long scan processes by semaphore files
(needs directory 'flag')
30.03.2008 + added AKA, Sysops name and Systemname queries.
+ added addtl. query links into the output form
for related queries.
29.03.2008 i first web query plattform (NLARCHIVE2 script)
02.03.2008 * modified nfmt1(), added save records nl_node, nl_line,
nl_entry controlled by hashes.
i full Nodelist import tests successful.
i started full NODELIST import from Fido-History-Project Archiv
01.03.2008 + added func_lib.php, moved out global sub functions from func_dli
+ added save records nl_nodelist (includes hashes)
+ added doy2date() (daynumber and year to date)
26.02.2008 * modified extractLastname()
24.02.2008 + added extractLastname()
22.02.2008 * import_if_nodelist()
+ added inspectfmt(), different nodelist formats switch
+ added subfunction func_nf1.php nfmt1()
21.02.2008 * import_if_nodelist()
+ added Nodelist admin line check
+ added CRC16() calculation and check
20.02.2008 + added import_subdir()
19.02.2008 i Starting 2nd port to PHP
+ added global logfile()
+ added compress.cfg parser
+ added packeridentifier()
+ added tempdir handling, create, cleanup, remove
+ added unpackndl2tmp()
14.02.2008 i Found Sources from 4th Dec 07
09.02.2008 i Harddisc crash, all sources v1.0 lost.
31.01.2007 i Version 1.0 of Port Perl to PHP Nodelist Importer
Started Nodelists import to FidoHis2 database.
04.12.2007 i Version 1.0 of Port Perl to PHP
Nodelist Importer for Fido-History-Project Node-
and Pointlists to MySQL5
30.11.2007 - v1.0 (PHP)
i Port to PHP
22.02.2007 i Suggestion by Alex Woick of DB structure
with a PL script for import nodelist data