Fixed "Entry duplication".
We fixed a entry duplication bug.
It was caused by the uri's column size which cuts the uri off on 255 byte.
But now, it won't be duplicated.
We changed the column to "text" to store the uri which is over 255 byte and added a column "rui_md5" to store md5ed uri.
we also created unique index on the table.
I hope it will be faster than before to find the specific entry and make crawler faster. :)
0 Comments:
Post a Comment
Subscribe to Post Comments [Atom]
<< Home