I wrote a python program that's parsing structure.rf.u8 file, as surprising as it is, python haven't crashed on any string operation as it should without forcing encoding on them. Looking further into that I discovered that there are none non-ascii characters in that data dump, and there are over 50 thousands records (out of over 760000) that are supposed to have some unicode characters (usually national letters, accents) but have question marks instead.
At first I though that there were some issues with converting that data by my python program, but then I looked into structure.rdf.u8 data dump, and found question marks there as well.
Here are few examples of how that data looks exactly:
Notice the title and dmoz_uri for those category. Within ODP sites the first category has some x-like character instead those two question marks ( @see http://www.dmoz.org/Arts/Animation/Anime/Titles/3/3×3_Eyes/ ).
Since that data looks... well... terribly, I have small question: is it possible to get hands on unicode encoded data not ascii?
At first I though that there were some issues with converting that data by my python program, but then I looked into structure.rdf.u8 data dump, and found question marks there as well.
Here are few examples of how that data looks exactly:
Code:
3??3 Eyes; dmoz_uri: Top/Arts/Animation/Anime/Titles/3/3??3_Eyes; dmoz-id: 425470
DNA??; dmoz_uri: Top/Arts/Animation/Anime/Titles/D/DNA??; dmoz-id: 426485
Kamikaze Kait?? Jeanne; dmoz_uri: Top/Arts/Animation/Anime/Titles/K/Kamikaze_Kait??_Jeanne; dmoz-id: 426017Notice the title and dmoz_uri for those category. Within ODP sites the first category has some x-like character instead those two question marks ( @see http://www.dmoz.org/Arts/Animation/Anime/Titles/3/3×3_Eyes/ ).
Since that data looks... well... terribly, I have small question: is it possible to get hands on unicode encoded data not ascii?
