up vote 1 down vote favorite
share [g+] share [fb]

Due to some encoding problems, some characters in my database have been interpreted uncorrectly. Some chars have been replaced with the 'replacement character'. I need to find all such chars and replace them with space. I have been googling different solutions using sed, bash, perl, etc with no luck. Could someone share a solution how to find such characters (i.e. looking for this character in hex/octal value) and replace them with any other character?

Thanks.

link|improve this question
74% accept rate
3  
bash is so the wrong tool for this. – Ignacio Vazquez-Abrams Feb 2 at 10:51
2  
Might just possibly be useful if you mentioned some tiny details like what database you're using... – Chopper3 Feb 2 at 13:33
1  
Is your database a file? Can you give an example how you see such replacement characters? – oHessling Feb 2 at 17:35
3  
The Unicode "Replacement Character" glyph is usually not what is in the underlying data storage system: It's normally subbed in at the display layer (because of a decoding failure), and doesn't actually tell you what the underlying code was/is. You need to know that (and what DB you're talking about) in order to go about cleaning things up... – voretaq7♦ Feb 2 at 17:58
Thank you all for the help and replies. – Andrew Feb 2 at 20:40
feedback

1 Answer

up vote 6 down vote accepted

The Unicode "Replacement Character" glyph is usually not what is in the underlying data storage system: It's normally subbed in at the display layer (because of a decoding failure), and doesn't actually tell you what the underlying code was/is.

You have two options here:

The first (and best) is to find out what the underlying glyph/code is and use your database tools in order to cleanin things up.

The second is to export your data through something that does the Unicode conversions, search/replace the "Replacement Character" code there, and re-import your data. This is generally infeasible for all but the smallest data sets (and if the problem is that your workstation can't figure out what to do with the Unicode glyphs it won't fix your problem).

link|improve this answer
Thank you for such a comprehensive answer. You are right, I have dumped my database and found number of different strange characters in my dump. The database seems to have been affected by malicious code. I would not figure out this without your help. – Andrew Feb 2 at 20:39
feedback

Your Answer

 
or
required, but never shown
discard

Not the answer you're looking for? Browse other questions tagged or ask your own question.