|
|
|
smart quotes strike again
Monday, February 12 2018
This afternoon it was sunny enough and warm enough for it to feel like there was a thaw underway. The rumpled glass of ice on the walkway to the front door crunched underfoot as it was continually undermined by running water, and there was enough water crossing the driveway for me to have trouble getting through in my Crocs when I went to check the mail.
In the course of my workday, I learned something interesting about the storage and presentation of unicode characters by MySQL. If you use MySQL LENGTH() function on a string containing weird unicode characters, it will return the number of bytes used. So if three of the characters are normal ASCII and one is unicode character requiring three bytes, then the string's length will be six. But if that unicode character is the second one in the string and one does a ASCII(SUBSTR(string, 2, 1)) on the string, one will only get the first byte of the three making up the character. All of this came to a head today when investigating problems with last names containing an apostrophe (such as O'Connell). Some of these names have proved impossible to search for, and today I discovered the reason: an apostrophe wasn't actually in the name, instead it was a "smart quote," which is one of those quotes curved in the typopgraphically-appropriate way for the context. Microsoft applications love smart quotes and junk up data with them at every opportunity (it's a battle I used to fight when converting Microsoft Word documents into web pages). In this case, the offending application appeared to be coming from Excel. Why would Excel insist on replacing apostrophes with smart quotes if it made it difficult to later find the records contaminated in this way? Perhaps this is Microsoft's way of locking customers into using their database technologies, as they may have been designed to find records containing smart quotes when apostrophes are specified. In any case, it's just another case of Microsoft making the computational environment just a little more polluted than it otherwise would be.
[REDACTED]
For linking purposes this article's URL is:
http://asecular.com/blog.php?180212 feedback
previous | next |
