Special characters converted to question marks on save

If you enter something in a different language that uses UTF-8 to display, such as Chinese (這是一個考驗) and you get nothing but ????? then you’re not UTF-8 compliant.

If you dont’ see chinese in my post right now, then it’s your browser. If you do see chinese then it might be your database.

It’s definitely not my browser. The special characters show up in the WordPress edit window but then turn into question marks when I press Save.

Yes, I see Chinese characters. I’m in Ubuntu and my computer and browser can handle Unicode just fine.

Well, something in your environment is NOT handling utf-8 “just fine” – otherwise you wouldn’t be here asking…

Are you using the wysiwyg thingy or is it turned off?
Is this a brand new WP install or was it an upgrade? If upgrade did it work before/ever?

Is your database your local database?
Do you have control over it or is it hosted for you?
Do you know if IT can handle utf-8 characters?

as Moshu said, somethings wrong as wordpress handles these things perfectly.

Well, something in your environment is NOT handling utf-8 “just fine” – otherwise you wouldn’t be here asking…

Everything is just fine as far as my personal computer displaying Unicode. I can see Chinese, Greek, Math, Arabic, Hebrew, Cyrillic, etc. Ubuntu and Firefox are very good about this. The problem is with WordPress or the server that WordPress is running on.

Are you using the wysiwyg thingy or is it turned off?

Yes, I am generating these with the Insert custom character button with an Ω on it.

Is this a brand new WP install or was it an upgrade? If upgrade did it work before/ever?

This does not work on my standard install, which has been upgraded several times, and it also does not work on a fresh install that I just created.

When I create custom characters using the tool in the WYSIWYG editor, they display fine, but when I press Save, they are converted into question marks.

On a brand new fresh install, if I copy and paste these characters into the Post box:

& ” ￠ € ￡ ￥ ? ? TM ‰ μ · ? … ′ ″ § ? ?
? ? ? ? ‘ ‘ ” ” ? ? < > ≤ ≥ – – ˉ ￣ ¤ |
¨ ? ? ? ? ° – ± ÷ ? × 1 2 3 ? ? ? ? ∫ ∑
∞ √ ≈ ≠ ≡ ∏ ? ∩ ? ′ ? a o ? ? à á ? ? ?
? ? ? è é ê ? ì í ? ? D ? ò ó ? ? ? ? ?
? ù ú ? ü Y ? T à á a ? ? ? ? ? è é ê ?
ì í ? ? e ? ò ó ? ? ? ? ? ? ù ú ? ü y t
? Α Β Γ Δ Ε Ζ Η Θ Ι Κ Λ Μ Ν Ξ Ο Π Ρ Σ Τ
Υ Φ Χ Ψ Ω α β γ δ ε ζ η θ ι κ λ μ ν ξ ο
π ρ ? σ τ υ φ χ ψ ω ← ↑ → ↓ ? ? ? ? ?

I get these after pressing Save:

& ” ￠ € ￡ ￥ ? ? TM ‰ μ · ? … ? ? § ? ?
? ? ? ? ‘ ‘ ” ” ? ? < > ? ? – – ˉ ? ¤ |
¨ ? ? ? ? ° – ± ÷ ? × 1 2 3 ? ? ? ? ? ?
? ? ? ? ? ? ? ? ? ′ ? a o ? ? à á ? ? ?
? ? ? è é ê ? ì í ? ? D ? ò ó ? ? ? ? ?
? ù ú ? ü Y ? T à á a ? ? ? ? ? è é ê ?
ì í ? ? e ? ò ó ? ? ? ? ? ? ù ú ? ü y t
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?
? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

Is your database your local database?

I don’t know what that means.

Do you have control over it or is it hosted for you?

WordPress was installed by Fantastico on cPanel. I have as much control over it as cPanel gives.

Do you know if IT can handle utf-8 characters?

Do I know if a database can handle UTF-8 characters? I don’t know what you’re asking.

Is your host a windows machine or a linux machine? What exactly is it?
What version is your mysql database?
What version is your PHP?
Do you have access to your mysql.ini file?

These are very basic questions you’ll need to be able to answer for us to help you.

And I suggest you read up on UTF-8 to understand more about the problem:

https://en.wikipedia.org/wiki/UTF-8

I know about UTF-8 already. I don’t know about mysql or PHP. I’ll try to find this out.

Apache version 1.3.37 (Unix)
MySQL version 4.1.22-standard-log
PHP version 4.4.6

I can access phpMyAdmin, which shows my WordPress databases? I have three installations, including a brand new one, and it lists _wrdp1, _wrdp2, _wrdp3. Inside are “Tables” with “Collations”, some of which are utf8_general_ci and some of which are latin1_swedish_ci. On the brand new installation, they are all latin1_swedish_ci.

if you have myadmin for mysql that will help greatly.

Go into the wordpress database for the one that you know you have entered multibyte characters and look at the wp_posts table.

the post_content field will have the content of that post and should contain those characters.

Are they ???? or actual characters?

Or if you create a brand new post and save it as a draft, making sure its the only draft you can run the query;

select post_content from wp_posts where post_status = 'draft';

If the symbols in the database are ????? then it could be that the database isn’t correctly set to utf8 or it could be that the multibyte string stuff for php isn’t working.

DO you have access to a phpinfo() page?

If you do have access to a phpinfo page do a search for “mbstring”.. “Multibyte Support” must be enabled.

(Someone correct me if thats incorrect.. I assume you guys use mbstring for all the multibyte stuff as opposed to rolling your own.. )

some of which are utf8_general_ci and some of which are latin1_swedish_ci. On the brand new installation, they are all latin1_swedish_ci.

I think the problem is collation. You should have all your DB utf8_general_ci. Otherwise your posts are saved wrong in the DB which then appearing unreadable in your blog.

You ‘ll find more about converting DB characters sets HERE.

PS: That might happened because you installed WP via Fantastico. So also check you config.php

Go into the wordpress database for the one that you know you have entered multibyte characters and look at the wp_posts table.

Are they ???? or actual characters?

They are question marks in the phpMyAdmin display of wp_posts.

DO you have access to a phpinfo() page?

If you do have access to a phpinfo page do a search for “mbstring”.. “Multibyte Support” must be enabled.

‘–enable-mbstring’ ‘–enable-mbstr-enc-trans’ ‘–enable-mbregex’

mbstring
Multibyte Support enabled
Japanese support enabled
Simplified chinese support enabled
Traditional chinese support enabled
Korean support enabled
Russian support enabled
Multibyte (japanese) regex support enabled

mbstring.internal_encoding ISO-8859-1

I think the problem is collation. You should have all your DB utf8_general_ci.

They are apparently set as latin1_swedish_ci by default?

PS: That might happened because you installed WP via Fantastico.

Ah, ok.

So also check you config.php

What am I checking for?

In a default installation you should have these two lines in your config.php

define(‘DB_CHARSET’, ‘utf8’);
define(‘DB_COLLATE’, ”);

Your DataBase and all your tables should be utf8_general_ci and this is crucial for all of us using other than latin characters. Backup and then read the link i provided you.

Shouldn’t WordPress be able to handle characters generated by itself??

WordPress gets and displays what’s stored in your DataBase. If you see unreadable characters in your DataBase then these characters appear in your posts.

Yep deuced is on to it.. Nice catch man.