Special chars in filenames

Hi,

The backup filename should be fine, we do encode them in UTF-8. Make sure you are using the latest version.

If your file has a character that is NOT UTF-8, then the plugin will replace it with a question mark. If you are seeing strange characters and not question marks, it’s probably a problem with your ZIP client not understanding UTF-8. If you let us know what client it is we can have a look if we find time.

If you are seeing question marks though… then it won’t be UTF-8 and it may be your file is using some other character set. When we read files from filesystem we only support ASCII and UTF-8. To support all the character sets, and to have to try and detect them, would be a very difficult thing to do, so we just assume UTF-8. Maybe if you know your filesystem character set we can make an option where you can tell the plugin what it is and it will try to convert the filename from that character set to UTF-8 so you don’t lose these characters – it’s just without knowing or trying to detect the original character set, which is difficult, we won’t be able to maintain them.

Hope this helps.

Regards,

Jason

Hi,
I do not encounter question marks in the filenames. The french “e accent cute” is translated with a “+” sign, followed by a special char (“r” like “registered”). I have verified the file names: they are encoded in UTF-8.
So I agree with you, it is probably the ZIP client. PhpInfo gives following informations:
extension version: $Id: php_zip.c 294817 2010-02-09 17:51:39Z pajoye $
Zip version: 1.9.1
LibZip version: 0.9.0
I am going to inform the sysadmin, perhaps he can do something.
Thank you for your answer,
Bests Regards,

Thierry

Hi Thierry,

Just tested this in some detail. Are you using Windows Explorer to extract the ZIP file? Turns out Microsoft Windows does not support Unicode characters so you’ll never get the character back with it.

Nearly all third-party utilities WILL support it though. If you are using a third-party utility already, just try a different one. I use 7-Zip myself and recommend that, and it is the tool I do most of the testing with.

I’ve also released 3.0.4 which makes the ZIP files we generate more standards-compliant, but I highly doubt this will fix your issue – using another ZIP utility should though.

Regards,

Jason

Hi Jason,
I have installed the 3.0.4 release. Effectively, now, the special chars are correctly displayed in Windows.
But there is a problem: they are not in UTF-8. So when I upload the files on the remote server, all the links will fail because those characters are encoded on 1 byte and not on 2 or more.
I have tried the last version of 7Zip, the problem is the same. I use Windows 7.
Note that when I download the files directly, with a ftp client like Filezilla, the special chars are not correctly displayed. This is normal because Windows does not support UTF-8. But at least, Windows Explorer does not modify the files names. So when we upload them on the remote server, the special chars are correctly encoded and the links are preserved. I am speaking about the french special chars, I do not know if this is true for other languages.

However, with the last version of your plugin, it is now possible to write a batch or a program that reconverts the files names in UTF-8, before the upload.

Perhaps you can find a solution; otherwize, I let you mark this topic as resolved and I will write this batch.

Thank you for your support and your reactivity,
Regards,
Thierry

Hi Thierry,

I will test this. Are you uploading using FileZilla?

The ZIP file we generate contains valid UTF-8 and is 100% to the standard, so your filenames will be in tact.

When you extract, 7-Zip should follow the spec 100% and I believe it does, and it will transcode those UTF-8 names into UTF-16 names (Windows is mainly UTF-16) and store them on the computer. This is why your filenames show correctly in Windows now.

The next thing is the transfer to the server. The software doing the transfer should know that Windows is a UTF-16 filesystem, and if the remote FTP server supports UTF-8 it should transcode the UTF-16 back to UTF-8 and then transfer. The remote FTP server will then transcode the filename to whatever the local filesystem supports (if it is Windows it will transcode back to UTF-16 – for most Linux servers it will simply store as UTF-8.)

This maintains the filename throughout. I suspect the issue is during your transfer back to the server, the transcoding is not happening properly, or the FTP client is Unicode-dumb and is asking Windows for ASCII filenames (thus the scrambled letters) and then converting THOSE to UTF-8 for transfer (or simply transferring those ASCII.)

Let us know your FTP client you use to transfer and I can do some testing. I doubt there is anything we can do on the plugin side to fix it but at least I’ll be able to advise people to avoid any specific broken FTP clients when working with Unicode.

Regards,

Jason

Hi Jason,

Yes, that’s it. I thought that our server accepts UTF-8, but I was wrong; it does not accept non ASCII chars. So the specials chars appear encoded with two ASCII characters.

I suppose this is due for compatibility issues. Our server hosts a lot of old web sites…

So I have to write this batch…

Anyway, this topic is now solved for me.

Thank you for your detailed answers and your reactivity.

Bests regards,

Thierry