View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0027585 | Community | OCCT:Application Framework | public | 2016-06-09 11:29 | 2018-01-29 11:17 |
| Reporter | BenjaminBihler | Assigned To | apn | ||
| Priority | normal | Severity | minor | ||
| Status | closed | Resolution | fixed | ||
| Platform | Windows | OS | VC++ 2015 | ||
| Product Version | 7.0.0 | ||||
| Target Version | 7.2.0 | Fixed in Version | 7.2.0 | ||
| Summary | 0027585: It is not possible to store OCAF documents to paths with special characters in their names | ||||
| Description | Please create the directory "c:\äöüß" and then consider the code pasted to "Steps To Reproduce". If the path string is changed to "c:/aous/example.xml", everything works (if that directory has been created). This is a pity. For being able to use OCCT in many projects, the users need the possibility to store the documents into arbitrary paths on their file systems. | ||||
| Steps To Reproduce | #include <TDocStd_Document.hxx> #include <AppStd_Application.hxx> #include <TDataStd_Name.hxx> int main(int, char**) { Handle(TDocStd_Application) app = new AppStd_Application; Handle(TDocStd_Document) doc; app->NewDocument("XmlOcaf", doc); TDF_Label child = doc->Main().NewChild(); Handle(TDataStd_Name) nameAttribute = TDataStd_Name::Set(child, TCollection_ExtendedString("Test")); PCDM_StoreStatus status = app->SaveAs(doc, TCollection_ExtendedString( "c:/äöüß/example.xml")); std::cout << "Status: " << status << std::endl; if( status != PCDM_SS_OK) { std::cerr << "Store error!" << std::endl; } else { std::cout << "Document is stored." << std::endl; } } | ||||
| Tags | No tags attached. | ||||
| Test case number | Not needed | ||||
| related to | 0026514 | closed | bugmaster | Open CASCADE | OSD_Path can not work with French symbols in file name. |
| related to | 0029081 | closed | Community | Foundation Classes, OSD_OpenStream - handle UNICODE file paths specifically in case of Mingw-w64 | |
| related to | 0029069 | closed | apn | Community | Samples - handle UNICODE filenames within C++/CLI CSharp sample |
|
|
You should not use standart 'char*' when the wide characters are used. For correct non-ASCII symbols management 'wchar_t*' is better. So, this "save" should work in your case: PCDM_StoreStatus status = app->SaveAs(doc, Standard_ExtString(L"c:/äöüß/example.xml")); |
|
|
No, it does not work here. The output is ___________________________________________________________ Resource Manager: Reference file "" loaded Resource Manager Warning: Environment variable "CSF_PluginUserDefaults" not set. Resource Manager: Reference file "" loaded Resource Manager Warning: Environment variable "CSF_StandardUserDefaults" not set. Status: 1 Store error! ___________________________________________________________ If I modify the path to be "c:/äöü/example.xml" and create the directory "c:\äöü", the output becomes: ___________________________________________________________ Resource Manager: Reference file "" loaded Resource Manager Warning: Environment variable "CSF_PluginUserDefaults" not set. Resource Manager: Reference file "" loaded Resource Manager Warning: Environment variable "CSF_StandardUserDefaults" not set. Resource Manager: Reference file "" loaded Resource Manager Warning: Environment variable "CSF_PluginUserDefaults" not set. Error: the file c:/äöü\example.xml cannot be opened for writing Status: 3 Store error! ___________________________________________________________ Does yours really work? I am using revision 264abd72f2508894ff6d514f2d9ff5c2443656f8. Do I need a newer revision for wide character support? |
|
|
Wide characters are supported by OCCT for years. Normally it should not depend on the version of OCCT you use. It is very strange that you have different errors for different paths. Status "1" means that it can not load the storage driver. So, normally this is because of the storage library does not exist or something is wrong with OCCT environment. This error is usually appeared before it starts to check the file path. So, I may guess that something is wrong in environment. I've made a simplest test: created an executable with source codes equal to your (except my fix added: Standard_ExtString(L"c:/äöüß/example.xml") ). Then I've compiled the same version of OCCT as you. To purify tests I've set only two environment variables (in addition to PATH that must refer to OCCT libraries) referenced to standard OCCT resources: CSF_PluginDefaults=<occt>\src\StdResource CSF_StandardDefaults=<occt>\src\StdResource And everything works for either "äöü" or "äöüß" folders. Could you reckeck your environment (even simplify it a lot like me to make debug very simple)? May be you have some your specific development in OCAF storage plugins? Also could you debug what is wrong on your side? You may set a breackpoint in XmlLDrivers_DocumentStorageDriver::Write method and then to go inside fstream "open" method to see that the file name is correct there or where it become wrong in the call stack before? |
|
|
mpv, I have checked the environment variables, I have run the sample application from a fresh console and I am just using the XmlOcaf format, nothing specific. It does not work. Have you tried VC++ 2015 or do you have the possibility to try it quickly? As a next step I will try debugging and then tell you my findings... Thanks for your help so far. |
|
|
I use Visual Studio 2015 64 bits Debug compilation. It works fine. You can send your test program and VS project-files to me to check or if you want I may send mine short example to you. It does not take too much space. |
|
|
This would be great, thank you. |
|
2016-08-17 16:15 developer |
main.zip (14,988 bytes) |
|
|
OK. You can see my "main" solution: just open "main.sln" and update the following options using your paths to OCCT and other products: Debugging/Environment C/C++/General/Additional Include Directories Linker/General/Additional Library Directories Then run in Debug (F5). |
|
|
Thank you for the file. It has ensured that we use the same compiler options and that our source files have the same encoding. But it still does not work here. I have tried it by - unpacking main.zip - entering the right paths into main.vcxproj with my editor - opening the VS2015 developer shell - calling .../vcvarsall.bat x86_amd64 - executing msbuild in the folder with the main.zip contents - set path=%path%;PathToOCCTBin - executing .../x64/Debug/main.exe The output is: C:\Users\benjamin.bihler\Desktop\main\x64\Debug>main TDocStd_Application::SaveAs() - folder c:/õ÷³▀ does not exist Status: -858993460 Store error! What makes me wonder: the file name appears incorrect both in the eclipse console and on the command line. In eclipse it is displayed as "TDocStd_Application::SaveAs() - folder c:/���� does not exist" and on the command line in the way pasted above. I will try to check what happens on other computers (with different configurations)... |
|
|
I guess this is good achievement: I have the same error if folder "c:/äöüß" does not exist. Could you create this folder (with good access rights) and try again? |
|
|
You are right, now it works. :-) During testing I have renamed or deleted the folder and therefore it has failed. Sorry for that. Now it does not work with MinGw64. But I will do research first, what is the problem and then post some details here... Thank you very much for your support! |
|
|
Okay, there is really everything fine with MSVC now. With MinGW-w64 it is not possible to read from or write to files containing special characters in their paths. The reason is in the methods OSD_OpenStream in the file OSD_OpenFile.cxx. There is code to open std::ifstream and std::ofstream objects with wide character file names, but this code is only used, if _MSC_VER is defined, which is of course not the case for MinGW-w64. Therefore the "Linux" code is used there, which seems to be incorrect. |
|
|
Dear Kgv, Could you check the last note from Benjamin: problem with MinGW compiler. It seems this may be related to the issue 0027197 |
|
|
> There is code to open std::ifstream and std::ofstream > objects with wide character file names, > but this code is only used, if _MSC_VER is defined, > which is of course not the case for MinGW-w64. > Therefore the "Linux" code is used there, which seems to be incorrect. The problem is not with OSD_OpenStream(), but with MinGW-w64. It does not provide the same extension as msvc required for proper opening of wchar_t* file names. Of course this extension will be never in C++ standard, but there just no alternative to it on Windows platform. The proper way to handle this issue is pushing MinGW-w64/libstdc++ developers to implement this feature: https://gcc.gnu.org/ml/libstdc++/2011-06/msg00066.html http://mingw-w64-public.narkive.com/4CqOTiLg/w-char-support-in-fstream Alternatively, the issue can be worked around by not using C++ streams at all (makes no sense), or by defining own file stream object. Using (GNU) libstdc++ extension: const TCollection_ExtendedString aFileNameW (theName, Standard_True); FILE* aFile = _wfopen ((const wchar_t*)aFileNameW.ToExtString(), L"r"); __gnu_cxx::stdio_filebuf<char> aFileBuffer (aFile, theMode); std::istream aStream (&aFileBuffer); As you might note, this code defines std::istream, not std::ifstream - there is no way to locally modify OSD_OpenStream.cxx. Therefore, the API should be extended and all places using OSD_OpenStream() should be modified to apply this workaround. Alternatively, for MinGW only, OSD_OpenStream() can fallback to using system locales (thus it will be possible to open UNICODE file names fitting into system locale, but not arbitrary names) and to fallback to short DOS names for opening existing files (luckily, Windows still writes short names to NTFS by default). |
|
|
I have written a mail to the mingw-w64-public mailing list to push them a little bit. If I get a good answer, I will post it here. In the meanwhile this issue should hibernate, shouldn't it? |
|
|
Ok. For me the current status of this issue is ok. |
|
|
I don't want to receive a reminder every day. How can this be deactivated? |
|
|
It seems "acknowledged" status is less noisy. |
|
|
Dear mpv, I have not yet found an answer for how to read in wchar_t* file names with MinGW (I will try to have a look at it within the next months). But the example we have discussed a few months ago does not work anymore also with VS2015 (64 Bit) since revision fb0b05319f52b12ba581e70f561234fe582c3dc3. Is this intentional? I have not found any upgrade information about that. Benjamin |
|
|
Dear mpv, the VC++2015 sample program output with the named OCCT revision is c:/���/example.xml Resource Manager: Reference file "Standard" loaded Resource Manager Warning: Environment variable "CSF_StandardUserDefaults" not set. TDocStd_Application::SaveAs() - folder c:/��� does not exist Status: -263571552 Store error! The code and the environment variables are the some, only the OCCT revision has changed. If I modify the directory name to contain no special characters it works with either revision. Also I have tried to write to files with special characters on Linux. I didn't manage (with neither OCCT revision). Is it supported? Is the following command wrong for Linux code? PCDM_StoreStatus status = app->SaveAs(doc, Standard_ExtString(L"/home/user/äöü/example.xml")); Thank you again for your help. Benjamin |
|
|
> Is the following command wrong for Linux code?PCDM_StoreStatus status = app->SaveAs(doc, Standard_ExtString(L"/home/user/äöü/example.xml")); This line makes no sense - you are casting 4-byte wchar_t string into 2-byte Standard_ExtString and the characters encoding depends on C++ compiler. |
|
|
Dear kgv, I realize that I am completely unsure how to create the correct TCollection_ExtendedString variables. I have tried to find documentation for that, but section 3.4 of occt_foundation_classes.pdf (which seems to be the right place) didn't really help me. On Windows using wchar_t and casting it to Standard_ExtString is correct, isn't it? What about Linux? Could you give me a hint how to do (as a definition in the code or if I have a QString variable that I would like to convert to TCollection_ExtendedString) that or which documentation to look at? Thank you very much, Benjamin |
|
|
I have checked the IESample on Linux. There if I import a STEP file I can only export it to paths that do not contain special characters. Otherwise on the console the error "Step File could not be created : -PATH-" appears and on the UI an error message dialog "The translation is not done." appears. I don't know how to deal with that. Isn't this a major bug? Or is it just a weak implementation of the sample? What would be the correct implementation? |
|
|
> On Windows using wchar_t and casting it to Standard_ExtString is correct, isn't it? OCCT 7.1.0 introduced TCollection_ExtendedString and TCollection_AsciiString constructors taking directly wchar_t* as argument - with automatic conversion from UTF-32 (Linux) and from UTF-16 (Windows). So you just need to drop direct (static) casting to Standard_ExtString type, which has a fixed size in 2 bytes on both Windows and Linux. However on Linux/Unix nobody uses wchar_t - there are usually no file opening methods (save the OS X, but there no much use from it as well). Modern Linux/Unix systems define system local to UTF-8, so that passing char* transparently is sufficient and does not require any extra attention if you are not using obscure or bad-configured Linux. > if I have a QString variable QString is usually converted like this: QString theFile; TCollection_AsciiString anUtf8Path (theFile.toUtf8().data()); TCollection_ExtendedString anUtf16Path (theFile.toUtf8().data(), true); // make sure to set isMultiByte flag in TRUE |
|
|
> I have checked the IESample on Linux. > There if I import a STEP file I can only export it to paths that do not contain special characters.
status = writer.Write( (Standard_CString)file.toLatin1().constData() );
It is a bug in IESample which should be fixed - converting into latin charcode is incorrect. |
|
|
It doesn't work here, neither on Windows (with MSVC) nor on Linux. I have checked my Linux locale, it is set to German/UTF8. I don't see any error here, the only possible difference might be that I use Qt5 on Linux. I feel bad if I bother you always with my code - so let's switch to your code! ;-) If you don't mind I will create an issue about the broken IESample (at the end of this week), fix it with your code lines posted above and then if it does not work here reproducably I will ask you to try whether it really works on your installation. Thank you so far for your support. |
|
|
kgv, you have been right. With the conversion given by you above I can convert QStrings to OCCT strings such that STEP and IGES import and export (and other methods expecting UTF-8 CStrings) succeeds. I have tried using TCollection_AsciiString and also TCollection_ExtendedString which I have converted to UTF-8-CStrings. This code works now in the OCCT samples and in my application both on Windows (with MSVC) and on Linux. :-) Still the problem reported above has stayed the same (both on Windows (MSVC) and on Linux with the slightly modified file path definition). Because of the success reported above I am now quite sure that there is no problem with my system configuration. I guess that there is a bug somewhere in TDocStd_Application. Please let me post my code sample here again. The environment variables "CSF_PluginDefaults" and "CSF_StandardDefaults" are set to the "src\StdResource" folder of the OCCT installation. --------------------------------------------- #include <TDocStd_Document.hxx> #include <AppStd_Application.hxx> #include <TDataStd_Name.hxx> int main(int, char**) { Handle(TDocStd_Application) application = new AppStd_Application; // Create first document Handle(TDocStd_Document) document; application->NewDocument("XmlOcaf", document); TDF_Label child1 = document->Main().NewChild(); Handle(TDataStd_Name) nameAttribute1 = TDataStd_Name::Set(child1, TCollection_ExtendedString("Test1")); const TCollection_ExtendedString utf16Path( Standard_ExtString(L"c:/äöüß/example.xml")); char* name = new char[utf16Path.LengthOfCString() + 1]; utf16Path.ToUTF8CString(name); std::cout << "Writing to " << name << std::endl; delete[] name; PCDM_StoreStatus status = application->SaveAs(document, utf16Path); if (status != PCDM_SS_OK) { std::cerr << "Store error!" << std::endl; } else { std::cout << "Document is stored." << std::endl; } } --------------------------------------------- If I run the program the output is: --------------------------------------------- Writing to c:/äöüß/example.xml Resource Manager: Reference file "Standard" loaded Resource Manager Warning: Environment variable "CSF_StandardUserDefaults" not set. TDocStd_Application::SaveAs() - folder c:/���� does not exist Store error! --------------------------------------------- Everything works fine, if I replace c:/äöüß by c:/aous (of course both directories must exist). Would you mind trying again whether this code snippet works for you? If it really does, could you give me a hint how I could spot the problem? Thank you. |
|
|
I have reproduced the problem with CAD Assistant on Windows - while opening (although saving works without issues) XBF file within exactly specified file path "C:\äöüß\555555.xbf"; will check what can be a reason. |
|
|
It worked in 7.0.0, so, it seems this is a regression. |
|
|
As stated in my commment from 10th of January, it has been introduced with commit fb0b05319f52b12ba581e70f561234fe582c3dc3. |
|
|
The problem appears due to implicit UTF-16 -> Ascii conversion within TCollection_AsciiString constructor taking TCollection_ExtendedString argument - for this specific UNICODE string the check in TCollection_ExtendedString::IsAscii() returns TRUE.
inline Standard_Boolean IsAnAscii(const Standard_ExtCharacter achar)
{
return ! ( achar & 0xff00 );
}
This check is no good, because all UTF-16 code points within the string are below 255. |
|
|
Branch CR27585 has been created by kgv. SHA-1: 7d2fa06ace5fd98e284eb685533b96ff66ee3465 Detailed log of new commits: Author: kgv Date: Fri Jan 13 20:24:11 2017 +0300 0027585: It is not possible to store OCAF documents to paths with special characters in their names TCollection_AsciiString - fixed inproper convertion from UTF-16 string. |
|
|
Patch is ready for review. |
|
|
No remarks, please test |
|
|
Dear BugMaster, Branch CR27585 from occt git-repository (and master from products git-repository) was compiled on Linux, MacOS and Windows platforms and tested. SHA-1: 7d2fa06ace5fd98e284eb685533b96ff66ee3465 Number of compiler warnings: occt component: Linux: 0 (0 on master) Windows: 0 (0 on master) MasOS: 0 (0 on master) products component: Linux: 63 Windows: 0 MacOS: 1145 Regressions/Differences: Not detected Testing cases: Not needed Testing on Linux: Total MEMORY difference: 93298133 / 92917397 [+0.41%] Total CPU difference: 21010.370000000203 / 21506.2300000003 [-2.31%] Testing on Windows: Total MEMORY difference: 58632277 / 58649007 [-0.03%] Total CPU difference: 19555.0061517985 / 19749.742200098677 [-0.99%] |
|
|
Branch CR27585 has been deleted by kgv. SHA-1: 7d2fa06ace5fd98e284eb685533b96ff66ee3465 |
|
occt: master bda0b631 2017-01-13 17:24:11 Committer: apn Details Diff |
0027585: It is not possible to store OCAF documents to paths with special characters in their names TCollection_AsciiString - fixed inproper convertion from UTF-16 string. |
Affected Issues 0027585 |
|
| mod - src/TCollection/TCollection_AsciiString.cxx | Diff File | ||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2016-06-09 11:29 | BenjaminBihler | New Issue | |
| 2016-06-09 11:29 | BenjaminBihler | Assigned To | => mpv |
| 2016-08-01 16:25 |
|
Note Added: 0056338 | |
| 2016-08-01 16:25 |
|
Status | new => feedback |
| 2016-08-04 08:42 |
|
Assigned To | mpv => BenjaminBihler |
| 2016-08-09 16:32 | BenjaminBihler | Note Added: 0056597 | |
| 2016-08-09 16:32 | BenjaminBihler | Assigned To | BenjaminBihler => mpv |
| 2016-08-11 16:05 |
|
Note Added: 0056654 | |
| 2016-08-11 16:05 |
|
Assigned To | mpv => BenjaminBihler |
| 2016-08-16 19:24 | BenjaminBihler | Note Added: 0056769 | |
| 2016-08-17 15:50 |
|
Note Added: 0056791 | |
| 2016-08-17 15:53 | BenjaminBihler | Note Added: 0056792 | |
| 2016-08-17 16:15 |
|
File Added: main.zip | |
| 2016-08-17 16:26 |
|
Note Added: 0056793 | |
| 2016-08-17 16:58 | BenjaminBihler | Note Added: 0056797 | |
| 2016-08-17 17:07 |
|
Note Added: 0056798 | |
| 2016-08-17 17:29 | BenjaminBihler | Note Added: 0056800 | |
| 2016-08-23 15:35 | BenjaminBihler | Note Added: 0056973 | |
| 2016-08-23 15:36 | BenjaminBihler | Assigned To | BenjaminBihler => mpv |
| 2016-08-25 17:44 |
|
Note Added: 0057077 | |
| 2016-08-25 17:44 |
|
Assigned To | mpv => kgv |
| 2016-08-25 17:44 |
|
Status | feedback => assigned |
| 2016-08-26 00:41 | kgv | Note Added: 0057084 | |
| 2016-08-26 00:42 | kgv | Assigned To | kgv => mpv |
| 2016-08-26 01:01 | kgv | Note Edited: 0057084 | |
| 2016-08-26 01:03 | kgv | Note Edited: 0057084 | |
| 2016-08-26 09:48 |
|
Assigned To | mpv => BenjaminBihler |
| 2016-08-26 09:48 |
|
Status | assigned => feedback |
| 2016-08-29 11:36 | BenjaminBihler | Note Added: 0057246 | |
| 2016-08-29 11:57 |
|
Note Added: 0057248 | |
| 2016-08-29 12:27 | BenjaminBihler | Note Added: 0057250 | |
| 2016-08-29 12:38 |
|
Note Added: 0057251 | |
| 2016-08-29 12:38 |
|
Status | feedback => acknowledged |
| 2016-11-09 11:13 |
|
Target Version | 7.1.0 => Unscheduled |
| 2017-01-10 18:23 | BenjaminBihler | Note Added: 0062467 | |
| 2017-01-11 12:21 | BenjaminBihler | Assigned To | BenjaminBihler => mpv |
| 2017-01-11 13:45 | BenjaminBihler | Note Added: 0062487 | |
| 2017-01-11 13:45 | BenjaminBihler | Status | acknowledged => feedback |
| 2017-01-11 13:50 | kgv | Note Added: 0062488 | |
| 2017-01-11 14:25 | BenjaminBihler | Note Added: 0062489 | |
| 2017-01-11 15:21 | BenjaminBihler | Note Added: 0062494 | |
| 2017-01-11 15:33 | kgv | Note Added: 0062496 | |
| 2017-01-11 15:34 | kgv | Note Edited: 0062496 | |
| 2017-01-11 16:06 | kgv | Note Added: 0062500 | |
| 2017-01-11 17:08 | BenjaminBihler | Note Added: 0062509 | |
| 2017-01-13 16:54 | BenjaminBihler | Note Added: 0062584 | |
| 2017-01-13 17:48 | kgv | Note Added: 0062588 | |
| 2017-01-13 17:48 | kgv | Note Edited: 0062588 | |
| 2017-01-13 17:54 |
|
Note Added: 0062589 | |
| 2017-01-13 17:54 |
|
Assigned To | mpv => kgv |
| 2017-01-13 17:54 |
|
Status | feedback => assigned |
| 2017-01-13 17:59 | BenjaminBihler | Note Added: 0062590 | |
| 2017-01-13 19:19 | kgv | Note Added: 0062593 | |
| 2017-01-13 19:19 | kgv | Assigned To | kgv => BenjaminBihler |
| 2017-01-13 19:19 | kgv | Status | assigned => feedback |
| 2017-01-13 19:19 | kgv | Target Version | Unscheduled => 7.2.0 |
| 2017-01-13 20:22 | kgv | Relationship added | related to 0026514 |
| 2017-01-13 20:25 | git | Note Added: 0062597 | |
| 2017-01-13 20:26 | kgv | Note Added: 0062598 | |
| 2017-01-13 20:26 | kgv | Assigned To | BenjaminBihler => abv |
| 2017-01-13 20:26 | kgv | Status | feedback => resolved |
| 2017-01-16 11:10 |
|
Note Added: 0062610 | |
| 2017-01-16 11:10 |
|
Assigned To | abv => bugmaster |
| 2017-01-16 11:10 |
|
Status | resolved => reviewed |
| 2017-01-16 17:29 |
|
Assigned To | bugmaster => apv |
| 2017-01-17 13:44 |
|
Test case number | => Not needed |
| 2017-01-17 13:46 |
|
Note Added: 0062700 | |
| 2017-01-17 13:46 |
|
Assigned To | apv => bugmaster |
| 2017-01-17 13:46 |
|
Status | reviewed => tested |
| 2017-01-20 16:12 | apn | Changeset attached | => occt master bda0b631 |
| 2017-01-20 16:12 | apn | Assigned To | bugmaster => apn |
| 2017-01-20 16:12 | apn | Status | tested => verified |
| 2017-01-20 16:12 | apn | Resolution | open => fixed |
| 2017-02-08 11:35 | git | Note Added: 0063587 | |
| 2017-09-04 17:30 | BenjaminBihler | Relationship added | related to 0029081 |
| 2017-09-29 16:17 |
|
Fixed in Version | => 7.2.0 |
| 2017-09-29 16:29 |
|
Status | verified => closed |
| 2017-10-09 09:53 |
|
Relationship added | has duplicate 0029069 |
| 2018-01-29 11:17 | kgv | Relationship replaced | related to 0029069 |
