View Issue Details

Related ChangesetsJump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0030964Open CASCADEOCCT:Data Exchangepublic2019-09-13 14:552020-12-02 17:12
Reporterkgv Assigned Tobugmaster  
PrioritynormalSeverityfeature 
Status closedResolutionfixed 
Target Version7.5.0Fixed in Version7.5.0 
Summary0030964: Data Exchange - use Standard_ReadLineBuffer within OBJ reader
DescriptionFast prototype shows that using Standard_ReadLineBuffer may give 3x time performance boost on reading of large OBJ files (~ 1 GiB):
  struct RWObj_ReaderFile : public Standard_ReadLineBuffer
  {
...
    bool ReadLine()
    {
      size_t aLineLen = 0;
      int64_t aReadBytes = 0;
      Line = Standard_ReadLineBuffer::ReadLine (File, aLineLen, aReadBytes);
      Position += aReadBytes;
      return Line != NULL;


To complete the patch, Standard_ReadLineBuffer or its sub-class should be extended by handling of special multi-line case with \ at the end of the line.

Steps To ReproduceNot required
TagsNo tags attached.
Test case numberde_mesh/stl_read/D1

Relationships

parent of 0031213 closedbugmaster Open CASCADE Coding Rules, RWStl_Reader - THE_BUFFER should NOT be a global object 
parent of 0031309 closedkgv Open CASCADE Data Exchange - RWObj_Reader fails to read mh03.obj with multi-line syntax 

Activities

git

2019-09-16 15:59

administrator   ~0087144

Branch CR30964 has been created by mzernova.

SHA-1: 5a7808160e8d767d50348786e73ac79d45bfabe3


No new revisions were added by this update.

git

2019-09-24 14:30

administrator   ~0087478

Branch CR30964 has been updated by mzernova.

SHA-1: dd692d8b7b2129b5480d911ec45431dd0c8777d3


Detailed log of new commits:

Author: mzernova
Date: Tue Sep 17 16:05:05 2019 +0300

    0030964: Data Exchange - use Standard_ReadLineBuffer within OBJ reader
    
    The sub-class of Standard_ReadLineBuffer has been created. It supports a processing of the special multi-line case with \ at the end of the line.

kgv

2019-09-24 14:54

developer   ~0087482 

+//! Auxiliary tool for buffered reading of lines from input stream.
+class Standard_ReadLineBufferMultiline: Standard_ReadLineBuffer
+{

It would be more straightforward adding Boolean flag to Standard_ReadLineBuffer rather than introducing new global class.

git

2019-10-02 17:49

administrator   ~0087702

Branch CR30964 has been updated by mzernova.

SHA-1: 9d8ff4fa57c27a1791cf563194561f778bb502a6


Detailed log of new commits:

Author: mzernova
Date: Thu Sep 26 17:18:49 2019 +0300

    Standard_RedLineBuffer was used to load Stl files

git

2019-10-02 17:55

administrator   ~0087713

Branch CR30964_1 has been created by mzernova.

SHA-1: d1fbcd38aeb9728e0b87c04ea3a5d9b49eb5a4c5


Detailed log of new commits:

Author: mzernova
Date: Tue Sep 17 16:05:05 2019 +0300

    0030964: Data Exchange - use Standard_ReadLineBuffer within OBJ reader
    
    Standard_ReadLineBuffer now supports a processing of the special multi-line case with \ at the end of the line.
    
    Standard_RedLineBuffer was used to load Stl files

user897

2019-10-02 17:57

  ~0087714

The patch CR30964 is ready to review

kgv

2019-10-02 18:13

developer   ~0087715

Please put some comparison numbers for STL reader here.

kgv

2019-10-02 18:15

developer   ~0087716 

,
+                        bool theMultiline = false

I think it would be more convenient putting this as class field with getter/setter instead of method argument.

kgv

2019-10-02 18:17

developer   ~0087717 

+      if (Line != NULL)
+      {
+        delete [] Line;
+      }

This looks broken.

git

2019-10-04 15:45

administrator   ~0087816

Branch CR30964_1 has been updated by mzernova.

SHA-1: ac59d773bd7de802b8b3c095b4ab6f0eae903e9e


Detailed log of new commits:

Author: mzernova
Date: Thu Oct 3 16:40:20 2019 +0300

    remarks from kgv

git

2019-10-04 15:55

administrator   ~0087819

Branch CR30964_2 has been created by mzernova.

SHA-1: a51c3f01780df30d2792240e076b45908fdfb70f


Detailed log of new commits:

Author: mzernova
Date: Tue Sep 17 16:05:05 2019 +0300

    0030964: Data Exchange - use Standard_ReadLineBuffer within OBJ reader
    
    Standard_ReadLineBuffer now supports a processing of the special multi-line case with \ at the end of the line.
    
    Standard_RedLineBuffer was used to load Stl files

user897

2019-10-04 16:14

  ~0087822

Time spent reading stl file:
    master: read in 211.481 s
    prototype: read in 89.087 s

user897

2019-10-04 16:16

  ~0087824

Please review the branch CR30964_2

abv

2019-10-06 09:29

manager   ~0087883

Kirill, please comment on why Standard_ReadLineBuffer is placed to package Standard instead of OSD or NCollection where we usually put this kind of tools?

kgv

2019-10-06 09:42

developer   ~0087885

Last edited: 2019-10-06 09:44

 > Kirill, please comment on why Standard_ReadLineBuffer
> is placed to package Standard
> instead of OSD or NCollection where we usually put this kind of tools?
Because
- Standard_ArrayStreamBuffer and Standard_ReadBuffer of similar kind are already in Standard,
- they are not Operating-System-Dependent to be placed into OSD
- and they are not much templates to be placed into NCollection.
One can say there is also FSD (File-System-Dependent), but this package has been used for very strange classes so far.

git

2019-10-17 15:29

administrator   ~0088259

Branch CR30964_2 has been updated by mzernova.

SHA-1: d7ae4f6fed57aa859834073ad4067151f530105b


Detailed log of new commits:

Author: mzernova
Date: Thu Oct 17 15:19:19 2019 +0300

    remarks from osa

git

2019-10-17 15:30

administrator   ~0088260

Branch CR30964_3 has been created by mzernova.

SHA-1: 079d222802b6f1ddb0c98c928bbbb340647b334c


Detailed log of new commits:

Author: mzernova
Date: Tue Sep 17 16:05:05 2019 +0300

    0030964: Data Exchange - use Standard_ReadLineBuffer within OBJ reader
    
    Standard_ReadLineBuffer now supports a processing of the special multi-line case with \ at the end of the line.
    
    Standard_RedLineBuffer was used to load Stl files

osa

2019-10-22 14:16

developer   ~0088419

>> if (isMultiline && myIsMultilineMode)
Check only isMultiline (it is true only when myIsMultilineMode is true)

>> if (myReadBuffer[myBufferPos] == '\n')
this condition should be as else if?

>> bool GetMultilineMode () const { return myIsMultilineMode; }
not protected

>> const Standard_Integer theMaxLineLen = 256
keep this input parameter and use it in Standard_ReadLineBuffer aBuffer (256); but not const value

>>Standard_Boolean isEmptyLine = false;
rename to isEOF

>>Standard_Real x, y, z;
use gp_XYZ aReadVertex

git

2019-10-28 12:15

administrator   ~0088630

Branch CR30964_3 has been updated by mzernova.

SHA-1: 41bf138b18e8e726d4561dba4154be985461fc15


Detailed log of new commits:

Author: mzernova
Date: Mon Oct 28 12:11:10 2019 +0300

    remarks from osa

user897

2019-10-28 12:17

  ~0088631

Please review the branch CR30964_3

kgv

2019-10-28 22:55

developer   ~0088641

Please don't forget switching bug state and assignee when asking for review.

user897

2019-10-29 12:35

  ~0088645

The patch CR30964 is ready to review

git

2019-11-05 14:43

administrator   ~0088728

Branch CR30964_3 has been updated by mzernova.

SHA-1: d899fa273a9b0ec9fe3feda14f09eaf47e7e0a28


Detailed log of new commits:

Author: mzernova
Date: Fri Nov 1 12:03:14 2019 +0300

    now only sequences of the form \\n and \\r\n are deleted

user897

2019-11-05 14:50

  ~0088729

Please review the branch CR30964_3

kgv

2019-11-05 14:59

developer   ~0088730 

  33     // allocate read buffer
  34     myReadBuffer.resize (theMaxBufferSizeBytes);
...
  84         // read new chunk from the stream
  85         if (!readStream (theStream, myReadBuffer.size(), myBytesLastRead))
...
 124           else if (myReadBuffer[myBufferPos + 1] == '\n')
 125           {
 126             myReadBuffer.erase (myReadBuffer.begin() + myBufferPos, myReadBuffer.begin() + myBufferPos + 2);

As readStream() fills entire myReadBuffer (and there is no extra garanteed '\0' tail in myReadBuffer), I have some doubts that myReadBuffer[myBufferPos+1] and myReadBuffer[myBufferPos+2] are protected from out-of-range fetches.
Please check and handle correctly, if issue exists.

git

2019-11-07 10:48

administrator   ~0088769

Branch CR30964_4 has been created by osa.

SHA-1: ee8c0ff78b0658129b8e8d30ebe9f650e2aa52ad


Detailed log of new commits:

Author: mzernova
Date: Tue Sep 17 16:05:05 2019 +0300

    0030964: Data Exchange - use Standard_ReadLineBuffer within OBJ reader
    
    Standard_ReadLineBuffer now supports a processing of the special multi-line case with \ at the end of the line.
    
    Standard_RedLineBuffer was used to load Stl files

git

2019-11-08 11:27

administrator   ~0088786

Branch CR30964_3 has been updated forcibly by mzernova.

SHA-1: a62dd29246b17f2bdfc894f60d23d869c5297211

git

2019-11-08 11:28

administrator   ~0088787

Branch CR30964_4 has been updated forcibly by mzernova.

SHA-1: 68abbaa061bbfa1902112af58f2aa38dee0df8e0

mzernova

2019-11-08 17:20

developer   ~0088811

Please review the branch CR30964_4

git

2019-11-15 14:29

administrator   ~0088997

Branch CR30964_3 has been updated by mzernova.

SHA-1: 92f19dc3a73808f3b2b80b63a4da639a8764c63e


Detailed log of new commits:

Author: mzernova
Date: Fri Nov 15 14:17:52 2019 +0300

    remarks from osa

git

2019-11-15 14:29

administrator   ~0088998

Branch CR30964_4 has been updated forcibly by mzernova.

SHA-1: 1b68edc14c2f3022d82ebaf75befe97fe7f5cee4

git

2019-11-15 14:50

administrator   ~0089001

Branch CR30964_4 has been updated forcibly by mzernova.

SHA-1: 753543924628bb94ba0badbc9ae32b16a2144b9c

git

2019-11-15 19:05

administrator   ~0089014

Branch CR30964_4 has been updated forcibly by mzernova.

SHA-1: ae6ece4dd312095c2d95753325167ed2fe794c17

git

2019-11-18 13:20

administrator   ~0089047

Branch CR30964_4 has been updated forcibly by mzernova.

SHA-1: 6f0f5e9f160ea3f6ea0231bb4d2624f41be41152

mzernova

2019-11-18 15:02

developer   ~0089051

http://vm-jenkins-test-12.nnov.opencascade.com:8080/view/CR30964_4-master-mzernova

mzernova

2019-11-18 15:02

developer   ~0089052

Time spent reading stl file (~ 4 GiB):
    master: read in 125.035 s
    prototype: read in 43.338 s

Time spent reading obj file (~ 1 GiB):
    master: read in 32.414 s
    prototype: read in 18.481 s

osa

2019-11-19 17:26

developer   ~0089078

The patch was reviewed

git

2019-11-20 14:11

administrator   ~0089096

Branch CR30964_3 has been updated forcibly by mzernova.

SHA-1: f468a435e276766d2302e21525c928e650010d3a

git

2019-11-20 14:11

administrator   ~0089097

Branch CR30964_4 has been updated forcibly by mzernova.

SHA-1: abfdff19af498a5c2a04db7bd7c940a4cd99bc5d

bugmaster

2019-11-21 09:28

administrator   ~0089115

Combination -
OCCT branch : WEEK-47
master SHA - 51ee6a7dbb3a740ba0cfc693ce5e7a83663f05f4
5f5b1aed1c6e139bbd34314eca77ae7abcd8895c
Products branch : WEEK-47 SHA - 1fe84f5f3e15bca9d1c6f8e9646341c4df675aec
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.

Number of compiler warnings:
No new/fixed warnings

Regressions/Differences/Improvements:
No regressions/differences

CPU differences:
Debian80-64:
OCCT
Total CPU difference: 16799.850000000122 / 16823.120000000046 [-0.14%]
Products
Total CPU difference: 10810.050000000052 / 10787.080000000053 [+0.21%]
Windows-64-VC14:
OCCT
Total CPU difference: 18294.203125 / 18299.59375 [-0.03%]
Products
Total CPU difference: 12837.546875 / 12853.21875 [-0.12%]


Image differences :
No differences that require special attention

Memory differences :
No differences that require special attention

git

2019-11-24 11:35

administrator   ~0089173

Branch CR30964_3 has been deleted by inv.

SHA-1: f468a435e276766d2302e21525c928e650010d3a

git

2019-11-24 11:35

administrator   ~0089174

Branch CR30964_4 has been deleted by inv.

SHA-1: abfdff19af498a5c2a04db7bd7c940a4cd99bc5d

git

2019-11-24 11:35

administrator   ~0089182

Branch CR30964_2 has been deleted by inv.

SHA-1: d7ae4f6fed57aa859834073ad4067151f530105b

git

2019-11-24 11:35

administrator   ~0089183

Branch CR30964_1 has been deleted by inv.

SHA-1: ac59d773bd7de802b8b3c095b4ab6f0eae903e9e

git

2019-11-24 11:35

administrator   ~0089184

Branch CR30964 has been deleted by inv.

SHA-1: 9d8ff4fa57c27a1791cf563194561f778bb502a6

Related Changesets

occt: master 51ee6a7d

2019-09-17 13:05:05

mzernova


Committer: bugmaster Details Diff		 0030964: Data Exchange - use Standard_ReadLineBuffer within OBJ reader

Standard_ReadLineBuffer now supports a processing of the special multi-line case with \ at the end of the line.

Standard_RedLineBuffer was used to load Stl files		 Affected Issues
0030964
mod - src/RWObj/RWObj_Reader.cxx Diff File
mod - src/RWStl/RWStl_Reader.cxx Diff File
mod - src/Standard/Standard_ReadLineBuffer.hxx Diff File
mod - tests/de_mesh/stl_read/D1 Diff File

Issue History

Date Modified Username Field Change
2019-09-13 14:55 kgv New Issue
2019-09-13 14:55 kgv Assigned To => kgv
2019-09-13 14:55 kgv Assigned To kgv => user897
2019-09-13 14:55 kgv Status new => assigned
2019-09-13 15:02 kgv Description Updated
2019-09-16 15:59 git Note Added: 0087144
2019-09-24 14:30 git Note Added: 0087478
2019-09-24 14:54 kgv Note Added: 0087482
2019-10-02 17:49 git Note Added: 0087702
2019-10-02 17:55 git Note Added: 0087713
2019-10-02 17:57 user897 Note Added: 0087714
2019-10-02 17:57 user897 Assigned To user897 => osa
2019-10-02 17:57 user897 Status assigned => resolved
2019-10-02 17:57 user897 Steps to Reproduce Updated
2019-10-02 18:13 kgv Note Added: 0087715
2019-10-02 18:15 kgv Note Added: 0087716
2019-10-02 18:17 kgv Note Added: 0087717
2019-10-04 15:45 git Note Added: 0087816
2019-10-04 15:55 git Note Added: 0087819
2019-10-04 16:14 user897 Note Added: 0087822
2019-10-04 16:16 user897 Note Added: 0087824
2019-10-06 09:28 abv Description Updated
2019-10-06 09:29 abv Note Added: 0087883
2019-10-06 09:42 kgv Note Added: 0087885
2019-10-06 09:43 kgv Note Edited: 0087885
2019-10-06 09:43 kgv Note Edited: 0087885
2019-10-06 09:44 kgv Note Edited: 0087885
2019-10-17 15:29 git Note Added: 0088259
2019-10-17 15:30 git Note Added: 0088260
2019-10-22 14:16 osa Note Added: 0088419
2019-10-22 14:16 osa Assigned To osa => user897
2019-10-22 14:16 osa Status resolved => assigned
2019-10-28 12:15 git Note Added: 0088630
2019-10-28 12:17 user897 Note Added: 0088631
2019-10-28 22:55 kgv Note Added: 0088641
2019-10-29 12:35 user897 Note Added: 0088645
2019-10-29 12:35 user897 Assigned To user897 => osa
2019-10-29 12:35 user897 Status assigned => resolved
2019-11-05 14:43 git Note Added: 0088728
2019-11-05 14:50 user897 Note Added: 0088729
2019-11-05 14:59 kgv Note Added: 0088730
2019-11-07 10:48 git Note Added: 0088769
2019-11-07 12:05 osa Assigned To osa => mzernova
2019-11-07 12:05 osa Status resolved => assigned
2019-11-08 11:27 git Note Added: 0088786
2019-11-08 11:28 git Note Added: 0088787
2019-11-08 11:29 mzernova Assigned To mzernova => osa
2019-11-08 11:29 mzernova Status assigned => resolved
2019-11-08 17:20 mzernova Note Added: 0088811
2019-11-15 14:29 git Note Added: 0088997
2019-11-15 14:29 git Note Added: 0088998
2019-11-15 14:50 git Note Added: 0089001
2019-11-15 19:05 git Note Added: 0089014
2019-11-18 13:20 git Note Added: 0089047
2019-11-18 15:02 mzernova Note Added: 0089051
2019-11-18 15:02 mzernova Note Added: 0089052
2019-11-19 17:26 osa Note Added: 0089078
2019-11-19 17:26 osa Assigned To osa => bugmaster
2019-11-19 17:26 osa Status resolved => reviewed
2019-11-20 14:11 git Note Added: 0089096
2019-11-20 14:11 git Note Added: 0089097
2019-11-21 09:28 bugmaster Note Added: 0089115
2019-11-21 09:28 bugmaster Status reviewed => tested
2019-11-21 09:29 bugmaster Test case number => de_mesh/stl_read/D1
2019-11-24 11:21 bugmaster Changeset attached => occt master 51ee6a7d
2019-11-24 11:21 bugmaster Status tested => verified
2019-11-24 11:21 bugmaster Resolution open => fixed
2019-11-24 11:35 git Note Added: 0089173
2019-11-24 11:35 git Note Added: 0089174
2019-11-24 11:35 git Note Added: 0089182
2019-11-24 11:35 git Note Added: 0089183
2019-11-24 11:35 git Note Added: 0089184
2019-12-03 12:19 kgv Relationship added parent of 0031213
2020-01-20 18:07 kgv Relationship added parent of 0031309
2020-12-02 16:40 emo Fixed in Version => 7.5.0
2020-12-02 17:12 emo Status verified => closed