View Issue Details

Jump to NotesJump to History
IDProjectCategoryView StatusDate SubmittedLast Update
0023979Open CASCADEOCCT:Data Exchangepublic2013-05-21 16:572024-02-01 02:46
ReportersdvAssigned Todpasukhi  
PrioritynormalSeverityminor 
Status assignedResolutionopen 
Target VersionUnscheduled 
Summary0023979: Data Exchange - Performance problem of STEP export for large files
DescriptionIn order to reproduce the bug you can use attached scripts. Scripts t1, t3, t40, t60, t200 load from BRep file 1000, 3000... 200000 prisms, convert them to STEP format and save the result.
Analyzing convertation with Intel VTune Amplifier I have not found significant hotspots (except hotspot described in the issue 23974). Also I have analyzed convertation with using CallCount. Comparing logs for 1000, 3000 and more shapes, does not detect unlinear dependencies.
If you run script and open Windows Task Manager, you will see reducing of the memory. When the memory will be completely allocated, all open applications become unresponsive.
Setting flag MMGT_OPT does not give visible improving of perfomance.
Steps To Reproducepload ALL
autodisplay 0

compound c

chrono h restart
set n 200
set m 500
for {set i 0} {$i < $n} {incr i} {
  for {set j 0} {$j < $m} {incr j} {
    set xbase [expr $i*10]
    set ybase [expr $j*10]
    set size 5
    vertex v1 $xbase $ybase 0
    vertex v2 [expr $xbase+$size] $ybase 0
    vertex v3 [expr $xbase+0.5*$size] [expr $ybase+0.866*$size] 0
    edge e1 v1 v2
    edge e2 v2 v3
    edge e3 v3 v1
    wire w e1 e2 e3
    mkplane f w
    prism p f 0 0 10
    add p c
  }
}
chrono h stop counter buildshape

newmodel
chrono h restart
stepwrite 0 c 100K.step
chrono h stop counter stepwrite

meminfo
TagsNo tags attached.
Test case numberperf de bug23979

Attached Files

  • T1000.7z (78,797 bytes)
  • T3000.7z (276,987 bytes)
  • T40000.7z (3,507,025 bytes)
  • T60000.7z (4,643,853 bytes)
  • compound_of_prisms.tcl (475 bytes)
  • OCC23979.txt (1,740 bytes)
  • bug23979 (311 bytes)

Relationships

related to 0023974 closedbugmaster Optimization of convertation to STEP. 
related to 0029830 closedbugmaster Data Exchange, STEPCAFControl_Reader poor performance - quadratic dependence 

Activities

sdv

2013-05-21 16:57

developer  

T1000.7z (78,797 bytes)

sdv

2013-05-21 17:01

developer  

T3000.7z (276,987 bytes)

sdv

2013-05-21 17:03

developer  

T40000.7z (3,507,025 bytes)

sdv

2013-05-21 17:04

developer  

T60000.7z (4,643,853 bytes)

sdv

2013-05-21 17:48

developer   ~0024453

In order to minimize secondary information of profiling with Intel VTune Amplifier, I reccomend to wrap the target function by methods __itt_resume() and __itt_pause().
Also it is important to specify settings of Amplifier project.
Start new analysis -> Project properties -> Advanced
For long-tme target is necessary increase "Duration time estimate" and "Collection data limit", otherwise result of profiling may be broken.

sdv

2013-05-21 18:17

developer   ~0024454

Also I have use "Intel Advisor" to find out critical loops.
In my opinion it would be useful to analyse possibility to make parallel loop
in the STEPControl_ActorWrite::TransferCompound

for ( i = 1; i <= nbs; i ++) {
    Handle(TransferBRep_ShapeMapper) subs = TransferBRep::ShapeMapper (FP,RepItemSeq->Value(i));
    Handle(StepGeom_Axis2Placement3d) AX1;
    
    Handle(Transfer_Binder) bnd = TransferSubShape(subs, SDR0, AX1, FP, NonManifoldGroup, isManifold);

    if (!AX1.IsNull()) ItemSeq->Append (AX1);
    // copy binders so as to have all roots in upper binder, but do not conflict
    while ( !bnd.IsNull() ) {
      Handle(Transfer_SimpleBinderOfTransient) bx =
        Handle(Transfer_SimpleBinderOfTransient)::DownCast(bnd);
      if ( !bx.IsNull() )
        // Single SDR is created for a non-manifold group (ssv: 12.11.2010)
        if (!isManifold && i > 1)
          break;
        else {
          Handle(Transfer_Binder) aResult = TransientResult( bx->Result() );
          mapOfResultToAdd.Add( aResult );
        }
      bnd = bnd->NextResult();
    }
  }

sdv

2013-05-21 18:22

developer   ~0024455

Reminder sent to: vro

Dear vro,
Could you attached snippet of code for generation BRep files which was used for convertation into STEP.

Best regards,
Sergey

vro

2013-05-22 15:38

developer   ~0024468

Private property, sorry.

vro

2013-05-22 16:00

developer   ~0024469

The test-model represents a set of identical shapes: prisms extruded from triangles. The triangles are the same and differ only by the location. It looks like a chess-field filled by the prisms in. The number of prisms varies up to 200 000. Probably a draw-script might generate the test-model, what do you think?

san

2013-05-22 19:54

developer   ~0024475

compound_of_prisms.tcl that generates the compound of 200K prisms is attached.

san

2013-05-22 19:58

developer  

compound_of_prisms.tcl (475 bytes)

san

2013-05-24 14:42

developer   ~0024508

DRAW reproducer script is inefficient, it takes more than a whole day for it to complete. Thus a dedicated DRAW command should be created that builds the compound of prisms as illustrated by compound_of_prisms.tcl but fully with C++.

bugmaster

2014-01-29 17:29

administrator   ~0027696

Test case is required.

ski

2014-11-19 11:06

developer   ~0034502

QA command OCC23979 and test case bug23979 were attached to this issue.
DRAW command build the compound of 200k prism, but it leads to out of memory (4GB RAM is not enough for execution of this command).

ski

2014-11-19 11:08

developer  

OCC23979.txt (1,740 bytes)

ski

2014-11-19 11:08

developer  

bug23979 (311 bytes)

msv

2018-06-09 15:22

developer   ~0076698

Last edited: 2018-06-09 15:23

 The script compound_of_prisms.tcl still can be used efficiently if we disable autodisplay feature in draw:
autodisplay 0

Then this script takes 67 s to create the shape.

msv

2018-06-09 16:05

developer   ~0076700

With the patch for 0029830 writing to STEP of the shape got by this script takes 514 s. However the process ended with failure (malloc), and the result file has zero size. The error message is:

 Step File Name : 200K.step(55000020 ents)     ****    Interruption SendAll par Exception :   ****
Standard_MMgrRaw::Allocate(): malloc failed
    Abandon
Error: File 200K.step written with fail messages


During run, the application took about 28 Gb memory.

msv

2018-06-09 16:17

developer   ~0076702

With the model reduced down to 100K prisms it completed OK. The time is 265 s. The pick occupied memory is 15 Gb. The produced STEP file has the size 1.5 Gb.

msv

2018-06-09 18:13

developer   ~0076709

I have tried to perfom the same on current master. The process has taken 2 hours and it seems it did not reach yet even a half of the progress (it took only 2.6 Gb memory so far). Looking at its stack I have found that the hot spot is the method Transfer_Binder::AddResult(). This hot spot has been eliminated in the patch for 29830.

msv

2018-06-09 18:17

developer   ~0076710

So, the remaining problem is probably too much high memory eating.
It is needed to investigate if this task can be done with less memory requirements.

msv

2018-06-09 18:18

developer   ~0076711

I have put the improved script in steps to reproduce.

git

2019-09-04 18:31

administrator   ~0086749

Branch CR23979_test has been created by abv.

SHA-1: 3870430b9cb0d759c69066cd89a114f100d1f595


Detailed log of new commits:

Author: abv
Date: Wed Sep 4 18:28:49 2019 +0300

    Test for 0023979: Performance problem of STEP export for large files

abv

2019-09-04 18:43

manager   ~0086750

Last edited: 2019-09-04 19:01

 Branch CR23979_test contains test script for the issue (with only 10000 prisms).

On current master on my workstation it takes 30 sec overall, among which 13 sec are spent for conversion of the shape to STEP model (with peak VM size ~ 1 GB) and 7 sec for writing the file to disk (with VM peak 1.5 GB). The file size is 144 MB, thus timing seems to be reasonable, however memory consuption during writing model to file seems to be too big -- to be investigated.

In OCCT 7.3.0, conversion to STEP took 373 sec on the same workstation, with other timings and VM usage the same as on master. Thus, we have essential improvement of performance of conversion to STEP on master (7.4.0) vs 7.3.0 (13 sec vs. 373 sec).

Issue History

Date Modified Username Field Change
2013-05-21 16:57 sdv New Issue
2013-05-21 16:57 sdv Assigned To => gka
2013-05-21 16:57 sdv File Added: T1000.7z
2013-05-21 17:01 sdv File Added: T3000.7z
2013-05-21 17:03 sdv File Added: T40000.7z
2013-05-21 17:04 sdv File Added: T60000.7z
2013-05-21 17:48 sdv Note Added: 0024453
2013-05-21 17:53 sdv Relationship added related to 0023974
2013-05-21 18:17 sdv Note Added: 0024454
2013-05-21 18:22 sdv Note Added: 0024455
2013-05-22 15:38 vro Note Added: 0024468
2013-05-22 16:00 vro Note Added: 0024469
2013-05-22 19:54 san File Added: compound_of_prisms.tcl
2013-05-22 19:54 san Note Added: 0024475
2013-05-22 19:58 san File Deleted: compound_of_prisms.tcl
2013-05-22 19:58 san File Added: compound_of_prisms.tcl
2013-05-24 14:42 san Note Added: 0024508
2014-01-29 17:29 bugmaster Note Added: 0027696
2014-01-29 17:29 bugmaster Assigned To gka => apn
2014-01-29 17:29 bugmaster Status new => assigned
2014-11-19 11:06 ski Note Added: 0034502
2014-11-19 11:08 ski File Added: OCC23979.txt
2014-11-19 11:08 ski File Added: bug23979
2014-11-19 11:08 ski Assigned To apn => gka
2014-11-19 11:26 abv Target Version => 7.1.0
2016-10-26 12:18 gka Target Version 7.1.0 => 7.2.0
2017-07-27 09:33 abv Target Version 7.2.0 => 7.4.0
2018-06-09 12:48 abv Assigned To gka => msv
2018-06-09 15:22 msv Note Added: 0076698
2018-06-09 15:23 msv Note Edited: 0076698
2018-06-09 16:05 msv Note Added: 0076700
2018-06-09 16:06 msv Relationship added related to 0029830
2018-06-09 16:17 msv Note Added: 0076702
2018-06-09 18:13 msv Note Added: 0076709
2018-06-09 18:17 msv Note Added: 0076710
2018-06-09 18:18 msv Note Added: 0076711
2018-06-09 18:18 msv Steps to Reproduce Updated
2018-06-09 18:19 msv Steps to Reproduce Updated
2018-06-09 18:19 msv Assigned To msv => gka
2019-09-04 18:09 kgv Summary Performance problem of STEP export for large files => Data Exchange - Performance problem of STEP export for large files
2019-09-04 18:31 git Note Added: 0086749
2019-09-04 18:43 abv Note Added: 0086750
2019-09-04 18:43 abv Target Version 7.4.0 => 7.5.0
2019-09-04 19:01 abv Note Edited: 0086750
2020-09-17 20:19 gka Target Version 7.5.0 => 7.6.0
2020-09-18 22:24 abv Test case number => perf de bug23979
2021-08-31 14:56 gka Target Version 7.6.0 => Unscheduled
2024-02-01 02:46 dpasukhi Assigned To gka => dpasukhi