MantisBT - Community
View Issue Details
0026515Community[OCCT] OCCT:Modeling Algorithmspublic2015-08-04 17:332015-10-23 20:50
srlockley 
bugmaster 
normalmajor 
closedfixed 
WindowsVC++ 201364 bit
[OCCT] 6.9.0 
[OCCT] 6.9.1[OCCT] 6.9.1 
bugs modalg_6 bug26567
0026515: Exponential memory usage problems with BOPDS_IndexedMapOfPaveBlock and NCollection_IncAllocator
In BopAlgo_PaveFiller, MakeBlocks(), NCollection_IncAllocator is used to allocate memory for the collections. When largish sets of tools (>150 SHAPES) are used in a boolean operation the memory consumption of this class rises exponentially, over 2GB of memory is consumed when the collections have the following numbers
aLSE=3607
aLBV=0
aMVOnIn=12
aMF=3739
aMVStick=0
aMVEF=0
aMI=18
aMPBOnIn=15
aMPBAdd=9799
aLPB=1
aMSCPB=2974
aMVI=4832
aDMExEdges=222
aMVTol=0
aDMI=222
aDMBV=0

These are not really large numbers and definitely should not consume 2GB of RAM
Removing the use of the allocator from the collection constructors completely resolves the problem, in fact there is no significant memory use at all in Task manager or processes viewer. Indicating the issue is in the allocation class.
I have tried to find the cause, it appears to be BOPDS_IndexedMapOfPaveBlock and BOPCol_MapOfInteger that causes excessive memory allocation.
Strangely, removing the use of the allocator not only improves memory performance but also speed in my environment (.NET, mix mode)
Cut more than 150 shapes from any shape. In fact the main class that clauses the problem BOPDS_IndexedMapOfPaveBlock only has 15 members in my example so the problem hopefully is simple, but I am afraid i couldn't grasp it.
No tags attached.
Issue History
2015-08-04 17:33srlockleyNew Issue
2015-08-04 17:33srlockleyAssigned To => msv
2015-08-04 18:11msvNote Added: 0043821
2015-08-04 18:11msvAssigned Tomsv => pkv
2015-08-04 18:11msvStatusnew => assigned
2015-08-04 18:12msvSummaryExponential memory usage problems with BOPDS_IndexedMapOfPaveBlockand NCollection_IncAllocator => Exponential memory usage problems with BOPDS_IndexedMapOfPaveBlock and NCollection_IncAllocator
2015-08-19 13:36abvTarget Version => 7.1.0
2015-08-25 12:38msvNote Added: 0044597
2015-08-25 12:38msvAssigned Topkv => srlockley
2015-08-25 12:38msvStatusassigned => feedback
2015-08-26 15:01msvRelationship addedhas duplicate 0026567
2015-08-27 13:43srlockleyNote Added: 0044739
2015-08-27 14:48msvNote Added: 0044745
2015-08-27 14:48msvAssigned Tosrlockley => isn
2015-08-27 14:50msvNote Added: 0044746
2015-08-27 14:50msvStatusfeedback => assigned
2015-09-10 15:25gitNote Added: 0045425
2015-09-10 15:27isnNote Added: 0045426
2015-09-10 15:27isnAssigned Toisn => msv
2015-09-10 15:27isnStatusassigned => resolved
2015-09-10 15:28isnNote Edited: 0045426bug_revision_view_page.php?bugnote_id=45426#r11543
2015-09-10 17:51msvNote Added: 0045447
2015-09-10 17:51msvAssigned Tomsv => isn
2015-09-10 17:51msvStatusresolved => assigned
2015-09-10 18:42gitNote Added: 0045453
2015-09-10 18:43isnAssigned Toisn => msv
2015-09-10 18:43isnStatusassigned => resolved
2015-09-11 11:15msvNote Added: 0045468
2015-09-11 11:15msvAssigned Tomsv => isn
2015-09-11 11:15msvStatusresolved => assigned
2015-09-11 11:34gitNote Added: 0045470
2015-09-11 11:35isnAssigned Toisn => msv
2015-09-11 11:35isnStatusassigned => resolved
2015-09-11 16:17msvNote Added: 0045505
2015-09-11 16:17msvAssigned Tomsv => bugmaster
2015-09-11 16:17msvStatusresolved => reviewed
2015-09-11 16:42apvAssigned Tobugmaster => apv
2015-09-11 17:43gitNote Added: 0045522
2015-09-11 17:43apvNote Added: 0045524
2015-09-14 14:25apvNote Added: 0045570
2015-09-14 14:25apvAssigned Toapv => isn
2015-09-14 14:25apvStatusreviewed => assigned
2015-09-14 14:57msvNote Added: 0045580
2015-09-14 16:21msvAssigned Toisn => apv
2015-09-14 16:21msvStatusassigned => resolved
2015-09-14 16:21msvAssigned Toapv => bugmaster
2015-09-14 16:21msvStatusresolved => reviewed
2015-09-14 16:22msvAssigned Tobugmaster => apv
2015-09-14 16:22msvStatusreviewed => assigned
2015-09-15 14:00gitNote Added: 0045634
2015-09-15 14:02abvNote Added: 0045635
2015-09-15 14:02abvStatusassigned => feedback
2015-09-15 14:02abvTarget Version7.1.0 => 6.9.1
2015-09-15 17:10apvTest case number => bugs modalg_6 bug26567
2015-09-15 17:42gitNote Added: 0045659
2015-09-15 17:44apvNote Added: 0045660
2015-09-15 17:44apvAssigned Toapv => bugmaster
2015-09-15 17:44apvStatusfeedback => tested
2015-09-18 13:31bugmasterChangeset attached => occt master 488e5b9d
2015-09-18 13:31bugmasterStatustested => verified
2015-09-18 13:31bugmasterResolutionopen => fixed
2015-10-16 14:55aivStatusverified => closed
2015-10-16 16:37gitNote Added: 0046981
2015-10-23 20:50aivFixed in Version => 6.9.1

Notes
(0043821)
msv   
2015-08-04 18:11   
The aim of using IncAllocator is to reduce the number of times of calling malloc/free. It never returns memory back to system till its destruction or till someone calls its Reset method. Therefore it should be used with care. If a container is constructed with IncAllocator then it is supposed one of two conditions to be true:
1. Life time of this container is small.
2. If life time is large then the container will accommodate new objects without removing existing objects, because allocator's Free method does not return memory to system.
Probably, for container-fields of BOP classes that live long it is needed to refuse passing IncAllocator to constructor.
(0044597)
msv   
2015-08-25 12:38   
Dear srlockley,
Could you, please attach input data and provide the draw script in "steps to reproduce" field? We need it to reproduce your problem and insert it in non-regression test base.
Thanks in advance,
Mikhail
(0044739)
srlockley   
2015-08-27 13:43   
I will try it was a complex operation that didn't really set well in a brep file.

I will see if I can convert it to a draw script but it may take me a while.

PS I am pretty sure the function is only caused by the two functions I pinpointed BOPDS_IndexedMapOfPaveBlock and BOPCol_MapOfInteger is there something different about these from other maps?
(0044745)
msv   
2015-08-27 14:48   
I did not understand the question, but the problem is here not with maps, but with overall algorithm behavior and mutual scope of containers and IncAllocator used for them. We have already another bug, which fix should help in your case. However, it would be fine to add to non-regression test base your case too, if possible.
(0044746)
msv   
2015-08-27 14:50   
Dear Ilya, please take the patch from the branch to related issue #26567, and merge it in master.
(0045425)
git   
2015-09-10 15:25   
Branch CR26515 has been created by isn.

SHA-1: abd17b99dddbe30bdd76c4f27e82b984c5d6f5a0


Detailed log of new commits:

Author: pkv
Date: Tue Aug 18 10:06:11 2015 +0300

    0026567: Exception in Boolean intersection command
    
    Correcting the usage of NCollection_IncAllocator
    
    Conflicts:
        src/BOPAlgo/BOPAlgo_BOP.cxx
        src/BOPAlgo/BOPAlgo_Builder.cxx
        src/BOPAlgo/BOPAlgo_BuilderSolid.cxx
        src/BOPAlgo/BOPAlgo_Builder_2.cxx
        src/BOPAlgo/BOPAlgo_MakerVolume.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller.cdl
        src/BOPAlgo/BOPAlgo_PaveFiller_1.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller_10.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller_3.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller_4.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller_5.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller_6.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller_7.cxx
        src/BOPAlgo/BOPAlgo_PaveFiller_8.cxx
        src/BOPAlgo/BOPAlgo_Section.cxx
        src/BOPDS/BOPDS_DS.cxx
        src/BOPDS/BOPDS_Iterator.cxx
        src/BOPDS/BOPDS_IteratorSI.cxx
        src/BOPDS/BOPDS_SubIterator.cxx
        src/BOPTest/BOPTest_BOPCommands.cxx
        src/BOPTest/BOPTest_PartitionCommands.cxx
        src/IntTools/IntTools_Context.cxx
(0045426)
isn   
2015-09-10 15:27   
(edited on: 2015-09-10 15:28)
Branch from issue ##26567 have been merged with the current master

(0045447)
msv   
2015-09-10 17:51   
Some changes were not transfered. Please check all changes manually.
(0045453)
git   
2015-09-10 18:42   
Branch CR26515 has been updated forcibly by isn.

SHA-1: a370d63830e66de0d944589caf1e0ed943b79426
(0045468)
msv   
2015-09-11 11:15   
The changes in the files

src\BOPAlgo\BOPAlgo_PaveFiller_7.cxx
src\BOPAlgo\BOPAlgo_Section.cxx
src\BOPTest\BOPTest_PartitionCommands.cxx

were not transfered. Please make the changes in them also.
(0045470)
git   
2015-09-11 11:34   
Branch CR26515 has been updated by isn.

SHA-1: 7242be6c1e85bb7d55d83ccb388c7ce5a51b2d75


Detailed log of new commits:

Author: isn
Date: Fri Sep 11 11:29:57 2015 +0300

    additional corrections

(0045505)
msv   
2015-09-11 16:17   
Reviewed.
(0045522)
git   
2015-09-11 17:43   
Branch CR26515 has been updated forcibly by apv.

SHA-1: dc9b9b99dee2e43209cab3523efe05dbf41b88f9
(0045524)
apv   
2015-09-11 17:43   
Branch CR26515 has been rebased on the current master
(0045570)
apv   
2015-09-14 14:25   
Dear BugMaster,

Branch CR26515 from occt git-repository (and master from products git-repository) was compiled on Linux and Windows platforms and tested.
SHA-1: dc9b9b99dee2e43209cab3523efe05dbf41b88f9

Number of compiler warnings:
occt component:
   Linux: 15 (15 on master)
   Windows: 0 (0 on master)
products component:
   Linux: 39 (39 on master)
   Windows: 0 (0 on master)

Regressions/Differences:
http://occt-tests/CR26515-master-occt-64/Debian70-64/summary.html [^]
http://occt-tests/CR26515-master-occt-64/Windows-64-VC10/summary.html [^]
bugs fclasses bug7287_1

Testing on Linux:
Total MEMORY difference: 90349201 / 92721808 [-2.56%]
Total CPU difference: 17980.68999999927 / 17686.84999999906 [+1.66%]

Testing on Windows:
Total MEMORY difference: 56152860 / 57753560 [-2.77%]
Total CPU difference: 16694.18181329908 / 16847.78039789915 [-0.91%]
(0045580)
msv   
2015-09-14 14:57   
Dear Alexey, please remove TODO of the improved test case bugs fclasses bug7287_1.
(0045634)
git   
2015-09-15 14:00   
Branch CR26515 has been updated by abv.

SHA-1: b84c457f8aa899ebae3f3c99f0a03be86f64bf83


Detailed log of new commits:

Author: abv
Date: Tue Sep 15 13:54:13 2015 +0300

    Adaptation of test case bugs fclasses bug7287_1 (limit for detection of memory leak reduced due to leak reduction)

(0045635)
abv   
2015-09-15 14:02   
I have pushed correction of test case bugs fclasses bug7287_1 to branch CR26515, please check it (only this test!).

Besides, while integrating this fix, note that title of its commit messages is that of #26567; this is correct.
(0045659)
git   
2015-09-15 17:42   
Branch CR26515 has been updated by apv.

SHA-1: 13471ac8a1aea34766494eb214db25b26f2f6dd8


Detailed log of new commits:

Author: apv
Date: Tue Sep 15 17:42:07 2015 +0300

    Adjusting test-case bugs fclasses bug7287_1

(0045660)
apv   
2015-09-15 17:44   
Result of testing case bugs fclasses bug7287_1:
http://occt-tests/CR26515-master-occt-64/Debian70-64/bugs/fclasses/bug7287_1.html [^]
http://occt-tests/CR26515-master-occt-64/Windows-64-VC10/bugs/fclasses/bug7287_1.html [^]
(0046981)
git   
2015-10-16 16:37   
Branch CR26515 has been deleted by kgv.

SHA-1: 13471ac8a1aea34766494eb214db25b26f2f6dd8