MantisBT - Open CASCADE
View Issue Details
0030413Open CASCADE[OCCT] OCCT:Testspublic2018-12-13 15:172018-12-18 21:52
kgv 
apn 
normalminor 
closedfixed 
[OCCT] 7.4.0 
[OCCT] 7.4.0[OCCT] 7.4.0 
perf ncollection A1
0030413: Tests - increase counters within perf/ncollection/A1
Test case perf/ncollection/A1 is currently unstable - it regularly fails within Jenkins regression testing producing additional noise in test result.

perf/ncollection/A1 has been originally designed for performance comparison between STL and OCCT collections, and after statistics has been collected, has been kept as a regression test for detecting possible performance regressions of OCCT collections as an expected ratio relative to STL collection.

This introduces dependency on STL implementation, so that test case contains numbers collected for 3 major C++ compilers used for OCCT testing - but obviously, this scheme is unstable, which has been proved by porting onto 0029911 when ratio changed dramatically without any modification of OCCT source code.

It is desired refactoring test case so that to eliminate dependency on C++ compiler features (and STL implementation in particular), which can be done by collecting reference numbers within IR tests (as in case of image comparison) instead of hard-coded numbers, and allow test to detect what it is actually expected to detect - performance regressions in OCCT collections caused by modifications of these collections.

So far, to make test case more stable, it is proposed increasing numbers.
Testing performance (Size | STL time | OCCT time | STL/OCCT boost)

std::vector vs NCollection_Array1 (sort):

10000	0.0079130000000000034	0.0082959999999999978	1.0484013648426633
20000	0.016650999999999999	0.017243999999999982	1.0356134766680669
40000	0.034741000000000022	0.036100999999999994	1.0391468293946626
80000	0.073471000000000009	0.07606400000000002	1.0352928366294185
160000	0.15362999999999999	0.15874600000000005	1.0333007876065876
320000	0.32477800000000001	0.33216699999999988	1.0227509252473994
640000	0.75680000000000014	0.8262830000000001	1.0918115750528541
1280000	1.6498179999999998	1.6562189999999992	1.0038798218955056

std::vector vs NCollection_Vector (sort):

10000	0.0081379999999997565	0.025908999999999516	3.1837060702875757
20000	0.017007000000000438	0.054781000000000191	3.2210854354088774
40000	0.035942999999999614	0.11549299999999985	3.2132264975099765
80000	0.076618999999999993	0.24440799999999996	3.1899137289706205
160000	0.16194500000000023	0.51461300000000065	3.1777023063385714
320000	0.32139999999999969	1.0500109999999996	3.2669912881145011
640000	0.69886100000000084	2.2402660000000001	3.2055959625733834
1280000	1.5643060000000002	5.2310089999999985	3.3439806534015708

std::vector vs NCollection_Array1 (replace):

10000	0.011427000000001186	0.011517000000001332	1.0078760829614191
20000	0.023536000000000001	0.023835999999999302	1.0127464309992904
40000	0.046772000000000702	0.04651399999999839	0.99448387924394277
80000	0.088563999999998089	0.088436999999998989	0.99856600876203538
160000	0.1853359999999995	0.17325999999999908	0.9348426641343266
320000	0.35631899999999916	0.36185700000000054	1.015542252868922
640000	0.75608300000000028	0.74718299999999971	0.98822880556764192
1280000	1.542783	1.5480400000000003	1.0034074785630904

std::vector vs NCollection_Vector (replace):

10000	0.0067440000000011935	0.028395999999997201	4.2105575326204292
20000	0.013583999999998042	0.056635999999997466	4.1693168433455261
40000	0.027198000000002054	0.111146999999999	4.0865872490620854
80000	0.05328100000000191	0.22369199999999978	4.1983446256637782
160000	0.1072919999999975	0.44358100000000178	4.134334339932261
320000	0.21853399999999823	0.89095100000000116	4.076944548674386
640000	0.50946800000000181	1.7935610000000004	3.5204585960256463
1280000	1.031245000000002	3.5499370000000035	3.442379841841654

std::list vs NCollection_List (replace):

10000	0.020875999999994121	0.022686000000000206	1.0867024334166793
20000	0.043179000000002077	0.05200699999999614	1.2044512378701138
40000	0.086531999999998277	0.095292000000000598	1.1012342254888654
80000	0.17473299999999625	0.20357600000000531	1.1650689909748568
160000	0.39604299999999881	0.45729000000000042	1.1546473488989877
320000	0.88696099999999944	1.0656340000000029	1.2014440319247448
640000	1.7476020000000005	2.186479999999996	1.2511315505475475
1280000	3.6411259999999999	4.5241540000000029	1.2425150901122353

std::list vs NCollection_Sequence (replace):

10000	0.02076900000000137	0.021122000000005414	1.0169964851463249
20000	0.041992999999997949	0.054650999999999783	1.3014311909128295
40000	0.084081999999995105	0.096910999999998637	1.1525772460217916
80000	0.17087599999999981	0.22116399999999459	1.294295278447499
160000	0.4138939999999991	0.50167199999999923	1.2120784548700883
320000	0.89987399999999695	1.1398530000000022	1.2666806686269478
640000	2.3090329999999994	2.2698439999999991	0.98302796018939509
1280000	4.9128919999999994	4.8151920000000032	0.980113546155707

std::list vs NCollection_Sequence (reverse):

10000	0.013255000000000905	0.013339999999999463	1.0064126744623578
20000	0.029623999999998318	0.029008000000004586	0.97920604914954879
40000	0.057690000000008013	0.050354999999996153	0.87285491419637995
80000	0.1286840000000069	0.11838900000000763	0.91999782412732956
160000	0.28668100000000152	0.29563699999999926	1.0312402984501858
320000	0.99029000000000167	0.8628659999999968	0.87132658110250061
640000	2.0023790000000048	2.3162939999999992	1.1567710208706712
1280000	4.2302040000000005	4.1994159999999994	0.99272186400466711

std::set vs NCollection_Map (search):

100000	0.01179899999999634	0.00086600000000203181	0.073396050512950284
200000	0.014381999999997674	0.00084699999999315878	0.058893060769941295
400000	0.017305999999990718	0.00081800000000953332	0.047266843869754542
800000	0.021918999999996913	0.00084199999999157171	0.038414161229604012
1600000	0.016789000000002829	0.00080199999999308602	0.047769372803201551
3200000	0.016085000000003902	0.00097100000000693854	0.06036680136815064

std::set vs NCollection_IndexedMap (search):

100000	0.016931999999997061	0.0012949999999989359	0.076482400188941688
200000	0.020969999999991273	0.0014510000000029777	0.069194086790824103
400000	0.024235000000004447	0.0014430000000089649	0.059541984733183416
800000	0.028135999999989281	0.0014650000000102636	0.052068524310876524
1600000	0.03218600000001004	0.0014500000000055024	0.045050643136924445
3200000	0.035428999999993493	0.0015180000000043492	0.042846255892196448

OK: performance of std::vector vs NCollection_Array1 (sort): is within expected limits
OK: performance of std::vector vs NCollection_Vector (sort): is within expected limits
OK: performance of std::vector vs NCollection_Array1 (replace): is within expected limits
OK: performance of std::vector vs NCollection_Vector (replace): is within expected limits
Error: performance of std::list vs NCollection_List (replace): become worse than before

OK: performance of std::list vs NCollection_Sequence (replace): is within expected limits
OK: performance of std::list vs NCollection_Sequence (reverse): is within expected limits
OK: performance of std::set vs NCollection_Map (search): is within expected limits
OK: performance of std::set vs NCollection_IndexedMap (search): is within expected limits
No tags attached.
related to 0029911closed bugmaster Porting to Debian80-64 : Regressions in Foundation Classes  
related to 0023880closed bugmaster Integration of grid "ncl" into the new testing system 
Issue History
2018-12-13 15:17kgvNew Issue
2018-12-13 15:17kgvAssigned To => apn
2018-12-13 15:17kgvRelationship addedrelated to 0029911
2018-12-13 15:23gitNote Added: 0081514
2018-12-13 15:24kgvRelationship addedrelated to 0023880
2018-12-13 18:00kgvNote Added: 0081519
2018-12-13 18:00kgvAssigned Toapn => abv
2018-12-13 18:00kgvStatusnew => resolved
2018-12-13 20:46abvNote Added: 0081521
2018-12-13 20:46abvAssigned Toabv => bugmaster
2018-12-13 20:46abvStatusresolved => reviewed
2018-12-14 11:37apnTest case number => perf ncollection A1
2018-12-14 11:37apnNote Added: 0081525
2018-12-14 11:37apnStatusreviewed => tested
2018-12-15 20:28apnChangeset attached => occt master eeabb8f8
2018-12-15 20:28apnAssigned Tobugmaster => apn
2018-12-15 20:28apnStatustested => verified
2018-12-15 20:28apnResolutionopen => fixed
2018-12-18 21:52gitNote Added: 0081571

Notes
(0081514)
git   
2018-12-13 15:23   
Branch CR30413 has been created by kgv.

SHA-1: cc83654c3de44b5e139b4ed951f49e9c585b0155


Detailed log of new commits:

Author: kgv
Date: Thu Dec 13 15:20:48 2018 +0300

    0030413: Tests - increase counters within perf/ncollection/A1
(0081519)
kgv   
2018-12-13 18:00   
Patch is ready for review.

http://jenkins-test-12.nnov.opencascade.com:8080/view/CR30413-master-KGV/ [^]
(0081521)
abv   
2018-12-13 20:46   
No remarks, please integrate
(0081525)
apn   
2018-12-14 11:37   
Combination -
OCCT branch : CR30413 SHA - cc83654c3de44b5e139b4ed951f49e9c585b0155
Products branch : master SHA - 4eefea36b8ea8a987b6f644ce91f592588345220
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.

Number of compiler warnings:
No new/fixed warnings

Regressions/Differences/Improvements:
No regressions/differences
(0081571)
git   
2018-12-18 21:52   
Branch CR30413 has been deleted by kgv.

SHA-1: cc83654c3de44b5e139b4ed951f49e9c585b0155