View Issue Details

IDProjectCategoryView StatusLast Update
0030413Open CASCADEOCCT:Testspublic2018-12-18 21:52
Reporterkgv Assigned Toapn  
PrioritynormalSeverityminor 
Status closedResolutionfixed 
Product Version7.4.0 
Target Version7.4.0Fixed in Version7.4.0 
Summary0030413: Tests - increase counters within perf/ncollection/A1
DescriptionTest case perf/ncollection/A1 is currently unstable - it regularly fails within Jenkins regression testing producing additional noise in test result.

perf/ncollection/A1 has been originally designed for performance comparison between STL and OCCT collections, and after statistics has been collected, has been kept as a regression test for detecting possible performance regressions of OCCT collections as an expected ratio relative to STL collection.

This introduces dependency on STL implementation, so that test case contains numbers collected for 3 major C++ compilers used for OCCT testing - but obviously, this scheme is unstable, which has been proved by porting onto 0029911 when ratio changed dramatically without any modification of OCCT source code.

It is desired refactoring test case so that to eliminate dependency on C++ compiler features (and STL implementation in particular), which can be done by collecting reference numbers within IR tests (as in case of image comparison) instead of hard-coded numbers, and allow test to detect what it is actually expected to detect - performance regressions in OCCT collections caused by modifications of these collections.

So far, to make test case more stable, it is proposed increasing numbers.
Steps To Reproduce
Testing performance (Size | STL time | OCCT time | STL/OCCT boost)

std::vector vs NCollection_Array1 (sort):

10000	0.0079130000000000034	0.0082959999999999978	1.0484013648426633
20000	0.016650999999999999	0.017243999999999982	1.0356134766680669
40000	0.034741000000000022	0.036100999999999994	1.0391468293946626
80000	0.073471000000000009	0.07606400000000002	1.0352928366294185
160000	0.15362999999999999	0.15874600000000005	1.0333007876065876
320000	0.32477800000000001	0.33216699999999988	1.0227509252473994
640000	0.75680000000000014	0.8262830000000001	1.0918115750528541
1280000	1.6498179999999998	1.6562189999999992	1.0038798218955056

std::vector vs NCollection_Vector (sort):

10000	0.0081379999999997565	0.025908999999999516	3.1837060702875757
20000	0.017007000000000438	0.054781000000000191	3.2210854354088774
40000	0.035942999999999614	0.11549299999999985	3.2132264975099765
80000	0.076618999999999993	0.24440799999999996	3.1899137289706205
160000	0.16194500000000023	0.51461300000000065	3.1777023063385714
320000	0.32139999999999969	1.0500109999999996	3.2669912881145011
640000	0.69886100000000084	2.2402660000000001	3.2055959625733834
1280000	1.5643060000000002	5.2310089999999985	3.3439806534015708

std::vector vs NCollection_Array1 (replace):

10000	0.011427000000001186	0.011517000000001332	1.0078760829614191
20000	0.023536000000000001	0.023835999999999302	1.0127464309992904
40000	0.046772000000000702	0.04651399999999839	0.99448387924394277
80000	0.088563999999998089	0.088436999999998989	0.99856600876203538
160000	0.1853359999999995	0.17325999999999908	0.9348426641343266
320000	0.35631899999999916	0.36185700000000054	1.015542252868922
640000	0.75608300000000028	0.74718299999999971	0.98822880556764192
1280000	1.542783	1.5480400000000003	1.0034074785630904

std::vector vs NCollection_Vector (replace):

10000	0.0067440000000011935	0.028395999999997201	4.2105575326204292
20000	0.013583999999998042	0.056635999999997466	4.1693168433455261
40000	0.027198000000002054	0.111146999999999	4.0865872490620854
80000	0.05328100000000191	0.22369199999999978	4.1983446256637782
160000	0.1072919999999975	0.44358100000000178	4.134334339932261
320000	0.21853399999999823	0.89095100000000116	4.076944548674386
640000	0.50946800000000181	1.7935610000000004	3.5204585960256463
1280000	1.031245000000002	3.5499370000000035	3.442379841841654

std::list vs NCollection_List (replace):

10000	0.020875999999994121	0.022686000000000206	1.0867024334166793
20000	0.043179000000002077	0.05200699999999614	1.2044512378701138
40000	0.086531999999998277	0.095292000000000598	1.1012342254888654
80000	0.17473299999999625	0.20357600000000531	1.1650689909748568
160000	0.39604299999999881	0.45729000000000042	1.1546473488989877
320000	0.88696099999999944	1.0656340000000029	1.2014440319247448
640000	1.7476020000000005	2.186479999999996	1.2511315505475475
1280000	3.6411259999999999	4.5241540000000029	1.2425150901122353

std::list vs NCollection_Sequence (replace):

10000	0.02076900000000137	0.021122000000005414	1.0169964851463249
20000	0.041992999999997949	0.054650999999999783	1.3014311909128295
40000	0.084081999999995105	0.096910999999998637	1.1525772460217916
80000	0.17087599999999981	0.22116399999999459	1.294295278447499
160000	0.4138939999999991	0.50167199999999923	1.2120784548700883
320000	0.89987399999999695	1.1398530000000022	1.2666806686269478
640000	2.3090329999999994	2.2698439999999991	0.98302796018939509
1280000	4.9128919999999994	4.8151920000000032	0.980113546155707

std::list vs NCollection_Sequence (reverse):

10000	0.013255000000000905	0.013339999999999463	1.0064126744623578
20000	0.029623999999998318	0.029008000000004586	0.97920604914954879
40000	0.057690000000008013	0.050354999999996153	0.87285491419637995
80000	0.1286840000000069	0.11838900000000763	0.91999782412732956
160000	0.28668100000000152	0.29563699999999926	1.0312402984501858
320000	0.99029000000000167	0.8628659999999968	0.87132658110250061
640000	2.0023790000000048	2.3162939999999992	1.1567710208706712
1280000	4.2302040000000005	4.1994159999999994	0.99272186400466711

std::set vs NCollection_Map (search):

100000	0.01179899999999634	0.00086600000000203181	0.073396050512950284
200000	0.014381999999997674	0.00084699999999315878	0.058893060769941295
400000	0.017305999999990718	0.00081800000000953332	0.047266843869754542
800000	0.021918999999996913	0.00084199999999157171	0.038414161229604012
1600000	0.016789000000002829	0.00080199999999308602	0.047769372803201551
3200000	0.016085000000003902	0.00097100000000693854	0.06036680136815064

std::set vs NCollection_IndexedMap (search):

100000	0.016931999999997061	0.0012949999999989359	0.076482400188941688
200000	0.020969999999991273	0.0014510000000029777	0.069194086790824103
400000	0.024235000000004447	0.0014430000000089649	0.059541984733183416
800000	0.028135999999989281	0.0014650000000102636	0.052068524310876524
1600000	0.03218600000001004	0.0014500000000055024	0.045050643136924445
3200000	0.035428999999993493	0.0015180000000043492	0.042846255892196448

OK: performance of std::vector vs NCollection_Array1 (sort): is within expected limits
OK: performance of std::vector vs NCollection_Vector (sort): is within expected limits
OK: performance of std::vector vs NCollection_Array1 (replace): is within expected limits
OK: performance of std::vector vs NCollection_Vector (replace): is within expected limits
Error: performance of std::list vs NCollection_List (replace): become worse than before

OK: performance of std::list vs NCollection_Sequence (replace): is within expected limits
OK: performance of std::list vs NCollection_Sequence (reverse): is within expected limits
OK: performance of std::set vs NCollection_Map (search): is within expected limits
OK: performance of std::set vs NCollection_IndexedMap (search): is within expected limits
TagsNo tags attached.
Test case numberperf ncollection A1

Relationships

related to 0029911 closedbugmaster Porting to Debian80-64 : Regressions in Foundation Classes  
related to 0023880 closedbugmaster Integration of grid "ncl" into the new testing system 

Activities

git

2018-12-13 15:23

administrator   ~0081514

Branch CR30413 has been created by kgv.

SHA-1: cc83654c3de44b5e139b4ed951f49e9c585b0155


Detailed log of new commits:

Author: kgv
Date: Thu Dec 13 15:20:48 2018 +0300

    0030413: Tests - increase counters within perf/ncollection/A1

kgv

2018-12-13 18:00

developer   ~0081519

Patch is ready for review.

http://jenkins-test-12.nnov.opencascade.com:8080/view/CR30413-master-KGV/

abv

2018-12-13 20:46

manager   ~0081521

No remarks, please integrate

apn

2018-12-14 11:37

administrator   ~0081525

Combination -
OCCT branch : CR30413 SHA - cc83654c3de44b5e139b4ed951f49e9c585b0155
Products branch : master SHA - 4eefea36b8ea8a987b6f644ce91f592588345220
was compiled on Linux, MacOS and Windows platforms and tested in optimize mode.

Number of compiler warnings:
No new/fixed warnings

Regressions/Differences/Improvements:
No regressions/differences

git

2018-12-18 21:52

administrator   ~0081571

Branch CR30413 has been deleted by kgv.

SHA-1: cc83654c3de44b5e139b4ed951f49e9c585b0155

Related Changesets

occt: master eeabb8f8

2018-12-13 12:20:48

kgv


Committer: apn Details Diff
0030413: Tests - increase counters within perf/ncollection/A1 Affected Issues
0030413
mod - tests/perf/ncollection/A1 Diff File

Issue History

Date Modified Username Field Change
2018-12-13 15:17 kgv New Issue
2018-12-13 15:17 kgv Assigned To => apn
2018-12-13 15:17 kgv Relationship added related to 0029911
2018-12-13 15:23 git Note Added: 0081514
2018-12-13 15:24 kgv Relationship added related to 0023880
2018-12-13 18:00 kgv Note Added: 0081519
2018-12-13 18:00 kgv Assigned To apn => abv
2018-12-13 18:00 kgv Status new => resolved
2018-12-13 20:46 abv Note Added: 0081521
2018-12-13 20:46 abv Assigned To abv => bugmaster
2018-12-13 20:46 abv Status resolved => reviewed
2018-12-14 11:37 apn Test case number => perf ncollection A1
2018-12-14 11:37 apn Note Added: 0081525
2018-12-14 11:37 apn Status reviewed => tested
2018-12-15 20:28 apn Changeset attached => occt master eeabb8f8
2018-12-15 20:28 apn Assigned To bugmaster => apn
2018-12-15 20:28 apn Status tested => verified
2018-12-15 20:28 apn Resolution open => fixed
2018-12-18 21:52 git Note Added: 0081571