View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0030413 | Open CASCADE | OCCT:Tests | public | 2018-12-13 15:17 | 2018-12-18 21:52 |
| Reporter | kgv | Assigned To | apn | ||
| Priority | normal | Severity | minor | ||
| Status | closed | Resolution | fixed | ||
| Product Version | 7.4.0 | ||||
| Target Version | 7.4.0 | Fixed in Version | 7.4.0 | ||
| Summary | 0030413: Tests - increase counters within perf/ncollection/A1 | ||||
| Description | Test case perf/ncollection/A1 is currently unstable - it regularly fails within Jenkins regression testing producing additional noise in test result. perf/ncollection/A1 has been originally designed for performance comparison between STL and OCCT collections, and after statistics has been collected, has been kept as a regression test for detecting possible performance regressions of OCCT collections as an expected ratio relative to STL collection. This introduces dependency on STL implementation, so that test case contains numbers collected for 3 major C++ compilers used for OCCT testing - but obviously, this scheme is unstable, which has been proved by porting onto 0029911 when ratio changed dramatically without any modification of OCCT source code. It is desired refactoring test case so that to eliminate dependency on C++ compiler features (and STL implementation in particular), which can be done by collecting reference numbers within IR tests (as in case of image comparison) instead of hard-coded numbers, and allow test to detect what it is actually expected to detect - performance regressions in OCCT collections caused by modifications of these collections. So far, to make test case more stable, it is proposed increasing numbers. | ||||
| Steps To Reproduce | Testing performance (Size | STL time | OCCT time | STL/OCCT boost) std::vector vs NCollection_Array1 (sort): 10000 0.0079130000000000034 0.0082959999999999978 1.0484013648426633 20000 0.016650999999999999 0.017243999999999982 1.0356134766680669 40000 0.034741000000000022 0.036100999999999994 1.0391468293946626 80000 0.073471000000000009 0.07606400000000002 1.0352928366294185 160000 0.15362999999999999 0.15874600000000005 1.0333007876065876 320000 0.32477800000000001 0.33216699999999988 1.0227509252473994 640000 0.75680000000000014 0.8262830000000001 1.0918115750528541 1280000 1.6498179999999998 1.6562189999999992 1.0038798218955056 std::vector vs NCollection_Vector (sort): 10000 0.0081379999999997565 0.025908999999999516 3.1837060702875757 20000 0.017007000000000438 0.054781000000000191 3.2210854354088774 40000 0.035942999999999614 0.11549299999999985 3.2132264975099765 80000 0.076618999999999993 0.24440799999999996 3.1899137289706205 160000 0.16194500000000023 0.51461300000000065 3.1777023063385714 320000 0.32139999999999969 1.0500109999999996 3.2669912881145011 640000 0.69886100000000084 2.2402660000000001 3.2055959625733834 1280000 1.5643060000000002 5.2310089999999985 3.3439806534015708 std::vector vs NCollection_Array1 (replace): 10000 0.011427000000001186 0.011517000000001332 1.0078760829614191 20000 0.023536000000000001 0.023835999999999302 1.0127464309992904 40000 0.046772000000000702 0.04651399999999839 0.99448387924394277 80000 0.088563999999998089 0.088436999999998989 0.99856600876203538 160000 0.1853359999999995 0.17325999999999908 0.9348426641343266 320000 0.35631899999999916 0.36185700000000054 1.015542252868922 640000 0.75608300000000028 0.74718299999999971 0.98822880556764192 1280000 1.542783 1.5480400000000003 1.0034074785630904 std::vector vs NCollection_Vector (replace): 10000 0.0067440000000011935 0.028395999999997201 4.2105575326204292 20000 0.013583999999998042 0.056635999999997466 4.1693168433455261 40000 0.027198000000002054 0.111146999999999 4.0865872490620854 80000 0.05328100000000191 0.22369199999999978 4.1983446256637782 160000 0.1072919999999975 0.44358100000000178 4.134334339932261 320000 0.21853399999999823 0.89095100000000116 4.076944548674386 640000 0.50946800000000181 1.7935610000000004 3.5204585960256463 1280000 1.031245000000002 3.5499370000000035 3.442379841841654 std::list vs NCollection_List (replace): 10000 0.020875999999994121 0.022686000000000206 1.0867024334166793 20000 0.043179000000002077 0.05200699999999614 1.2044512378701138 40000 0.086531999999998277 0.095292000000000598 1.1012342254888654 80000 0.17473299999999625 0.20357600000000531 1.1650689909748568 160000 0.39604299999999881 0.45729000000000042 1.1546473488989877 320000 0.88696099999999944 1.0656340000000029 1.2014440319247448 640000 1.7476020000000005 2.186479999999996 1.2511315505475475 1280000 3.6411259999999999 4.5241540000000029 1.2425150901122353 std::list vs NCollection_Sequence (replace): 10000 0.02076900000000137 0.021122000000005414 1.0169964851463249 20000 0.041992999999997949 0.054650999999999783 1.3014311909128295 40000 0.084081999999995105 0.096910999999998637 1.1525772460217916 80000 0.17087599999999981 0.22116399999999459 1.294295278447499 160000 0.4138939999999991 0.50167199999999923 1.2120784548700883 320000 0.89987399999999695 1.1398530000000022 1.2666806686269478 640000 2.3090329999999994 2.2698439999999991 0.98302796018939509 1280000 4.9128919999999994 4.8151920000000032 0.980113546155707 std::list vs NCollection_Sequence (reverse): 10000 0.013255000000000905 0.013339999999999463 1.0064126744623578 20000 0.029623999999998318 0.029008000000004586 0.97920604914954879 40000 0.057690000000008013 0.050354999999996153 0.87285491419637995 80000 0.1286840000000069 0.11838900000000763 0.91999782412732956 160000 0.28668100000000152 0.29563699999999926 1.0312402984501858 320000 0.99029000000000167 0.8628659999999968 0.87132658110250061 640000 2.0023790000000048 2.3162939999999992 1.1567710208706712 1280000 4.2302040000000005 4.1994159999999994 0.99272186400466711 std::set vs NCollection_Map (search): 100000 0.01179899999999634 0.00086600000000203181 0.073396050512950284 200000 0.014381999999997674 0.00084699999999315878 0.058893060769941295 400000 0.017305999999990718 0.00081800000000953332 0.047266843869754542 800000 0.021918999999996913 0.00084199999999157171 0.038414161229604012 1600000 0.016789000000002829 0.00080199999999308602 0.047769372803201551 3200000 0.016085000000003902 0.00097100000000693854 0.06036680136815064 std::set vs NCollection_IndexedMap (search): 100000 0.016931999999997061 0.0012949999999989359 0.076482400188941688 200000 0.020969999999991273 0.0014510000000029777 0.069194086790824103 400000 0.024235000000004447 0.0014430000000089649 0.059541984733183416 800000 0.028135999999989281 0.0014650000000102636 0.052068524310876524 1600000 0.03218600000001004 0.0014500000000055024 0.045050643136924445 3200000 0.035428999999993493 0.0015180000000043492 0.042846255892196448 OK: performance of std::vector vs NCollection_Array1 (sort): is within expected limits OK: performance of std::vector vs NCollection_Vector (sort): is within expected limits OK: performance of std::vector vs NCollection_Array1 (replace): is within expected limits OK: performance of std::vector vs NCollection_Vector (replace): is within expected limits Error: performance of std::list vs NCollection_List (replace): become worse than before OK: performance of std::list vs NCollection_Sequence (replace): is within expected limits OK: performance of std::list vs NCollection_Sequence (reverse): is within expected limits OK: performance of std::set vs NCollection_Map (search): is within expected limits OK: performance of std::set vs NCollection_IndexedMap (search): is within expected limits | ||||
| Tags | No tags attached. | ||||
| Test case number | perf ncollection A1 | ||||
|
|
Branch CR30413 has been created by kgv. SHA-1: cc83654c3de44b5e139b4ed951f49e9c585b0155 Detailed log of new commits: Author: kgv Date: Thu Dec 13 15:20:48 2018 +0300 0030413: Tests - increase counters within perf/ncollection/A1 |
|
|
Patch is ready for review. http://jenkins-test-12.nnov.opencascade.com:8080/view/CR30413-master-KGV/ |
|
|
No remarks, please integrate |
|
|
Combination - OCCT branch : CR30413 SHA - cc83654c3de44b5e139b4ed951f49e9c585b0155 Products branch : master SHA - 4eefea36b8ea8a987b6f644ce91f592588345220 was compiled on Linux, MacOS and Windows platforms and tested in optimize mode. Number of compiler warnings: No new/fixed warnings Regressions/Differences/Improvements: No regressions/differences |
|
|
Branch CR30413 has been deleted by kgv. SHA-1: cc83654c3de44b5e139b4ed951f49e9c585b0155 |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2018-12-13 15:17 | kgv | New Issue | |
| 2018-12-13 15:17 | kgv | Assigned To | => apn |
| 2018-12-13 15:17 | kgv | Relationship added | related to 0029911 |
| 2018-12-13 15:23 | git | Note Added: 0081514 | |
| 2018-12-13 15:24 | kgv | Relationship added | related to 0023880 |
| 2018-12-13 18:00 | kgv | Note Added: 0081519 | |
| 2018-12-13 18:00 | kgv | Assigned To | apn => abv |
| 2018-12-13 18:00 | kgv | Status | new => resolved |
| 2018-12-13 20:46 |
|
Note Added: 0081521 | |
| 2018-12-13 20:46 |
|
Assigned To | abv => bugmaster |
| 2018-12-13 20:46 |
|
Status | resolved => reviewed |
| 2018-12-14 11:37 | apn | Test case number | => perf ncollection A1 |
| 2018-12-14 11:37 | apn | Note Added: 0081525 | |
| 2018-12-14 11:37 | apn | Status | reviewed => tested |
| 2018-12-15 20:28 | apn | Changeset attached | => occt master eeabb8f8 |
| 2018-12-15 20:28 | apn | Assigned To | bugmaster => apn |
| 2018-12-15 20:28 | apn | Status | tested => verified |
| 2018-12-15 20:28 | apn | Resolution | open => fixed |
| 2018-12-18 21:52 | git | Note Added: 0081571 |
