Skip to content

Conversation

@zclllyybb
Copy link
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

After all actual task was finished by this operator, the close time shouldn't be added into ExecTime. it may make weird profiles like :

   - Total: 36sec636ms
......
               OLAP_SCAN_OPERATOR (id=4. nereids_id=1858. ...
                  - ExecTime: avg 16sec665ms, max 59sec941ms, min 4sec926ms

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31968 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6396f254edf901167007d1dc5da0673f8daf8339, data reload: false

------ Round 1 ----------------------------------
q1	17640	5187	5055	5055
q2	2118	343	193	193
q3	10132	1336	734	734
q4	10198	792	324	324
q5	7508	2157	1863	1863
q6	201	186	157	157
q7	898	755	591	591
q8	9274	1417	1146	1146
q9	5270	4852	4815	4815
q10	6886	1975	1573	1573
q11	532	286	282	282
q12	346	381	229	229
q13	17821	4069	3245	3245
q14	242	252	222	222
q15	907	814	804	804
q16	671	677	627	627
q17	652	773	519	519
q18	6669	6518	6519	6518
q19	1276	989	657	657
q20	397	342	237	237
q21	2754	2151	1903	1903
q22	362	324	274	274
Total cold run time: 102754 ms
Total hot run time: 31968 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5300	5249	5277	5249
q2	264	342	255	255
q3	2177	2693	2267	2267
q4	1398	1734	1301	1301
q5	4247	4156	4238	4156
q6	224	179	141	141
q7	2279	2112	1816	1816
q8	2646	2386	2391	2386
q9	7561	7619	7513	7513
q10	2866	3062	2576	2576
q11	524	468	454	454
q12	679	743	593	593
q13	3926	4397	3679	3679
q14	311	316	311	311
q15	886	836	813	813
q16	703	724	703	703
q17	1146	1391	1499	1391
q18	8238	7774	7702	7702
q19	919	884	941	884
q20	2315	2128	1989	1989
q21	4761	4187	4130	4130
q22	554	552	508	508
Total cold run time: 53924 ms
Total hot run time: 50817 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.35 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6396f254edf901167007d1dc5da0673f8daf8339, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.04	0.04
query3	0.26	0.09	0.08
query4	1.61	0.12	0.11
query5	0.29	0.26	0.25
query6	1.20	0.68	0.67
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.58	0.50	0.51
query10	0.56	0.55	0.55
query11	0.14	0.10	0.09
query12	0.14	0.11	0.10
query13	0.64	0.62	0.61
query14	1.08	1.05	1.06
query15	0.87	0.88	0.86
query16	0.39	0.42	0.40
query17	1.15	1.17	1.16
query18	0.22	0.20	0.20
query19	2.08	2.01	2.02
query20	0.02	0.02	0.01
query21	15.39	0.27	0.14
query22	5.38	0.06	0.04
query23	16.12	0.28	0.10
query24	1.92	0.26	0.31
query25	0.09	0.07	0.07
query26	0.15	0.13	0.13
query27	0.06	0.06	0.05
query28	3.21	1.17	0.97
query29	12.57	3.94	3.15
query30	0.27	0.13	0.13
query31	2.82	0.62	0.41
query32	3.23	0.58	0.50
query33	3.16	3.23	3.21
query34	16.33	5.39	4.77
query35	4.77	4.75	4.78
query36	0.66	0.49	0.48
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.19	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.04
Total cold run time: 98.19 s
Total hot run time: 28.35 s

@doris-robot
Copy link

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.49% (19271/36716)
Line Coverage 35.96% (179009/497780)
Region Coverage 32.36% (138764/428826)
Branch Coverage 33.32% (60062/180271)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.52% (25739/35991)
Line Coverage 54.15% (268908/496603)
Region Coverage 51.64% (223734/433236)
Branch Coverage 53.07% (96059/181003)

@zclllyybb zclllyybb changed the title [Fix](profile) Remove additional timing after OLAP_SCAN_OPERATOR task completion [Fix](profile) Split operators CloseTime from ExecTime Feb 2, 2026
@zclllyybb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32132 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf35d5e84399aebd44bd8aa8881cac3335a4eb0a, data reload: false

------ Round 1 ----------------------------------
q1	17608	5377	5063	5063
q2	1985	312	195	195
q3	10242	1305	771	771
q4	10218	861	314	314
q5	7477	2132	1906	1906
q6	195	180	150	150
q7	902	725	595	595
q8	9259	1343	1122	1122
q9	5153	4809	4831	4809
q10	6826	1940	1574	1574
q11	517	306	282	282
q12	335	384	214	214
q13	17813	4077	3244	3244
q14	243	248	216	216
q15	905	830	810	810
q16	669	672	620	620
q17	658	825	444	444
q18	6828	6665	7520	6665
q19	1316	1057	655	655
q20	447	377	240	240
q21	2885	2294	1960	1960
q22	384	324	283	283
Total cold run time: 102865 ms
Total hot run time: 32132 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5522	5666	5554	5554
q2	279	357	260	260
q3	2444	2909	2515	2515
q4	1410	1823	1420	1420
q5	4734	4704	4525	4525
q6	233	179	154	154
q7	2049	2045	1830	1830
q8	2620	2369	2368	2368
q9	7421	7513	7304	7304
q10	2845	3097	2528	2528
q11	550	449	433	433
q12	611	678	547	547
q13	3541	3991	3262	3262
q14	278	301	255	255
q15	831	804	783	783
q16	627	675	641	641
q17	1079	1210	1200	1200
q18	7614	7334	7381	7334
q19	820	801	777	777
q20	1984	2067	1894	1894
q21	4492	4205	4113	4113
q22	544	540	515	515
Total cold run time: 52528 ms
Total hot run time: 50212 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 28.36 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf35d5e84399aebd44bd8aa8881cac3335a4eb0a, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.12	0.11
query5	0.26	0.24	0.24
query6	1.16	0.68	0.67
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.57	0.55	0.54
query11	0.14	0.09	0.09
query12	0.14	0.11	0.10
query13	0.64	0.61	0.61
query14	1.06	1.06	1.07
query15	0.88	0.88	0.87
query16	0.38	0.40	0.42
query17	1.13	1.13	1.11
query18	0.23	0.22	0.21
query19	2.13	2.00	2.05
query20	0.02	0.01	0.01
query21	15.41	0.28	0.14
query22	5.52	0.06	0.05
query23	16.40	0.29	0.11
query24	1.04	0.28	0.25
query25	0.08	0.08	0.08
query26	0.16	0.14	0.13
query27	0.08	0.05	0.06
query28	3.30	1.16	0.97
query29	12.57	3.86	3.15
query30	0.29	0.13	0.12
query31	2.82	0.65	0.42
query32	3.29	0.60	0.50
query33	3.33	3.22	3.21
query34	16.39	5.37	4.76
query35	4.78	4.78	4.80
query36	0.65	0.49	0.50
query37	0.11	0.07	0.07
query38	0.07	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.16
query41	0.09	0.03	0.03
query42	0.05	0.03	0.03
query43	0.06	0.03	0.04
Total cold run time: 98.12 s
Total hot run time: 28.36 s

@hello-stephen
Copy link
Contributor

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.53% (19351/36838)
Line Coverage 35.99% (179659/499160)
Region Coverage 32.36% (139285/430414)
Branch Coverage 33.34% (60287/180838)

@hello-stephen
Copy link
Contributor

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.61% (25855/36103)
Line Coverage 54.24% (270093/497962)
Region Coverage 51.90% (225654/434807)
Branch Coverage 53.24% (96666/181564)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants