feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline by vividf · Pull Request #115 · tier4/AWML

vividf · 2025-10-08T15:13:51Z

AWML Unified Deployment Framework

Introduced a unified deployment and evaluation framework for AWML, significantly improving consistency, scalability, and maintainability across projects

PR feat(deployment):prototype deployment framework #137: Core deployment framework (ONNX Runtime + TensorRT support)
PR feat(deployment): centerpoint deployment integration #181: CenterPoint integration into the unified pipeline

Key Contributions

1. Unified Deployment Pipeline

Standardized model deployment flow: PyTorch → ONNX → TensorRT
Eliminated duplicated and project-specific deployment scripts
Enabled config-driven, reusable pipelines across multiple models
Supported seamless backend switching (PyTorch / ONNX / TensorRT)

2. Cross-Backend Verification

Implemented automatic output comparison across backends
Ensured numerical consistency between PyTorch, ONNX Runtime, and TensorRT
Prevented silent accuracy regressions during deployment
Improved reliability for production-level inference

3. Integrated Evaluation Framework

Built unified evaluation pipeline based on T4MetricV2
Supports:
- mAP / mAPH
- Per-class metrics
- Threshold-based evaluation
- Enables consistent benchmarking across different backends and models

How to run

python -m deployment.cli.main centerpoint deployment/projects/centerpoint/config/deploy_config.py   projects/CenterPoint/configs/t4dataset/Centerpoint/second_secfpn_8xb16_121m_j6gen2_base_amp_t4metric_v2.py   --rot-y-axis-reference

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

KSeangTan

Only review the first part, please address the comments accordingly first.

KSeangTan · 2026-04-21T06:35:51Z

+    Args:
+        log_file_path: Absolute or resolved path to the log file.
+    """
+    path = os.path.abspath(os.path.expanduser(log_file_path))


Use pathlib instead

KSeangTan · 2026-04-21T06:36:39Z

+        ArgumentParser with deployment arguments
+    """
+    if parser is None:
+        parser = argparse.ArgumentParser(


Why parser can be None?

KSeangTan · 2026-04-21T06:37:00Z

+from deployment.projects.registry import project_registry
+
+
+def _discover_project_packages() -> List[str]:


Sequence instead of List

KSeangTan · 2026-04-21T06:39:17Z

+            raise ValueError("Missing 'export' section in deploy config.")
+        export_raw = self.deploy_cfg.get("export")
+        if export_raw is not None and not isinstance(export_raw, Mapping):
+            raise TypeError("deploy config 'export' must be a mapping.")


mapping might not be an appropriate type in the error message here, it should be Dict instead?

KSeangTan · 2026-04-21T06:40:28Z

+            raise TypeError("deploy config 'components' must be a mapping.")
+
+    @staticmethod
+    def _parse_deploy_log_path(raw: Optional[Any]) -> Optional[str]:


I dont think we should accept Any here, please make it str only. These type checking should actually be done by IDE in static checking instead of runtime.

KSeangTan · 2026-04-21T06:41:34Z

+        """Require a non-empty checkpoint path that exists as a regular file."""
+        if not isinstance(checkpoint_path, str):
+            raise TypeError(f"checkpoint_path must be a string, got {type(checkpoint_path).__name__}.")
+        path = os.path.expanduser(checkpoint_path.strip())


Use pathlib instead

KSeangTan · 2026-04-21T06:44:05Z

+
+    def _needs_cuda_device(self) -> bool:
+        """Determine if current deployment config requires a CUDA device."""
+        if self.export_config.should_export_tensorrt:


I believe this function is unnecessary complicated, for example, we only need to check if tensorrt is enabled since we will use only tensorrt with cuda right?

KSeangTan · 2026-04-21T06:44:32Z

+        return self._verification_config
+
+    @property
+    def devices(self) -> DeviceConfig:


device_config

KSeangTan · 2026-04-21T06:45:41Z

+        if not output_names:
+            output_names = ("output",)
+        settings_dict = {
+            "opset_version": onnx_config.opset_version,


self._onnx_export_config.opset_config

KSeangTan · 2026-04-21T06:46:32Z

+
+
+def _normalize_dynamic_axes(raw: Mapping[str, Any]) -> Dict[str, Dict[int, str]]:
+    """Normalize dynamic_axes inner dict keys to int."""


What does it mean by normalizing?

vividf changed the title ~~feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline~~ WIP: feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline Nov 14, 2025

vividf force-pushed the feat/new_deployment_and_evaluation_pipeline branch from 68d1700 to 7ac5b8b Compare January 29, 2026 08:08

vividf and others added 10 commits March 11, 2026 13:24

chore: init

c77cf92

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add deployment framework intro

117dead

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

feat(deployment):prototype deployment framework (#137)

41a3e9a

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

feat(deployment): restructure deployment framework (#160)

d80db0a

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add missing pipeline directory and docstring

a1edc32

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

feat(deployment): refactor config and clean code (#180)

a362b27

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: refactor base config

d99296c

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up code: device spec, remove unused fucntion .etc

e23c210

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: refactor export compenent

dd46052

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix more Device spec

2b28f60

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf force-pushed the feat/new_deployment_and_evaluation_pipeline branch from 5256306 to 2b28f60 Compare March 11, 2026 04:27

vividf and others added 14 commits March 25, 2026 20:49

chore: enable only trt evaluation

85f0d64

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix verification

e7439a6

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix main

ca6f730

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix base evaluator

d150f5e

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up base

23df89b

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add base logger

36d7568

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: update docs

5bd5b76

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: add more readme

0eed1f1

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: base add -> None for __init__

d706ff7

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: remove task profile

9dfb6a1

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix matrix report base

a75d739

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean metrics

efa71d1

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: update project registry base

0c7c57c

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

feat(deployment): centerpoint deployment integration (#181)

ddc5106

Signed-off-by: vividf <yihsiang.fang@tier4.jp> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

vividf self-assigned this Apr 14, 2026

vividf added 2 commits April 16, 2026 16:16

chore: improve docs

d0b491d

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up docs

5080b40

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf added 4 commits April 16, 2026 20:56

chore: clean up code

a46f933

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix verification

b7b6954

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: fix verification 2

f873e85

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

chore: clean up __init__.py

9329b11

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf marked this pull request as ready for review April 20, 2026 12:41

vividf requested a review from KSeangTan as a code owner April 20, 2026 12:41

vividf changed the title ~~WIP: feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline~~ feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline Apr 20, 2026

chor: clean code

1d4c27f

Signed-off-by: vividf <yihsiang.fang@tier4.jp>

vividf force-pushed the feat/new_deployment_and_evaluation_pipeline branch from 8756669 to 1d4c27f Compare April 20, 2026 13:05

KSeangTan requested changes Apr 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline#115

feat(AWML): introduce new deployment and onnx/tensorRT evaluation pipeline#115
vividf wants to merge 31 commits intomainfrom
feat/new_deployment_and_evaluation_pipeline

vividf commented Oct 8, 2025 •

edited

Loading

Uh oh!

KSeangTan left a comment

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

KSeangTan Apr 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		from deployment.projects.registry import project_registry


		def _discover_project_packages() -> List[str]:



		def _normalize_dynamic_axes(raw: Mapping[str, Any]) -> Dict[str, Dict[int, str]]:
		"""Normalize dynamic_axes inner dict keys to int."""

Conversation

vividf commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

AWML Unified Deployment Framework

Key Contributions

1. Unified Deployment Pipeline

2. Cross-Backend Verification

3. Integrated Evaluation Framework

How to run

Uh oh!

KSeangTan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

vividf commented Oct 8, 2025 •

edited

Loading