-
Notifications
You must be signed in to change notification settings - Fork 527
[SYTEMDS-3902] Transfer scipy compressed matrices to java runtime #2379
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Conversation
|
@Baunsgaard @mboehm7 https://github.com/apache/systemds/blob/8de93a1f996348c260d48c1c29340d5a88005e6f/src/main/python/systemds/context/systemds_context.py#L772C4-L772C20 the naming "from_numpy" is misleading since we support now scipy compressed matrices as well We could also use something like "from_pydata" which merges the two separate methods from_numpy and from_pandas to single method, internally we could route to the corresponding handler based on the input instance type. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #2379 +/- ##
============================================
+ Coverage 71.51% 71.54% +0.03%
- Complexity 47441 47466 +25
============================================
Files 1539 1539
Lines 182605 182631 +26
Branches 35916 35919 +3
============================================
+ Hits 130585 130665 +80
+ Misses 42028 41972 -56
- Partials 9992 9994 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This sounds good to me, unifying the API would be fine.
|
|
sure, I agree |
7a35b34 to
bc56a18
Compare
This commit implements optimized data transfer for Scipy sparse matrices from Python to the Java runtime. Key changes include the addition of `convertSciPyCSRToMB` and `convertSciPyCOOToMB` in the Java utility layer to directly handle compressed sparse row and coordinate formats. On the Python side, the `SystemDSContext` now supports a `sparse_data_transfer` flag and a new `from_py` method to unify data ingestion. These updates allow sparse data to be transferred without being converted to dense arrays, improving efficiency. Additionally, several data conversion methods were refactored for better maintenance.
bc56a18 to
cce93be
Compare
This commit implements direct transfer support for Scipy sparse matrices (CSR and COO formats) between Python and the Java runtime.
MatrixBlockrepresentations.SystemDSContextto support asparse_data_transferflag and extendedfrom_numpyto acceptscipy.sparsematrices.Benchmark results:
