model.save(model_path) has error when the file size exceed the 773Kb. #5988
Unanswered
howtokim
asked this question in
Community | Q&A
Replies: 1 comment
-
|
Model save failing above 773KB — unusual limit! Here's how to fix: Likely cause: Filesystem or buffer issue 1. Check disk space df -h .
# Ensure enough space2. Try different save format # Instead of single file
model.save_pretrained(model_path, max_shard_size="500MB")
# Or use safetensors
model.save_pretrained(model_path, safe_serialization=True)3. Increase buffer size import io
io.DEFAULT_BUFFER_SIZE = 1024 * 1024 # 1MB buffer4. Check for memory issues import torch
torch.cuda.empty_cache()
# Save to CPU first
model = model.cpu()
model.save(model_path)5. Permissions chmod -R 755 /path/to/model_dir6. Try torch.save directly torch.save(model.state_dict(), "model.pt")Debug: import os
print(os.statvfs(".").f_bsize) # Block size773KB is suspiciously specific — might be a config limit somewhere. We handle model serialization at RevolutionAI. What's your exact error message? |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
bellow source program has no problem to be saved at normal.
but, if the file size exceed the 773K bytes.
the program is stopped at the line "model.save(model_save_path)"
there wa no except message. just stopped at the line.
<< source program>>
ticker = ticker.replace('KRW-', '')
# 매수 의견을 기록할 리스트 초기화
self.acc_reward_cnt = 0
self.acc_action_cnt = 0
I checked the disk's free space and folder size.
and also check the CPU and Memory during the program is running.
it seems there is nothing problem.
Beta Was this translation helpful? Give feedback.
All reactions