MiniRocket with MMAP and Dataloaders #909
Unanswered
smagmaster
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi there,
First of I want to thank the tsai library it has been a tremendous help in every way, although I need some help now...
So I used the MiniRocketClassifier() model to classify movements of an EMG data. The first dataset worked perfectly fine but now I have a bigger dataset and I always get Memory errors all over the place.
So I have read the tutorial with using mmaps to not allocate to much memory on the RAM but for some reason it is not working properly. Here is what I have done so far:
`
X_train = np.load(f'{save}/X_train.npy', mmap_mode='r') ...
Convert numpy arrays to torch tensors
X_train_tensor = torch.tensor(X_train, dtype=torch.float32)
y_train_tensor = torch.tensor(y_train, dtype=torch.long)
X_test_tensor = torch.tensor(X_test, dtype=torch.float32)
y_test_tensor = torch.tensor(y_test, dtype=torch.long)
Create datasets and dataloaders
train_dataset = TensorDataset(X_train_tensor, y_train_tensor)
test_dataset = TensorDataset(X_test_tensor, y_test_tensor)
train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)
`
and then trained the model. Before I saved and loaded the variables with 'r' and 'c' but it did not make a difference. I have tried using batches as well to reduce the memory allocation but this is just bad for the minirocket model in general because mini-rocket does not work that way.
My Question here would be:
How can I use the mmaps and dataloader correctly to train a large dataset in tsai minirocket?
Any help is appreciated.
Thank you so much in advance and have a good day!
Best regards,
A desperate student
Beta Was this translation helpful? Give feedback.
All reactions