Agree with your comment, the only issue was that we would be left with such a small test set with minority class extremely under-represented. Ideally, we would have used simple under-sampling of the majority class, but that wasn’t leading to useful dataset. So yes, I think what you suggested is the best way to do it.