MemoryError

[2718455213wcx]

I used drian in lopai to parse the thunderbird dataset (29.8gb) without getting MemoryError, but I did get MemoryError when I parsed the split thunderbird dataset (2.92gb) using Brain in logpai. When I parse a 1000m thunderbird dataset there is no MemoryErro.Why is that? Can Brain only parse data sets around 1gb in size?
Traceback (most recent call last):
File "E:\logbert-main\TBird\data_process.py", line 137, in
parse_log(data_dir, output_dir, log_file, parser_type)
File "E:\logbert-main\TBird\data_process.py", line 77, in parse_log
parser.parse(log_file)
File "E:\logbert-main\TBird..\logparser\Brain.py", line 58, in parse
group_len, tuple_vector, frequency_vector = self.get_frequecy_vector(
File "E:\logbert-main\TBird..\logparser\Brain.py", line 261, in get_frequecy_vector
set.setdefault(str(lenth), []).append(token)
MemoryError

[gaiusyu]

You can try splitting the data set into small enough chunks until you don't get any memory errors. If your PC has more memory, Brain will be able to parse larger data sets. Maybe I will improve Brain to save more memory overhead in the future😂

[a13382735176]

thank you man！