Written by 
I started with a file that was ~21.9 GB. I have limited computing power, so I decide to split the file into geographical regions (climate classifications). I read them in using xarray, and when I exported the subfiles using .to_netcdf, it takes a really long time and the file size was much bigger – up […]
Introduction Slack handles a lot of log data. In fact, we consume over 6 million log messages per second. That equates to over 10 GB of data per second! And it’s all stored using Astra, our in-house, open-source log search engine. To make this data searchable, Astra groups it by time and splits the data […]