# Generating Billions of Numerical Data sets

Discussion in 'Business Applications' started by Bartolome, May 4, 2019.

Not open for further replies.

Joined:
May 4, 2019
Messages:
6
Hey guys,
I am currently working on some data sets. As of late, I have been generating numerical data sets in google sheets, but after about 20000 tables filled, the sheet starts to seriously lag. I need to create approximately 8 billion number sequences and store them in a table. I am working off of a Chrome OS system (Chromebook) and am looking for a free way to handle this amount of data.

Bartolome

2. ### SpywareDr

Joined:
Mar 30, 2014
Messages:
3,120

Joined:
May 4, 2019
Messages:
6
Thank you for the PDF SpywareDr. The code presented in the work however was not fit for a Chromebook, which is what I'm working from. If there is any other way to generate a data table with the aforementioned number of sequences, please let me know.

Again, thank you.

4. ### SpywareDr

Joined:
Mar 30, 2014
Messages:
3,120
I rather doubt you or your Chromebook would live long enough it to generate "Billions of Numerical Data sets".

Joined:
May 4, 2019
Messages:
6
I am writing an application that needs to store and analyze large amounts of electrical and temperature data.

Basically I need to store large amounts of hourly electricity usage measurements for the past several years and for many years to come for tens of thousands of locations and then analyze the data in a not very complex manner.

The information that I need to store (for now) is Location ID, Timestamp (Date and Time), Temperature and Electricity Usage.

About the amount of the data that needs to be stored, this is an approximation, but something along those lines:
20 000+ locations, 720 records per month (hourly measurements, approximately 720 hours per month), 120 months (for 10 years back) and many years into the future. Simple calculations yield the following results:

20 000 locations x 720 records x 120 months (10 years back) = 1 728 000 000 records.

These are the past records, new records will be imported monthly, so that's approximately 20 000 x 720 = 14 400 000 new records per month.

The total locations will steadily grow as well.

On all of that data, the following operations will need to be executed:

1. Retrieve the data for a certain date AND time period: all records for a certain Location ID between the dates 01.01.2013 and 01.01.2017 and between 07:00 and 13:00. (Copy and paste into other databases)
The data will be written monthly but will be read by hundreds of users (at least) constantly, so the read speed is of significantly more importance.

My questions are the following:

1. (NOTE: I'm using a Chromebook to do this); where can I edit this data without lagging (google sheets lags after 20000 spots are filled)?

As Seen On