1. Computer problem? Tech Support Guy is completely free -- paid for by advertisers and donations. Click here to join today! If you're new to Tech Support Guy, we highly recommend that you visit our Guide for New Members.

Generating Billions of Numerical Data sets

Discussion in 'Business Applications' started by Bartolome, May 4, 2019.

Thread Status:
Not open for further replies.
Advertisement
  1. Bartolome

    Bartolome Thread Starter

    Joined:
    May 4, 2019
    Messages:
    6
    Hey guys,
    I am currently working on some data sets. As of late, I have been generating numerical data sets in google sheets, but after about 20000 tables filled, the sheet starts to seriously lag. I need to create approximately 8 billion number sequences and store them in a table. I am working off of a Chrome OS system (Chromebook) and am looking for a free way to handle this amount of data.

    Thanks in advance,
    Bartolome
     
  2. SpywareDr

    SpywareDr

    Joined:
    Mar 30, 2014
    Messages:
    3,120
  3. Bartolome

    Bartolome Thread Starter

    Joined:
    May 4, 2019
    Messages:
    6
    Thank you for the PDF SpywareDr. The code presented in the work however was not fit for a Chromebook, which is what I'm working from. If there is any other way to generate a data table with the aforementioned number of sequences, please let me know.

    Again, thank you.
     
  4. SpywareDr

    SpywareDr

    Joined:
    Mar 30, 2014
    Messages:
    3,120
    :confused: I rather doubt you or your Chromebook would live long enough it to generate "Billions of Numerical Data sets".
     
  5. Bartolome

    Bartolome Thread Starter

    Joined:
    May 4, 2019
    Messages:
    6
    I am writing an application that needs to store and analyze large amounts of electrical and temperature data.

    Basically I need to store large amounts of hourly electricity usage measurements for the past several years and for many years to come for tens of thousands of locations and then analyze the data in a not very complex manner.

    The information that I need to store (for now) is Location ID, Timestamp (Date and Time), Temperature and Electricity Usage.

    About the amount of the data that needs to be stored, this is an approximation, but something along those lines:
    20 000+ locations, 720 records per month (hourly measurements, approximately 720 hours per month), 120 months (for 10 years back) and many years into the future. Simple calculations yield the following results:

    20 000 locations x 720 records x 120 months (10 years back) = 1 728 000 000 records.

    These are the past records, new records will be imported monthly, so that's approximately 20 000 x 720 = 14 400 000 new records per month.

    The total locations will steadily grow as well.

    On all of that data, the following operations will need to be executed:

    1. Retrieve the data for a certain date AND time period: all records for a certain Location ID between the dates 01.01.2013 and 01.01.2017 and between 07:00 and 13:00. (Copy and paste into other databases)
    The data will be written monthly but will be read by hundreds of users (at least) constantly, so the read speed is of significantly more importance.

    My questions are the following:

    1. (NOTE: I'm using a Chromebook to do this); where can I edit this data without lagging (google sheets lags after 20000 spots are filled)?
    2. Can I do this for free (i.e. not buying additional software or using online applications that cost money)?
     
  6. Sponsor

As Seen On
As Seen On...

Welcome to Tech Support Guy!

Are you looking for the solution to your computer problem? Join our site today to ask your question. This site is completely free -- paid for by advertisers and donations.

If you're not already familiar with forums, watch our Welcome Guide to get started.

Join over 733,556 other people just like you!

Thread Status:
Not open for further replies.

Short URL to this thread: https://techguy.org/1226674

  1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.
    Dismiss Notice