The 'big bad data police' will call if we don't get more efficient

Why digital growth and energy efficiency don’t have to be mutually exclusive

What will be impact of big data on our carbon footprint? Let’s assume the Internet of Things by 2040 consists of 50+ billion devices. These devices send data, maybe an average of 50,000 data points per day per device, and let’s average each data set at 100 bytes. That gives us 2.5 Exabytes of data per day in 2040. A present day statistic is even more revealing. In 2000 the Amsterdam International Internet Exchange processed 18 Terabytes of data per day. In 2015 it processed 27,842 Terabytes per day. Data traffic increased 1.546 times in 15 years’ time.

ICT consumes on average 6-9% of global energy, data centres about 2-3%. If we apply this percentage to data growth, we face an interesting challenge. Governments, companies and individuals produce and consume increasingly more data. Data needs electricity for it to be stored, processed and sent. So big data assumes either more electricity will be generated, which will be polluting when power remains fossil fuels based. Or data will be more efficiently handled against the same amount of power. Let’s envision both scenarios and their effects.

Data storing, processing and sending gains little efficiency by 2040

Electricity supply is unable to keep up with the demands of big data without exploding carbon pollution. How would the world look? Would governments step in to regulate Facebook usage, only between 12-15h00? Would citizens have right to only 12 Google searches per day? Store only 365 pics per year? Ban nonsense tweets? Alternate smart cities supplying digital services on pair – un-pair days? Tax companies on data usage? Cap data volumes by citizen, by income, by importance? This might seem laughable now, but data-rationing is a likely outcome if humanity does not tackle data growth and the underlying power consumption.   

A proposed solution for greater technological efficiency

Another scenario is where technology saves the day. Today an estimated 10 million servers run idle, consuming energy and not processing any tasks. Potential efficiency gains are massive. Cold storage uses compute power only when necessary. But cold storage needs to rewrite standard Operating Systems and adjust your hardware.

It would be efficient to write application specific, e.g. database Operating Systems, and adjust hardware around the specific needs of a database. The present all-purpose Operating Systems and Virtual Machines are incredibly energy inefficient in handling the different workload types that are thrown at them. Just like the ‘Ford T’ model evolved in countless car types tailored towards usage, ICT hardware and Operating Systems should evolve into a plethora of different versions each customised to usage and energy efficiency.

Distributed computing could be another part of the solution. Instead of having all data grouped in one central place, the data centre, and then processed, it would make sense to have data processing at the base of a cell tower, or even in a mobile phone. Carriers could even propose to use idle mobile phone capacity in return for a loyalty program. This would divide up data processing and assign it to the link in the value chain (device – network – data centre) that makes most sense in terms of energy efficiency. This division of data storing and processing could be context sensitive, assigning tasks at certain moments in time or under certain conditions (available solar, wind energy).

Why we shouldn’t need to worry about a future ‘data police force’

So should we be afraid of the data-police? I hope not, but there is a challenge in powering big data. And I see cause for both despair and happiness. Separate government agencies, one promoting ‘digital growth’, the other ‘energy efficiency’ are just as absurd as companies where the CIO is not concerned about electricity bills because it is paid by the facilities department. These contradictions push us towards a data-rationing future. App developers that write code for smart phones, on the other hand, know about battery constraints and take energy efficient programming deadly serious. They should rewrite the whole ICT-stack. And for the ICT vendors, lighten up and evolve your Ford T models (Windows, Mac OS, Linux) into countless tailored variations that are possible once you put your hearts and minds behind it. And ensure big data is clean, sustainable and unconstrained.