Data Mining

Big Data in SA Businesses: What is the Secret to Success?

“BIG Data” conjures up images of…what? The term jumps out at us from the trade media almost as often as “Cloud” these days.  Big computers? Big files? Big screens? Maybe “BIG headache” is more apt. What is “data”? It’s the plural of “datum” – a piece of information, the word being derived nearly 300 years ago from the Latin meaning “something given”.

Our digitally connected, always on, access from anywhere at any time world has enabled us to capture and store every item of information that we encounter, whether from our past or our present. “Information” includes everything perceptible by humans, and more. Tangible, intangible, fact or fiction – it makes no difference, as long as it can be digitised. Language, sound or image – all can be added to the store. The sources range from chats, logs, sensors, transactions and files, to telemetry, broadcasts and satellites. At the top end, we have the Large Hadron Collider and the Square Kilometre Array (SKA) generating more data in seconds than we used to acquire in years.

We started digitising information in bits, which became bytes, and we have steadily progressed through the kilo-, mega-, giga-, tera-, peta- and exa- prefixes to cope with the escalating size of the datasets we are creating… which creates “Big Data” – pieces of information that are too big, too complex to be managed within the traditional database structures developed in the last 50 years… which creates the need for new processing methodologies to cope with seeking value from this vast pool of information, using the global networks, strategically located data centres and software tools to organise, search and analyse the data. The network is the key ingredient, as big data is too big to move – it resides in the data centre storage facilities, wherever they may be, and cannot be moved en masse to another location. Available, affordable and reliable broadband access is essential.

The SKA and the network highlight both the challenge and the opportunity for South Africa. Providers like Dimension Data, Business Connexion and Hetzner (for example) offer excellent local data centre facilities. Their international relationships enable them to draw on a skills pool to supplement the South African expertise that is still passing the embryo stage outside of the relatively few large enterprises that are engaging with big data. However, the implementation of the SKA over the next ten years will require a geometrically greater “pipe” than currently exists in or around the country to create access to the data from the Array sites by the users around the globe. Meeting this need will enhance the opportunities for smaller enterprises to share the benefits.

Successful utilisation of big data follows a change in management approach from linear, structured models to systems thinking where the relationships between the connected components are better understood from a holistic perspective. In managing our information systems, we have already replaced the spreadsheet and the dashboard with the business intelligence (BI) methodology. However, BI is focused on measuring movements and detecting trends, whereas big data offers the opportunity to infer, reveal and predict. Our computer resources move from being digital support systems to digital nervous systems, feeling the sensations experienced by the enterprise within the context of its environment.

This is probably the biggest challenge to succeeding with extraction of value from big data. As one analyst put it, “Big data requires big judgement.” We have often used the expression “seeing the big picture”. It is not only the gathering and storage of information, it is the ability to sift the relevant from the irrelevant and visualising the relationships that add value and identifying where change will make a positive contribution.

The visible pioneers of using big data in their business models are companies such as Google, Amazon and Facebook. Less visible but no less influential are government entities such as the USA’s National Security Agency. Recent events have highlighted why big data might be a Bad Thing. Is there a line to be drawn between using information your customers and potential customers make available (wittingly or unwittingly) in refining your marketing and selling tactics, and collecting every scrap of information about everyone you can, in the hope of using it for or against them at some point in the future? That governments spy on their (and others’) citizens is a long-established fact of life. We should not be surprised that they use modern technology to do so on the widest scale. However, the risk to the vast majority of individuals is minimal, compared to the benefits of being better served by better informed private and public sector providers, using the very same technology.

The question of how the data is used needs to be answered. Is it good quality, verifiable information? The ageless mantra of “garbage in, garbage out” is every bit as applicable today. Is the data relevant to the usage within the business? Does the usage represent an invasion of privacy or will it lead to unfair profiling of the people it describes? There are definite areas where big data is a Good Thing – such as early detection of infectious diseases, improvement of learning, more efficient commuting, accurate weather forecasting – areas where the public good is served by the rapid aggregation and analysis of information leading to better decisions.

Used ethically, we can decide what problems need solving, and then make use of big data to support the development of appropriate solutions. Products and services can be more rapidly developed and delivered, based on reliable models and more accurate prediction and optimisation. Big data is scalable, from enterprise to global level. Management of big data needs a new approach, using a combination of skills supporting the “big picture”. South Africa can ride the wave but must educate its business and information managers, up to C-level, in how best to learn this new approach and apply it effectively.


Adrian Schofield is Manager: Applied Research at Joburg Centre for Software Engineering at Wits University


« Reducing Bill Shock with Intelligent Roaming


New Zealand Goes from Film Set to IT Outsourcing Hub »
Adrian Schofield

Adrian Schofield has spent more than half his life working in and for the South African ICT industry at national, regional and global levels. For the last 10 years, he has conducted applied research for the Joburg Centre for Software Engineering and is currently serving as a Board member of the Institute of Information Technology Professionals South Africa (IITPSA), the Africa ICT Alliance (AfICTA)... See More

  • Mail