Data lakes flow with business opportunities
Cloud Computing

Data lakes flow with business opportunities

The world's cities are littered with large red brick warehouse buildings either crumbling into disrepair or that have been repurposed into living and office space. The buildings were rejected as storage of goods moved from sacks that needed to be hoisted on the crane jibs built into their walls, to the container, which in turn created new ports outside of cities and transformed logistics. Data warehouses could become the technology equivalent to the city centre warehouse you see in Amsterdam or London, as they are being replaced by the data lake.

Data lakes provide CIOs and organisations with an increased fluidity to how they ingest and manage data. Lakes create ecosystems in the natural world, bringing together feathered, furred and scaled animals. Organisations pushing off into data lake waters are finding a wealth of ecosystems that they can channel into their lake and in doing so, their own environment thrives.

"Across all vertical markets, organisations are recognising that they need data to drive advantage and we have seen the adoption of data lakes increase in the last 12 months," says Sacha Tomey, CTO and Director of Adatis, a data analytics specialist in Sofia and the UK.

Driving this growth is a business-wide understanding of the importance of data in discovering market opportunities. Some sectors are seeing margins decrease as a result of low-cost entrants and/or online rivals. For other verticals, globalisation and changing customer behaviour requires an increased focus and understanding of the customer or to improve the business operations.

Research by business intelligence provider MicroStrategy found that 38% of organisations had their entire analytics capability in the cloud and that 60% of the surveyed organisations had a Chief Data Officer (CDO), thus increasing the demand for the ability to access and interrogate data.

Demand for data lakes is likely to continue, 59% of organisations plan on hiring data analysts and 65% of UK CIOs report "improved efficiency and productivity" from their analytics use, the MicroStrategy findings report.


Business benefits

One organisation already bearing fruits from its use of data lakes is the Food Standards Agency (FSA), a UK governmental body that administers food safety. "There is shed loads of data in the food sector, so we asked ourselves: ‘why don't we use that and as an open and transparent organisation publish the findings out in the open'," says Julie Pierce, Director of data and digital at the FSA.

As a result, Pierce and the Food Standards Agency has already delivered business benefits and improved food safety for society. The Food Standards Agency has a surveillance element to its remit, monitoring food crime and health standards.

"We use a lot of HMRC (the UK taxation office) and Defra (Department for the Environment, Food and Rural Affairs) data as well as sources from other countries and food industry itself. We use the data lake to join all this data up," she says.

Using a data lake, the Food Standards Agency identified Aflatoxin dangers; Aflatoxin is a fungus found on figs, which can be predicted using weather data. As a result, the agency was able to stop imports leaving and arriving in UK ports. Pierce says this has reduced wastage in ports and business processes and prevented possible illnesses that will add pressure on the UK's under funded healthcare system.

As a scalable cloud technology, a data lake allows CIOs to be inventive and creative about the number and types of data resources they pour into the lake. James Holmes, CIO for international shipping insurance provider North P&I says data lakes and a general data focus is helping the organisation better understand risk levels and the organisation is using a wide range of shipping data, such as AIS the marine equivalent to GPS. North P&I is also taking the opportunity to use that data to collaborate with marine startups to drive innovation in underwriting and claims.

"We have over 30 years' worth of data and I firmly believe there is a fantastic opportunity," Holmes says.

"One of the business benefits is the ability for a lake to consume many types of data, which provides more business opportunities," Tomey says. "The data warehouse is now for more traditional data, such as your transactional information. You have to think about what and why you are storing data in a warehouse.

"With a lake, the deeper the lake the more innovation the data a user can open up," Tomey adds.

Lakes are providing organisations with business improvements. CIO Holmes says his financial services business has: "a better understanding of risk, so that intelligence helps us make better informed decisions," which benefits North P&I and its customers. Loss prevention is improving, and the CIO expects to see the teams within the business increasing their understanding as a result of greater exposure to data.

"This is fundamentally about moving to educate the business and there is a growing understanding and appetite for data and data driven ways of working," Holmes adds.

"It can help widen the scope of what is possible," says Tomey, adding that as more sectors have to offer personalised services, then using broad data sets from within and outside of the organisation will enable greater customer understanding.

David Espley, CTO for global legal and business information provider LexisNexis says data lakes enable the organisation to adopt artificial intelligence (AI) to create new insights and analysis of the legal archive the business has.

But Ian Cohen, Group CIO of international transport company Addison Lee warns peers that data lakes must not turn into "data swamps".

"Poor tools on poor data will get the wrong outcome. As an industry we have the ability to use buzzwords way before the actual solutions are ready for the prime time. There are some huge claims that organisations are making about doing AI and in most cases, they are just doing machine learning or advanced pattern matching."


Cloud lakes

The adoption of cloud technology is helping data lakes swim to the fore. For Pierce at the Food Standards Agency this means a series of lakes; the government agency creates a lake to solve a business issue, which for an organisation with an agenda to be open, is more effective than a single super-lake.

"You can be more agile and bring data in and out as and when you need it. I challenge data lakes that are monoliths, you only build a data lake when you know what the question is," the FSA data leader and former CIO has seen, "people spend millions building something and then they are not sure what question to ask".

Holmes adds that as data lakes spread, so CIOs will have an increasing demand for cloud technology skills and data scientists.

Tomey at Adatis agrees, "be driven by the business objective" and Adatis see cloud technology increasing the flow of data lakes. "Storage is cheap, so you can take on all the data, you have scalability on the cloud, which allows you to be ambitious, but it is important to think about the governance."


« Just don't call it 'Deep': AI in Latin American business


AlienVault and LogRhythm NextGen SIEM: Buyer's guide and reviews »
Mark Chillingworth

Mark Chillingworth is a CIO and CTO journalist, ghost writer, moderator and advisor with over 11 years experience. From 2010 to 2016 he was Editor in Chief of the award-winning CIO UK. In 2011 he created the CIO 100, an annual transformation power list of the UK’s most influential CIOs and launched the UK’s first CIO Podcast in 2016.

  • Mail


Do you think your smartphone is making you a workaholic?