By Matthew WallBusiness reporter, BBC News
A recently launched Japanese-US satellite will monitor global rainfall, helping meteorologists use big data analytics to predict storms around the world
As Technology of Business begins a month-long series of features on the theme of Big Data, we kick off with a Q&A backgrounder answering some of those basic questions you were too afraid to ask.
What is big data exactly?
Good question. After all, we've always had large amounts of data haven't we, from loyalty card schemes, till receipts, medical records, tax returns and so on?
As Laurie Miles, head of analytics for big data specialist SAS, says: "The term big data has been around for decades, and we've been doing analytics all this time. It's not big, it's just bigger."
But it's the velocity, variety and volume of data that has merited the new term.
So what made it bigger?
Most traditional data was structured, or neatly organised in databases. Then the world went digital and the internet came along. Most of what we do could be translated into strings of ones and noughts capable of being recorded, stored, searched, and analysed.
There was a proliferation of so-called unstructured data generated by all our digital interactions, from email to online shopping, text messages to tweets, Facebook updates to YouTube videos.
As Technology of Business begins a month-long series of features on the theme of Big Data, we kick off with a Q&A backgrounder answering some of those basic questions you were too afraid to ask.
What is big data exactly?
Good question. After all, we've always had large amounts of data haven't we, from loyalty card schemes, till receipts, medical records, tax returns and so on?
As Laurie Miles, head of analytics for big data specialist SAS, says: "The term big data has been around for decades, and we've been doing analytics all this time. It's not big, it's just bigger."
But it's the velocity, variety and volume of data that has merited the new term.
So what made it bigger?
Most traditional data was structured, or neatly organised in databases. Then the world went digital and the internet came along. Most of what we do could be translated into strings of ones and noughts capable of being recorded, stored, searched, and analysed.
There was a proliferation of so-called unstructured data generated by all our digital interactions, from email to online shopping, text messages to tweets, Facebook updates to YouTube videos.
As the number of mobile phones grows globally, so does the volume of data they generate from call metadata, texts, emails, social media updates, photos, videos, and location
And the number of gadgets recording and transmitting data, from smartphones to intelligent fridges, industrial sensors to CCTV cameras, has proliferated globally, leading to an explosion in the volume of data.
These data sets are now so large and complex that we need new tools and approaches to make the most of them.
And the number of gadgets recording and transmitting data, from smartphones to intelligent fridges, industrial sensors to CCTV cameras, has proliferated globally, leading to an explosion in the volume of data.
These data sets are now so large and complex that we need new tools and approaches to make the most of them.
Early adopters of Big Data analytics have gained a significant lead
over the rest of the corporate world” Bain & Company
How much data is there?
Nobody really knows because the volume is growing so fast. Some say that about 90% of all the data in the world today has been created in the past few years.
According to computer giant IBM, 2.5 exabytes - that's 2.5 billion gigabytes (GB) - of data was generated every day in 2012. That's big by anyone's standards. "About 75% of data is unstructured, coming from sources such as text, voice and video," says Mr Miles.
And as mobile phone penetration is forecast to grow from about 61% of the global population in 2013 to nearly 70% by 2017, those figures can only grow. The US government's open data project already offers more than 120,000 publicly available data sets.
Where is it all stored?
The first computers came with memories measured in kilobytes, but the latest smartphones can now store 32GB and many laptops now have one terabyte (1,000GB) hard drives as standard. Storage is not really an issue anymore.
Nobody really knows because the volume is growing so fast. Some say that about 90% of all the data in the world today has been created in the past few years.
According to computer giant IBM, 2.5 exabytes - that's 2.5 billion gigabytes (GB) - of data was generated every day in 2012. That's big by anyone's standards. "About 75% of data is unstructured, coming from sources such as text, voice and video," says Mr Miles.
And as mobile phone penetration is forecast to grow from about 61% of the global population in 2013 to nearly 70% by 2017, those figures can only grow. The US government's open data project already offers more than 120,000 publicly available data sets.
Where is it all stored?
The first computers came with memories measured in kilobytes, but the latest smartphones can now store 32GB and many laptops now have one terabyte (1,000GB) hard drives as standard. Storage is not really an issue anymore.
The US National Security Agency has built a huge data centre in Bluffdale, Utah - codenamed Bumblehive - capable of storing a yottabyte of data - that's one thousand trillion gigabytes
For large businesses "the cost of data storage has plummeted," says Andrew Carr, UK and Ireland chief executive of IT consultancy Bull. Businesses can either keep all their data on-site, in their own remote data centres, or farm it out to "cloud-based" data storage providers.
A number of open source platforms have grown up specifically to handle these vast amounts of data quickly and efficiently, including Hadoop, MongoDB, Cassandra, and NoSQL.
For large businesses "the cost of data storage has plummeted," says Andrew Carr, UK and Ireland chief executive of IT consultancy Bull. Businesses can either keep all their data on-site, in their own remote data centres, or farm it out to "cloud-based" data storage providers.
A number of open source platforms have grown up specifically to handle these vast amounts of data quickly and efficiently, including Hadoop, MongoDB, Cassandra, and NoSQL.
Do you know your big data numbers?
1,000 bytes = one kilobyte (kB)
1,000 kB = one megabyte (MB)
1,000 MB = one gigabyte (GB)
1,000 GB = one terabyte (TB)
1,000 TB = one petabyte (PB)
1,000 PB = one exabyte (EB)
1,000 EB = one zettabyte (ZB)
1,000 ZB = one yottabyte (YB)
(Figures are decimal, not binary)
Data is only as good as the intelligence we can glean from it, and that entails effective data analytics and a whole lot of computing power to cope with the exponential increase in volume.
But a recent Bain & Co report found that of 400 large companies those that had already adopted big data analytics "have gained a significant lead over the rest of the corporate world."
"Big data is not just historic business intelligence," says Mr Carr, "it's the addition of real-time data and the ability to mash together several data sets that makes it so valuable."
Practically, anyone who makes, grows and sells anything can use big data analytics to make their manufacturing and production processes more efficient and their marketing more targeted and cost-effective.
It is throwing up interesting findings in the fields of healthcare, scientific research, agriculture, logistics, urban design, energy, retailing, crime reduction, and business operations - several of which we'll be exploring over the coming weeks.
By analysing weather, soil, topography and GPS tractor data, farmers can increase crop yields
"It's a big deal for corporations, for society and for each individual," says Ralf Dreischmeier, head of The Boston Consulting Group's information technology practice.
Can we handle all this data?
Big data needs new skills, but the business and academic worlds are playing catch up. "The job of data scientist didn't exist five or 10 years ago," says Duncan Ross, director of data science at Teradata. "But where are they? There's a shortage."
And many businesses are only just waking up to the realisation that data is a valuable asset that they need to protect and exploit. "Banks only use a third of their available data because it often sits in databases that are hard to access," says Mr Dreischmeier.
"We need to find ways to make this data more easily accessible."
Businesses, governments and public bodies also need to keep sensitive data safe from hackers, spies and natural disasters - an increasingly tall order in this mobile, networked world.
Who owns it all?
That's the billion dollar question. A lot depends on the service provider hosting the data, the global jurisdiction it is stored in, and how it was generated. It is a legal minefield.
"It's a big deal for corporations, for society and for each individual," says Ralf Dreischmeier, head of The Boston Consulting Group's information technology practice.
Can we handle all this data?
Big data needs new skills, but the business and academic worlds are playing catch up. "The job of data scientist didn't exist five or 10 years ago," says Duncan Ross, director of data science at Teradata. "But where are they? There's a shortage."
And many businesses are only just waking up to the realisation that data is a valuable asset that they need to protect and exploit. "Banks only use a third of their available data because it often sits in databases that are hard to access," says Mr Dreischmeier.
"We need to find ways to make this data more easily accessible."
Businesses, governments and public bodies also need to keep sensitive data safe from hackers, spies and natural disasters - an increasingly tall order in this mobile, networked world.
Who owns it all?
That's the billion dollar question. A lot depends on the service provider hosting the data, the global jurisdiction it is stored in, and how it was generated. It is a legal minefield.
Facebook's logo - created using photos of its global users - adorns the wall of a new data centre in Sweden - its first outside the US. But who has rights to all the data?
Does telephone call metadata - the location, time, and duration of calls rather than their conversational content - belong to the caller, the phone network or any government spying agency that happens to be listening in?
When our cars become networked up, will it be the drivers, owners or manufacturers who own the data they generate?
Social media platforms will often say that their users own their own content, but then lay claim to how that content is used, reserving the right to share it with third parties. So when you tweet you effectively give up any control over how that tweet is used in future, even though Twitter terms and conditions say: "What's yours is yours."
Privacy and intellectual property laws have not kept up with the pace of technological change.
Does telephone call metadata - the location, time, and duration of calls rather than their conversational content - belong to the caller, the phone network or any government spying agency that happens to be listening in?
When our cars become networked up, will it be the drivers, owners or manufacturers who own the data they generate?
Social media platforms will often say that their users own their own content, but then lay claim to how that content is used, reserving the right to share it with third parties. So when you tweet you effectively give up any control over how that tweet is used in future, even though Twitter terms and conditions say: "What's yours is yours."
Privacy and intellectual property laws have not kept up with the pace of technological change.
No comments:
Post a Comment