Big Data. Small Data. What is really the difference?
This is the age of Big Data. It surrounds us, like the clouds in the skies, seeming to be a solid mass. Yet, it is nothing but a haze, when we look inside from an airplane on our way home from vacation.
It is not tangible or clearly defined. However, at Innovation Lab, we have found a statement that tries to epitomize the concept. Big Data is the difference between what we want to do with data, and what we can do with data.
This is an age-old problem. Since the dawn of day, people have struggled with the compiling and structuring of information – and to turn that into decisions regarding future business strategies. In fact, this was why International Business Machines came to build computers. Originally, IBM produced typewriters, but a lot of the information typed in by staff in e.g. banks was compiled in archives, from where searches would take ages to perform. That was Big Data back then. In other words, the requirement to find files quickly, to provide information for decision making, and the inability to do so, made them create the computer, which we use today. In fact, the reason why files on a computer are called files, was because originally, it was physical files. The digital files were stored in a database. A file storage, from where the files could be instantly extracted and written to printers on demand.
Today, the usage of data to drive decision-making is a must-have for bigger businesses. However, these still struggle with data. Furthermore, as the amount of data increases exponentially, our ability to interact with data does not follow.
Youtube can show videos, but cannot decode the content, narrative, and meaning of a guy eating chili or a girl doing makeup. Images post a big problem too. Even Google and Facebook, for all their clear minds and unlimited resources, cannot figure out how to make real sense of an image…and the list goes on…
Small data is equally puzzling for decision makers. Usually, small data is a product of a small business or a business that is not traditionally data driven. We don’t see many mechanics do analysis on the number of bolts and joints, used for different vehicle types, over a year in order to optimize the stock of spare parts. Also, a flower shop owner will have a reasonable sense of season, flower species, and quantities to acquire from their supplier, but deep analysis of the exact optimal mix over time to optimize revenue is an uncommon practice. However, the challenge is the same. But the difference lies in the structuring, collection, and analysis, whereas the big businesses challenge lies in the complexity of data.
In Denmark, as in most – if not all – countries, the major part of the GDP is provided by small and medium businesses. Therefore, an added benefit to the GDP of an increased use of data for decision-making must come from that particular segment of businesses. Alas, enter the toolbox, without the tools.
SMBs do not have the tools and competencies needed to work with data. Business analysis is for professionals, like yours truly. It is expensive and not easily accessible. That person, who develops a tool to ease the SMBs in their use of data, and ease their decision-making, without having a statistical or financial background, will make a lot of money. However, we still need to see someone take up the challenge.
In Big Data, the data is too complex to analyze, are too large to understand, or is moving too fast to make sense of. With Small data, the collecting and analysis of data are the main problems. Both problem statements require solutions, and both are equally difficult.
If you want to know more about Big and Small Data, reach out to me at bigdata@ilab.dk, and let’s have a talk! I have also defined Quantum Computing, if you are looking for even more practical takes on complex concepts.