Explorers: A Profession in Decline?
Imagine living in a time when this map was an accurate representation of the accepted world view. A time when both the civilized and uncharted worlds coexisted in the collective mind and the title of “Explorer” accurately described those who ventured to beyond the horizons.
Today, while our internet connection is the biggest obstacle to accessing any corner of the globe, there are still unmapped unknowns waiting for intrepid “Explorers.” If you yearn for the days when venturing beyond the known world was still in vogue, you’re in luck! Big Data’s ecosystem is primed for exploration with discoveries on-par with anything Magellan or Darwin ever stumbled upon. The landscape in this case is, of course, the data-scape with a virtual horizon spanning from the micro to macroscopic.
Tiny Dancers, Big Stage
In 1674, a Dutch textile merchant discovered a new-world after building the first decent microscope in an effort to assess linen quality. Among the many observances this new tool enabled, what fascinated Antoine van Leeuwenkoek were the simple bacterium he could see and the speed at which they can reproduce.
“These creatures flit about so quickly. Wait a minute…”
“Good God, MAN! THEY’RE EVERYWHERE!”
“I’ve gotta tell someone!”
Eventually, Antoine van Leeuwenkoek spoke with the right people, the gateway into the microworld was opened, and the first lesson in Big Data was learned. Scale. One bacterium doubling every 4-20 minutes turns into millions within five hours. His discoveries ultimately paved the way to the Golden Age of Antibiotics. Similarly, Einstein was the first to explore the universe from the sub-atomic universe to black holes. Fast forward 60 years and scientists are using his observations to develop technologies that take advantage of the “spooky action at a distance”. In each of these, modern instruments of the day were devised to take into account “scale” as a means to both define boundaries and expand horizons. Scale ranges from the fraction to the infinite. Scale depends on what you want to know so let’s bring it into view!
Are Great Datasets Born Or Made?
Like most (data) engineering answers, “It depends”! Bacterium, atoms, and stars come in extremely large numbers, but so do sales transactions, Twitter comments, and commodity sensor feeds. The nebula of modern global life cradled the conceptualization of Big Data. Up until the end of the last century, we lacked the technology to meet Big Data’s three requirements…the Big “V’s”…Volume, Velocity, and Variety. The moment our technology (commodity computing services, advances in Data Science) caught up, we harmonized data’s Big V’s, fed it into our systems, and “MAN, THEY’RE EVERYWHERE!” a new horizon was discovered. These tools made it possible to connect everyone to everything. Much like the neural biological network in “Avatar”, there isn’t a place on earth that is beyond its reach.
Big Data’s nervous system possesses a robust ecology. Unlike the often hostile environments found in microbiology and quantum physics, this ecosystem can be tailored to a nurturing environment where data co-exists regardless of format, size, or owner. This harmony of programming and policy, produces interesting interactions such as:
- The accurate prediction of regional rate of market inflation, three months before governments have issued the official data that would bring them to the same conclusions.
- Smart phone app data used to alert psychiatrists days before their patients are aware of an onset of depression, manic episodes, or schizophrenic relapses.
- The locations and spread of insect infestations using overhead data collections at extremely high spatial, temporal, and spectral resolutions that also enable arborists to catalog and assess other biologic systems in near-real times.
- Boston Children’s Hospital can predict the location and magnitude of an impending influenza outbreak up to two weeks in advance based on social media tipping, cuing, nodal analysis and sentiment analysis in real time!
You Had Me at “Hello World”
Each of those examples required tools that can provide the viewer observations that span any data in time and space; where Volume, Velocity, and Variety are the veritable petri dishes that allow data ecologies to grow and Advanced Analytics the finely-tuned lenses and light sources to view the data. I dub this tool, the “Macroscope.” Anyone can use it, like your first microscope where you looked at pond water and saw that first single-celled organism.
But think back to the vastness of the world you (like Leeuwenkoek) just discovered and how it would take other visionaries, scientists and institutions to connect it all. Big Data’s enlightenment isn’t unique to one industry; it’s applications will find its way into every industry as professionals from psychiatrist to dairy farmers and visionaries from graphic designers to data scientists continue to observe its ecosystem. The one smart move left for you to do is to get yourself a Macroscope, and point it at a problem that’s both too big and too small to see.