Mining For Gold In A Mountain Of Data

By

After shopping at your favorite grocery store week after week, you finally earned a free turkey.

The cashier scanned your loyalty card at every checkout, rewarding you with points toward a holiday turkey or ham – while at the same time sending an itemization of everything you bought to a database.

The grocery analyzes the “big data” it collects from you and other shoppers to uncover hidden patterns, correlations and other insights. The result is smarter business moves, more efficient operations, higher profits and the promise of happier customers.

Researchers estimate that more than 2.5 exabytes (that’s 2.5 billion gigabytes) of data are generated throughout the world every day. The use of loyalty cards, fitness trackers, web-based email, public services and social media – including every post, comment, like, photo and geotag – all contribute to this vast warehouse of information.

Big data involves not only collecting data, but storing, analyzing, searching, sharing, transferring, visualizing, querying and updating it. In fact, big data is so voluminous and complex that traditional ways of processing have proved inadequate. Hundreds or even thousands of computers running in parallel are needed for proper analyses.

To help address these computational bottlenecks, a team from the Industrial and Systems Engineering department at Lehigh University gathered with their colleagues at King Abdullah Univ. of Science and Technology in Saudi Arabia Feb. 5-7, 2018.

The KAUST Research Workshop on Optimization and Big Data brought researchers from across academia and industry to discuss big data optimization algorithms, theory, applications and systems.

Tamás Terlaky, the George N. and Soteria Kledaras ’87 Endowed Chair professor, was the keynote speaker at KAUST. Terlaky opened the workshop with his presentation, “60 Years of Interior Point Methods: From Periphery to Glory.”

Terlaky’s keynote focused on a technique pioneered in 1984 known as Interior Point Method (IPM). This novel methodology ignited far-reaching, intensive research toward discovering effective ways to solve large-scale optimization problems such as those found in big data analytics.

“Increasingly, we are getting different kinds of solutions in optimization,” Terlaky said. “Computation has become ubiquitous, and thanks also to the ‘Interior Point Revolution’ we have seen tremendous advances in computing.”

The concepts of IPMs and “machine learning” – where computers acquire the ability to learn and make decisions – were first proposed in the ’50s and were ahead of their time, Terlaky said. With computer technology still in its infancy, they failed to make any real impact. By the ’80s, however, the stars aligned to make the IPM revolution possible.

Now that we are in the era of big data, Terlaky said, recent advances in computer and information technology both enables and requires revolutionary advances in machine learning methodologies. “History always repeats itself,” Terlaky said. “You should learn from it.”

Leave a Reply

Your email address will not be published. Required fields are marked *