I didn’t go to many games, but I remember flipping through the program at a game once. There was a page where fans could score the game, marking each play as it occurred and helping us to recall afterward the reasons we had hope for the next game.

Baseball, a Statistician’s Game

Baseball aficionados have long kept detailed records about games played. Statistics exist going all the way back to 1871. Want to know which batter led the league in Strike Outs in 1929? You can. It was the appropriately named Hack Wilson with 83. Who had the most Doubles in a single season? That’s Earl Webb in 1931 with 67. What pitcher has had the worst ERA with at least 5 games played? Phillies fans may remember Patrick Schuster’s 2016 performance when he had a whopping 45 ERA in 6 games. 

With all this data readily available, Major League Baseball gives IT Professionals a fertile playing field for learning analytics and statistics.

Learning Data Science with Baseball

Want to learn statistical analysis with R or Python? Or perhaps you would like to explore PowerBI? MLB gives us plenty of data with which to play and learn. Want to learn how to join tables and write subqueries? Again MLB has a rich dataset for you. 

Here’s a source of data that I’ve used for testing and for creating presentations when I speak at conferences and SQLSaturdays: SeanLahman.com. The site provides free downloads of MLB data going back to 1871. It’s available in CSV, Microsoft, Access, and SQL Server files.

Download the data and begin experimenting today. And if you want to develop a deeper sense of empathy, just look at the stats for the 1978 Braves, who had a record of 69-93 and gave up 150 more runs than they scored. Now that’s character building.

Looking for some humorous examples of data analysis gone wrong? Check out Beware of Spurious Correlations when Analyzing Your “Big Data.”

The post Data! Getcha Data Here! appeared first on The SERO Group.

These terms, and many like them, have been in the news a lot recently. And with good reason. Many organizations are taking their first tentative steps toward sifting through the vast amounts of data collected in disparate systems in search of hidden nuggets of insight to improve operations or some other aspect of business. Rushing headlong into a statistical analysis of their systems, they hope to find the next big thing.

Statistics for Statistic’s Sake?

But statistics for statistic’s sake is not very useful. A deep analysis of data is most interesting and useful when the results can be used to predict future outcomes based on prior correlations with a relatively high degree of certainty.

Baseball, for example, is an industry rife with relatively meaningless statistics. The traditional metrics of batting average and earned run average have proven to add relatively little value when evaluating a players contribution to a team’s chance of winning. Yet they have been given significant weight both on the field and in the back office. These statistics have been used to negotiate contracts, determine batting order, and influence the starting pitching rotation.

True Key Performance Indicators

Over the past ten years or so, a ton of other statistics have been identified and found to be far more predictive of successful outcomes. For example, on-base percentage and the ability to draw walks are more indicative of the positive offensive contributions of a player. From a pitching perspective, the ability to get a batter to hit a ground ball translates into more outs and hence more team wins than the traditional ERA metric. If you are interested in sports and data analytics, I would highly recommend the following two books: Moneyball: The Art of Winning an Unfair Game by Michael Lewis and Big Data Baseball by Travis Sawchik. Both are fascinating and entertaining reads, especially for the data-minded sports fan.

Like Major League Baseball, many organizations are combing through copious amounts of data looking for ways to improve operations, sales, manufacturing, etc. As an example, Target can identify pregnant patrons based on subtle changes in their buying habits, often before they have shared their good news with relatives. Disney has used RFID in its theme parks to track queue length, visitor travel patterns, etc, so they can adjust and improve operations.

Analyzing Your “Big Data”

As your organization begins to evaluate using data analytics to unearth hidden correlations that may influence your strategic initiatives, remember that to be truly relevant, statistics alone cannot tell the whole story. Subject matter experts from operations, HR, engineering, manufacturing, and other areas within the business should be involved with the project. They can offer great insight and help you to ask better questions. Without their expertise, meaningless correlations can be identified and inadvertently given credence.

Some Spurious Examples

Some humorously extreme examples of these spurious correlations can be found at Tylervegin.com. For example, the divorce rate in Maine has an uncanny correlation to the amount of margarine consumed per capita annually. Spreading margarine on a biscuit causes a nasty breakup in Maine? Of course not, but the statistics imply otherwise.

Chart courtesy of TylerVigen.com.

Likewise, the number of drivers killed annually in collisions with trains closely parallels the amount of crude oil imported into the US from Norway. If the U.S. wants to see the death toll drop to zero, lawmakers should ban importing crude oil from Norway? That’s preposterous. Well, to most of us, anyway. I don’t want to predict what Congress may do.

Chart courtesy of TylerVigen.com.

Obviously, these correlations do not imply causation. And, that’s the point. Without the right subject matter experts in the room, an analytics team may draw incorrect inferences and lead the organization on a wild goose chase.

Asking the Right Questions

Asking the right questions is key. And who better to ask great questions than Subject Matter Experts, end users, etc., when analyzing your “Big Data?”

Looking for some data to experiment with? Check out Data! Getcha Data Here!

Are you inviting the right people to your project’s table?

The post Beware of Spurious Correlations when Analyzing Your “Big Data” appeared first on The SERO Group.