Big Data and the Internet of Things

 

By Melanie Jutras

04/15/2016

It seems hard to believe, but Big Data is about to become even bigger. This is, in part, due to the increased data being produced by the Internet of Things (IoT). The IoT could potentially involve as many as 50 billion connected devices by the year 2020.(1) Consider the staggering amounts of data that might be produced by this many devices. How can we access this data, how can we analyze it and how can we put it to good use? Dr. Kirk Borne, one of the most knowledgeable Data Science Speakers in the world will be speaking at ODSC East regarding his views on open data and how we can put it to good use.

Learn more about this topic and others just like it at one of our upcoming Open Data Science Workshops, trainings and tutorials.

Although Dr. Borne is an expert in the field of Data Science, he also feels strongly about the importance of people from all professions becoming data literate.(2) Everybody needs to understand what Big Data is and how we can use it.  Now at the advent of a data surge due to the Internet of Things data literacy and user friendly tools will become increasingly important.

When we talk about the Internet of Things, people tend to think of personal devices and gadgets used by individuals. While gadgets such as wearable technology and smart home monitoring devices are part of the picture, they barely scratch the surface. The IoT is essentially made up of sensors that can be placed anywhere in order to collect data.  Sensors will collect data in industries such as agriculture, automotive and retail.

Among other things, they will be used for security, monitoring and automation. If you stop to think about how much data might be produced from 50 billion connected devices, you will soon realize that the Internet of Things is really not so much about the devices that are connected, rather it is about the enormous data stream that these “things” will produce.

A recent blog post by Dr. Vincent Granville, highlights a number of sensor data set repositories.(3) Data sets that have been collected across many sectors including energy, healthcare, weather and transportation are available for viewing and analysis. One of these data sets published by Microsoft Research provides 15 million data points related to sensor data collected from taxi cabs in order to research driving directions.(4)

More information on this data can be found in the research paper, T-Drive: Driving Directions based on Taxi Traces.(5) Another sensor data set that is provided involves 160 million observations recorded by 20 thousand weather stations published on datahub by the Linking Open Data Cloud organization.(6) These are just a couple of examples of data that is being collected every day and available for analysis.

While it is powerful to have the ability to gather millions of pieces of sensor data, the next obvious problem is dealing with the data.  How does one go about managing and analyzing such a large data set? There are various products available for analysis and visualization of enterprise data.  A list of some of these can be found in a recent Data Science Central blog post, Eight IoT Analytics Products.(7)

Popular commercial products such as Dell Statistica and IBM IoT Platform are highlighted. These are valuable for professionals in many different fields who may not be data science researchers, yet they need to deal with Big Data. Another one of the analytics products highlighted is Intel IoT Analytics Platform.

Their IoT cloud analytics site is provided as a service to the IoT development community. Intel’s Internet of Things group has played a key role in an ongoing project building an open cloud-based platform to accelerate cancer research.(8) This is just one example of using data for social good. We all ought to be thinking about what types of questions can be answered from Big Data and how we can put it to good use. With the amount of expected data available from connected devices, the possibilities for analysis are endless.

Find this blog useful?

Help others read it by commenting and sharing.

1.     Zdnet.com/article/the-internet-of-things-and-big-data-unlocking-the-power/
2.     Searchdatamanagement.techtarget.com/feature/Kirk-Borne-on-data-science-and-big-data-analytics-data-literacy
3.     Datasciencecentral.com/profiles/blogs/great-sensor-datasets-to-prepare-your-next-career-move-in-iot-int
4.     Research.microsoft.com/apps/pubs/?id=152883
5.     Research.microsoft.com/en-us/projects/tdrive/
6.     Atahub.io/dataset/knoesis-linked-sensor-data
7.     Datasciencecentral.com/profiles/blogs/eight-iot-analytics-products
8.     Eweek.com/cloud/intel-unveils-analytics-technologies-for-big-data-iot.html

Jupyter Developer Meetings

By: Gordon Fleetwood – ODSC data science team contributor

The IPython/Jupyter notebook is a staple of the Data Scientist's toolbox due its great visual and practical functionality. It turns out the minds behind this incredible platform have regular meetings which are available to watch on YouTube.

It's a fascinating behind-the-scenes look at what cool additions the developers are working on. Some of these include the nascent  Apache Toree project, splitting one notebook into two with the click of a button, easily injecting code from one notebook into another, adding to-do lists,and the exciting option to turn Jupyter notebooks into a dashboard or a web app.

This last idea could potentially be a game changer, especially in the Python ecosystem. Unlike R which has Shiny as a native application to build data-centric web apps, Python users have to adapt other tools.

Two of these are the web frameworks Django and Flask, and the more Shiny-esque DataSpyre and Pxyley, neither of which seem to have caught on that much. If Jupyter notebooks could become web apps with the click of a button, it could be the start of Shiny finding a worthy rival in the space.

While injecting code from another notebook may not be as superficially attractive as web apps, it would add important flexibility to a Data Scientist’s workflow. The ultimate goal seems to be enabling a person to import another notebook as easily as one would import a package.

Such functionality would allow different parts of analyses to be linked together seamlessly and allow for greater modularity.

Learn more about this topic and others like it at our Open Data Science Workshops, training sessions and conferences.

Find this blog useful?

Help others find it by commenting and sharing.

Great Data Science Books

By: Gordon Fleetwood – ODSC data science team contributor

Writing any book is a momentous task. At ODSC, we have been fortunate to have speakers at our Big Data Science Conference who have completed this task, and added to the rich library of Data Science literature. Here are a few of these books.

Applied Predictive Modeling, Max Kuhn

Max Kuhn is a superstar in the R world known for this creation of the caret library, an all-purpose package which is R's equivalent of scikit-learn. Applied Predictive Modeling is a tome dedicated to every aspect of the model building process, from data pre-processing and feature engineering to model selection. All of the theory is accompanied by R code snippets showing the practical applications of concept through caret.

Think Bayes, Allen Downey

Bayesian Statistics is at the heart of Data Science, and Think Bayes is a superb way to from Bayesian apprentice to Bayesian master. The thoroughness and clarity that Mr. Downey has brought to his numerous talks over the years is on display in written form.

Python for Data Analysis, Wes McKinney

Once there were only R dataframes, and Python was left in the dark with only the csv module for company. Then came Wes McKinney, and so dataframes came to Python through the pandas library. Python for Data Analysis is written around pandas and the various operations it supports for use on the dataframes it produces. Mr. McKinney goes through data cleaning, visualization, aggregation all the way to time series. It's pretty comprehensive, to say the least.

R for Everyone: Advanced Analytics and Graphics, Jared Lander

Jared Lander is another name that R users will be instantly familiar with. R For Everyone lives up to its name by starting from the very basics of the language, and working its way up to advanced usage like running statistical tests, building models, and making R packages. The existence of a companion video series makes is the icing on the cake that is a comprehensive look at the one of Data Science's most popular languages.

ODSC conferences give attendees the opportunity to speak with and have books signed by authors. Don’t miss out on this opportunity at ODSC East!

Find this blog useful?

Help others find it by commenting and sharing.

Cool Data Science GitHub Repos

By: Gordon Fleetwood – ODSC data science team contributor

We’ll be addressing this topic and others just like it at our upcoming Big Data Science Conference, ODSC East. 100+ speakers, 30+ workshops and 15+ training sessions all under one roof.

GitHub is the most popular central storage space for open-source projects, a good portion of which are now Data Science related with the field's rapid rise over recent years. Here are a couple cool Data Science repos to check out.

TPOT is the brainchild of Randal Olson, a post-doc researcher at the University of Pennsylvania. It uses genetic algorithms to automate dealing with features, selecting models, and optimizing hyperparameters to find the best pipeline to use on your data. It's built on top of sci-kit learn, so the best model pipeline is delivered in a familiar package.

There are a couple of Machine Learning engines out there in the shadows of the most popular frameworks like sci-kit learn or caret. One of these is Leaf, a Rust project by the startup Autumn. It may be a language that is not a traditional Data Science staple, but the claims made on the readme about its foundations, flexibility, and operational capacity are intriguing enough to keep a watch on its progress.

Some of the more popular Python libraries for topic modeling are gensim, blob, pattern, and nltk. Into this crowded field comes topik, a topic modeling package from Continuum Analytics, the same people who brought Data Science the Anaconda package distribution. However, topik isn't coming to compete with these packages. It is built upon them, and its goal is to provide a high level interface for users. It's a great way for users to do topic modeling out of the box rather than going through the lego assembling process that the usual frameworks require.

This R package comes from Etsy's Hilary Parker, and fits quite smoothly into the nominative determinism slot. The idea is simple. It serves as a wrapper around R objects to turn its results into full length explanations of what these numbers mean. It's a great tool for those just getting into Data Science, and would probably be a nice reference for experienced practitioners as well.

Pivot tables are a mainstay of the Business Intelligence world where Excel and Tableau rule the roost. There are interfaces for pivot tables in Python and R - pandas' implementation for one - but the interface is a step below that of either Excel or Tableau in terms of ease of use. pivottablejs takes care of this gap beautifully for users of Python and the Jupyter notebook. Once you give the package a dataframe you'll get an interactive interface to create a pivot table within the notebook. All you have to do is drag and drop.

What's going in deep neural networks? This highly effective application of Machine Learning is famously opaque, and efforts to make it less so are ongoing. One such attempt is tdb, a visual debugger for deep learning. It's built on top of TensorFlow and tries to give the user a sense of how data is flowing through the network by using data visualizations.

Quantopian's qgrid gives you extra flexibility in how you use your dataframes. Instead of filtering or sorting values using code, qgrid allows one to do all of this by pointing and clicking. It evens allows for interactive changing of values.

En garde, Monsieur! As unit testing is to software engineering, so is data validation to Data Science. Engarde is a libray that aids in the validation process to make sure that your assumptions don't become headaches later on.

This R package seeks to standardize the setup for analysis projects. At the outset it generates a complete directory subdivided by folders and helper files for various operations which pop up in a project. Some of these include connecting to a database, data validation, loading data into memory, and even logging. The obvious benefit is the strength such a framework lends to reproducibility of analyses.

These are just some of the interesting Data Science works on display through GitHub. We'll keep looking and post another set of notable projects soon.

Find this blog useful?

Help others find it by commenting and sharing.

Influencing Data Visualization Forever

By: Gordon Fleetwood – ODSC data science team contributor

Data Viz, Big Data and other related topics will be discussed at our Data Science Workshops, training sessions and conferences. Take a look at some of Data Science’s most influential players.

Santiago Giraldo works for CartoDB and does a lot of data visualization work with geospatial data. One of his projects addressed gerrymandering, the manipulation of the boundaries of electoral constituencies to favor a political party. Specifically, he looked at this phenomenon in New York, and how it could be tied to income inequality.

Larry Buchanan’s experience in visualization stretches back years before his current position at the New York Times, where he created the piece linked to above. The stop of every subway in New York City is shown, and the median income associated with the neighborhood surrounding each of them. It is both awesome and sobering to follow the rise and fall of each graph’s points as it goes through boroughs and neighborhoods.

The map shows the magnitude of people needing to take advantage of feeding programs in 2013 across the world. The percentage of these people suffering from kwashiokor, a disease characterized by a severe lack of protein, is highlighted.

Mr. Cherven is a Data Visualization Specialist at General Motors, and his talk at ODSC East will focus on the open-source tools available for visualizing complex networks. His work in this field is extensive and most of its public face focuses on baseball.

Visualizing scientific concepts is extremely important in communicating their key ideas. This is where Bang Wong, Creative Director of the Broad Institute of MIT and Harvard, and others like him come in. Vis Skunkworks is one of his initiatives which seeks to provide clarity for concepts in genomics through visualization.

The visualization highlighted here, however, is a network of voting patterns in the United States House of Representatives in the Fall of 2014. With a click of a button you can see which votes passed or failed, and who on either side of the aisle voted for it.

Find this article useful?

Help others see it by commenting and sharing.

The Pros and Cons of Deep Learning

 
We will be discussing Deep Learning and related topics are our Big Data Conferences, training sessions and workshops.

Deep learning is a collection of statistical machine learning techniques used to learn feature hierarchies often based on artificial neural networks. That’s it. Not so scary after all.

For sounding so innocuous under the hood, there’s a lot of rumble in the news about what might be done with DL in the future. Let’s start with an example of what has already been done to motivate why it is proving interesting to so many.

Deep learning has been all over the news lately. In a presentation I gave at Boston Data Festival 2013 and at a recent PyData Boston meetup I provided some history of the method and a sense of what it is being used for presently. This post aims to cover the first half of that presentation, focusing on the question of why we have been hearing so much about deep learning lately.

What does it do that couldn’t be done before? We’ll first talk a bit about Deep learning in the context of the 2013 kaggle-hosted quest to save the whales1 The game asks its players the following question: given a set of 2-second sound clips from buoys in the ocean, can you classify each sound clip as having a call from a North Atlantic right whale or not?

The practical application of the competition is that if we can detect where the whales are migrating by picking up their calls, we can route shipping traffic to avoid them, a positive both for effective shipping and whale preservation.

The content is aimed at data scientists who might have heard a little about deep learning and are interested in a bit more context. Regardless of your background, hopefully you will see how deep learning might be relevant for you. At the very least, you should be able to separate the signal from the noise as the media hype around deep learning increases.

Find this blog useful?

Help others read it by commenting and sharing.

7 Important Model Evaluation Error Metrics Everyone should know

 

Finding the right model for your prediction is important but highly dependent on the metric you use to assess the quality of the predictions and the predictive power of the model. In this article, Tavish Srivastava, decrypts 7 evaluation metrics: Definition, usage and their influence on the model selection. 

The metrics include the classics: RMSE, ROC-AUC and the confusion matrix, alternatives metrics such as the Gini Coefficient and Gain Chart which are frequently used in Kaggle competitions and less frequent ones such as the Kolmogorov Smirnov Chart or the Concordant – Discordant Ratio.

A very well illustrated article which offers a good introduction on the importance of the metric for model selection.

Classifying Bees With Google TensorFlow

 The Bees Classifier Metis Challenge on DataDriven.org consisted in predicting the type of bees appearing in a 4000 images. Given the set of images it was up to the participants to build their own set of features using image processing techniques.

In this article, Philippe Dagher Data Scientist and Kaggler, builds a basic Google Tensorflow algorithm to determine the genus—Apis (honey bee) or Bombus (bumble bee)—based on photographs of the insects. A good coding example on how to apply Google TensorFlow on a real life dataset.

Find the blog useful?

Help others see it by commenting and sharing.

Learn more about this topic and others like it at our Data Visualization Conference, trainings and workshops.

References:  

http://nasdag.github.io/blog/2016/01/19/classifying-bees-with-google-tensorflow
http://www.analyticsvidhya.com/blog/2016/02/7-important-model-evaluation-error-metrics