I started learning data science about a years ago.. This is mostly geared towards people who are in the same position I was in.

A lot of advice around learning data science starts with "first learn python", or "first take a linear algebra course". This advice is fine, but if I followed it, I never would have learned any data science. Being data scientist requires a solid foundation typically in computer science and applications, modelling, statistics, analytics and math.

What sets the data scientist apart is strong business acumen, coupled with the ability to communicate findings to both business and IT leaders in a way that can influence how an organization approaches a business challenge. Good data scientists will not just address business problems, they will pick the right problems that have the most value to the organization.

Here is a list of books on doing machine learning / data science in R and Python which I’ve come across in last one year. Since, reading helps to keep close to the topic it will also works and reference guide.

Disclosure: The amazon links in this article are affiliate links. If you buy a book through this link, we would get paid through Amazon. This is one of the ways for us to cover our costs while we continue to create these awesome articles. Further, the list reflects our recommendation based on content of book and is no way influenced by the commission.

Python and R

1) Python for Data Analysis

I was new to Python and to Data Analysis when I started working through this book. I found the chapters on numpy particularly useful: what was also helpful was to have everything on one place, rather than having to scratch around for it online.

The book covers the basics of Python, as well as IPython, Numpy and Pandas. I still use it now as a reference. If you’re well versed in python and data analysis, it’s probably worth the purchase price; but if you’re new to it all, I would definitely recommend it.

2)Think Python

However, Think Python is a book I'd recommend over again and again to anyone who seeks a gentle introduction to the good parts of the Python language.

The book, as I've found is often recommended by professionals everywhere - for example, right on Quora, stack overflow. Personally, it's one of those few books I've managed to go through cover-to-cover .

3)R for Data Science: Import, Tidy, Transform, Visualize, and Model Data

If you've used `R` in the past but mainly use base functions then this will be a great refresher for you. If you're new to the world of `R` then this book will give you a solid foundation of how to get started. It is a collection of R packages designed to work together to make data science fast and fluent.

Statistical and Math for data analysis

1) Think stat (2nd edition)

This book covers more aspect of statistics required to get your hands dirty by learning to do practical work. This book works completely fine for beginner as well .

2) Introduction to Probability

An intuitive, yet precise introduction to probability theory, stochastic processes, statistical inference, and probabilistic models used in science, engineering, economics, and related fields. This is the currently used textbook for "Probabilistic Systems Analysis," an introductory probability course at the Massachusetts Institute of Technology, attended by a large number of undergraduate and graduate students.

3) An Introduction to Statistical Learning with Applications in R

“An Introduction to Statistical Learning With Applications in R” gives you an overview of analyzing, organizing, and leveraging data using the powerful and popular R programming language. Written by Gareth James, professor of data sciences at USC; Daniela Witten, professor of biostatistics at University of Washington; and Robert Tibshirani and Trevor Hastie, professors of statistics at Stanford, it is ideal for both statisticians and non-technical professionals who are looking to understand data management, analysis, and presentation techniques.

Big data

Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie, or Die

In “Predictive Analytics“, Eric Siegel, a renowned expert in data analytics and former professor at Columbia University, explains how scientists use big data to help predict, well, anything – from what you will buy, to where you will travel, to when you will quit your job, and more. The Seattle Post-Intelligencer called the book “mesmerizing,” and also praised its relevance to multiple business departments.

Hadoop, the Definitive Guide

Apache Hadoop is a framework used to process large amounts of data. Tom White is an expert Hadoop consultant, trainer, and member of the Apache Software Foundation. His guide, “Hadoop: The Definitive Guide: Storage and Analysis at Internet Scale,” will help you understand how to build and manage scalable systems using Hadoop. It’s a good reference for programmers, and for IT managers tasked with running Hadoop clusters.

Big Data: A Revolution That Will Transform How We Live, Work, and Think

This book is written by Kenneth Cukier and Viktor Mayer Schonberger. This book takes you on a world tour of values added by big data across all industries. This book will help you to stay ahead of the key trends defining businesses in coming years. Jeff Jonas, Chief Scientist, IBM Entity Analytics said, ‘The book teems with great insights on the new ways of harnessing information, and offers a convincing vision of the future. It is essential reading for anyone who uses — or is affected by — big data.’

We will keep on updating the list with few more resource and books .Feel free to share your views and suggestion which will be helpful to refer to become a better data scientist.

8 Effective plots with Matplotlib and Pandas Dataframe

In the previous post we learned some matplotlib plotting techniques.This is second part of matplotlib where we are going to work with some random dataset.This post will also cover basic different type of plotting you can produce in matplotlib.This type of plotting are mostly used to understand the type of data and produce useful insights.Learning and understanding matplotlib will take some longer learning time and some patience.While we are trying to learn by creating sample random dataset in dataframe and visualizaling different kind of plots.
I have come to appreciate matplotlib because it is extremely powerful. The library allows you to create almost any visualization you could imagine. Additionally, there is a rich ecosystem of python tools built around it and many of the more advanced visualization tools use matplotlib as the base library.

In [25]:

%matplotlib inline
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('ggplot')

The plot method on Series and DataFrame is just a simple wrapper around plt.plot()
If the index consists of dates, it calls gcf().autofmt_xdate() to try to format the x-axis nicely as show in the plot window.

In [26]:

ts = pd.Series(np.random.randn(1000), index=pd.date_range('1/1/2000', periods=1000))
ts.head(5)

Out[26]:

2000-01-01   -0.383130
2000-01-02    1.122075
2000-01-03    0.264544
2000-01-04    0.205977
2000-01-05    0.652250
Freq: D, dtype: float64

This dataframe consist of data containing random date with random 1000 data points.I have clearly shown the top 5 data by calling head(5) of the dataset.Now, We are going to plot on cumulative sum based on years. This has been shown in below Line plot visualization.

In [27]:

ts = ts.cumsum()
ts.plot()
plt.show()

On DataFrame, plot() is a convenience to plot all of the columns, and include a legend within the plot.

In the next example we are going to plot multiple plots which will give fair idea about data.This dataframe also contains data for time with random values in 4 marked cloumns A,B,C,D. We are going to see the trends on value in the random generated data.

In [28]:

df = pd.DataFrame(np.random.randn(1000, 4), index=pd.date_range('1/1/2016', periods=1000), columns=list('ABCD'))

df.head(5)

Out[28]:

	A	B	C	D
2016-01-01	0.734441	-0.967202	1.941327	-0.848996
2016-01-02	1.702695	0.071849	0.668847	-0.751232
2016-01-03	-2.273635	0.385259	-1.347990	0.087448
2016-01-04	0.202025	-1.137845	-0.893557	-0.744962
2016-01-05	-0.094856	-0.090228	0.843362	0.447179

In [29]:

df = df.cumsum()
plt.figure()
df.plot()
plt.show()

<matplotlib.figure.Figure at 0xa34bb70>

Futher we can ses in the above visuallixation which shows a trends using line plot.We are generating cumulative sum for all the columns seperately and plotting it.Based on deomanstrated visualization A columns has produced higher values than others.

In the demonstration ,You can plot one column versus another using the x and y keywords in plot():

In [30]:

df3 = pd.DataFrame(np.random.randn(1000, 2), columns=['B', 'C']).cumsum()
df3['A'] = pd.Series(list(range(len(df))))
df3.plot(x='A', y='B')
plt.show()

In [31]:

df3.tail()

Out[31]:

	B	C	A
995	38.306553	55.991689	995
996	38.052156	55.195136	996
997	38.501586	53.555756	997
998	37.491278	51.671606	998
999	38.449854	50.618397	999

Plots other than line plots¶

Plotting methods allow for a handful of plot styles other than the default Line plot. These methods can be provided as the kind keyword argument to plot(). These include:

‘bar’ or ‘barh’ for bar plots
‘hist’ for histogram
‘box’ for boxplot
‘kde’ or 'density' for density plots
‘area’ for area plots
‘scatter’ for scatter plots
‘hexbin’ for hexagonal bin plots
‘pie’ for pie plots

For example, a bar plot can be created the following way:.we are going to use the same dataset used above to plot the line plot. I have shown the top 5 column from the dataframe

In [32]:

df.head(6)

Out[32]:

	A	B	C	D
2016-01-01	0.734441	-0.967202	1.941327	-0.848996
2016-01-02	2.437136	-0.895354	2.610174	-1.600229
2016-01-03	0.163501	-0.510094	1.262183	-1.512781
2016-01-04	0.365526	-1.647939	0.368626	-2.257742
2016-01-05	0.270669	-1.738168	1.211989	-1.810563
2016-01-06	1.906568	-2.422695	2.048194	-2.288180

To plot a bar plot we are fetching index for date 2016-01-06 00:00:00 from dataset and plotting based on the values.
.ix is the most general indexer and will support any of the inputs in .loc and .iloc. .ix also supports floating point label schemes. .ix is exceptionally useful when dealing with mixed positional and label based hierachical indexes.

In [33]:

plt.figure()
df.ix[5].plot(kind='bar')
plt.axhline(0, color='k')
plt.show()

In [34]:

df.ix[5]

Out[34]:

A    1.906568
B   -2.422695
C    2.048194
D   -2.288180
Name: 2016-01-06 00:00:00, dtype: float64

stack bar chart¶

Ahhh!.. we are moving to next building a stcked bar chart. we will be creating a small dataset for this demonstration.we will call bar() with stacked=true to plot vertical a stacked bar and we are going to plot a horizontal plot in the next example.

In [35]:

df2 = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df2.plot.bar(stacked=True)
plt.show()

horizontal bar chart¶

In [36]:

df2.plot.barh(stacked=True)
plt.show()

Box plot¶

Make a box plot from DataFrame column optionally grouped by some columns or other inputs

In [37]:

df = pd.DataFrame(np.random.rand(10, 5), columns=['A', 'B', 'C', 'D', 'E'])
df.plot.box()
plt.show()

area plot¶

In a stacked area plot, the values on the y axis are accumulated at each x position and the area between the resulting values is then filled.

In [38]:

df = pd.DataFrame(np.random.rand(10, 4), columns=['a', 'b', 'c', 'd'])
df.plot.area()
plt.show()

Plotting with Missing Data¶

Pandas tries to be pragmatic about plotting DataFrames or Series that contain missing data. Missing values are dropped, left out, or filled depending on the plot type.

Plot Type	NaN Handling
Line	Leave gaps at NaNs
Line (stacked)	Fill 0’s
Bar	Fill 0’s
Scatter	Drop NaNs
Histogram	Drop NaNs (column-wise)
Box	Drop NaNs (column-wise)
Area	Fill 0’s
KDE	Drop NaNs (column-wise)
Hexbin	Drop NaNs
Pie	Fill 0’s

If any of these defaults are not what you want, or if you want to be explicit about how missing values are handled, consider using fillna() or dropna() before plotting.

density plot¶

In [39]:

ser = pd.Series(np.random.randn(1000))
ser.plot.kde()
plt.show()

lag plot¶

Lag plots are used to check if a data set or time series is random. Random data should not exhibit any structure in the lag plot. Non-random structure implies that the underlying data are not random.

In [40]:

from pandas.tools.plotting import lag_plot
plt.figure()
data = pd.Series(0.1 * np.random.rand(1000) + 0.9 * np.sin(np.linspace(-99 * np.pi, 99 * np.pi, num=1000)))
lag_plot(data)
plt.show()

PIE Chart¶

In [41]:

# Data to plot
labels = 'Python', 'C++', 'Ruby', 'Java'
sizes = [215, 130, 245, 210]
colors = ['gold', 'yellowgreen', 'lightcoral', 'lightskyblue']
explode = (0.1, 0, 0, 0)  # explode 1st slice
 
# Plot
plt.pie(sizes, explode=explode, labels=labels, colors=colors,
        autopct='%1.1f%%', shadow=True, startangle=140)
 
plt.axis('equal')
plt.show()