Data Science Practical : Practical 5

Data Pre-processing and text analytics using Orange

What is text analytics?

Text analysis is a machine learning technique that allows us to automatically extract and classify text data such as tweets, emails, support tickets, product reviews, and survey responses. It is often used to describe, interpret and understand texts. It often aims to connect the text to a broader social, political, cultural or artistic context.

What is Sentiment Analysis?

Sentiment analysis is a machine learning tool that analysis texts for polarity from positive to negative. By training machine learning tools with examples of emotions in text, machines automatically learn how to detect sentiment without human input.

WHY?

Sentiment analysis is extremely useful in social media monitoring as it allows us to gain an overview of the wider public opinion behind certain topics. Sentiment analysis is useful for quickly gaining insights using large volumes of text.

Preprocessing is a key component in Data Science. Orange tool has various ways to achieve the activities.

Discretization:

It is the process of transferring continuous functions, models, variables, and equations into discrete counterparts. This process is usually carried out as a first step toward making them suitable for numerical evaluation and implementation data by the models.

Discretization replaces continuous features with the corresponding categorical features:

Sample Code:

import Orange

store = Orange.data.Table("Mart_Production.tab")

disc = Orange.preprocess.Discretize()

disc.method = Orange.preprocess.discretize.EqualFreq(n=3)

d_store = disc(store)

print("Original dataset:")

for e in store[:3]:

print(e)

print("Discretized dataset:")

for e in d_store[:3]:

print(e)

Continuization:

Given a data table, return a new table in which the discretize attributes are replaced with continuous or removed.

binary variables are transformed into 0.0/1.0 or -1.0/1.0 indicator variables, depending upon the argument zero_based.
multinomial variables are treated according to the argument multinomial_treatment.
discrete attribute with only one possible value are removed

Sample Code:

import Orange

products = Orange.data.Table("Products")

continuizer = Orange.preprocess.Continuize()

products1 = continuizer(titanic)

Normalization:

It is a systematic approach of decomposing tables to eliminate data redundancy(repetition) and undesirable characteristics like Insertion, Update and Deletion Anomalies.

Sample Code:

>>> from Orange.data import Table

>>> from Orange.preprocess import Normalize

>>> data = Table("Customers")

>>> normalizer = Normalize(norm_type=Normalize.NormalizeBySpan)

>>> normalized_data = normalizer(data)

Randomization:

A method based on chance alone by which study participants are assigned to a treatment group. Randomization minimizes the differences among groups by equally distributing people with particular characteristics among all the trial arms.

Sample Code:

>>> from Orange.data import Table

>>> from Orange.preprocess import Randomize

>>> data = Table("Returns")

>>> randomizer = Randomize(Randomize.RandomizeClasses)

>>> randomized_data = randomizer(data)

Dataset Description

The aim is to build a predictive model and find out the sales of each product at a particular store. Using this model, BigMart will try to understand the properties of products and stores which play a key role in increasing sales.

Total Rows:- 8523

Total Columns:- 13

Target Variable:- Item_Outlet_SalesEffect of Different kind of pre-processing on data

Work Flow for pre-processing