Practical 11

Dataset name :  diabetes_data_upload.csv

Task-1: Dataset Description using Orange tool.

Task-2: Generate the Dashboard of preprocessed dataset from task-1. Find the Maximum data insights by plotting using PowerBI dashboard visualization.

Code Reference 

Practical 10

AIM: Using image data, predict the gender and age range of an individual in Python. Test the data science model using your own image. 

Theory:

OpenCV is a cross-platform library using which we can develop real-time computer vision applications. It mainly focuses on image processing, video capture and analysis including features like face detection and object detection.

Additional Python Libraries Required :

OpenCV pip install opencv-python 

argparse pip install argparse 

The contents of this Project :

opencv_face_detector.pbtxt

opencv_face_detector_uint8.pb

age_deploy.prototxt

age_net.caffemodel

gender_deploy.prototxt

gender_net.caffemodel

A few pictures to try the project on detect.py

For face detection, we have a .pb file- this is a protobuf file (protocol buffer); it holds the graph definition and the trained weights of the model. We can use this to run the trained model. And while a .pb file holds the protobuf in binary format, one with the .pbtxt extension holds it in text format. These are TensorFlow files. For age and gender, the .prototxt files describe the network configuration and the .caffemodel file defines the internal states of the parameters of the layers.

Usage :

• Download my Repository

• Open your Command Prompt or Terminal and change directory to the folder where all the files are present

• Detecting Gender and Age of face in Image Use Command : python detect.py --image <image_name>

Note: The Image should be present in same folder where all the files are present

• Detecting Gender and Age of face through webcam Use Command : python detect.py

• Press Ctrl + C to stop the program execution.

Entering Input

Showing age

Practical 9

Aim : Execute queries in the Neo4j graph database and in Gephi tool perform the operations of loading csv data, running graph statistics scripts and displaying various graphical layouts.

Theory: 

What is graph data?

Graph data science is linked to data science, where connexions are first-class people, and a stronger, more detailed forecast can be made using the ties between the data points.

How graph data is stored? and how it is represented?

Graph data is contained in storage files, each of which includes data, such as nodes, associations, labels and properties, for a certain part of the graph. A data graph is a graphic display of knowledge from which we can more readily interpret similarities and patterns quickly.

How to visualize graph data and devise conclusion/result?

By looking at the graph correctly, we can imagine graph data and we can get the successful knowledge that shows mechanisms that might be present in the graphs and assist the user to understand that.

Neo4j is an open-source, NoSQL, native graph database that provides an ACID-compliant transactional backend for your applications. This means that the data is stored exactly as you whiteboard it, and the database uses pointers to navigate and traverse the graph.

Neo4j uses property graphs to extract added value of data of any company with great performance and in an agile, flexible and scalable way. 1. Performance: Graph databases such as Neo4j perform better than relational (SQL) and non-relational (NoSQL) databases.

Let's get started with Neo4j database!!

You can download Neo4j according to your system requirement from here. Although here I am going to use online Neo4j Sandbox for demo purpose.

1. First of all sign up for the No4j sandbox.

2. After successful login, you will be asked to create or launch a existing project. Here I am going to use pre built "Movies" project.

Find

Example queries for finding individual nodes.

1. Click on any query example
2. Run the query from the editor
3. Notice the syntax pattern
4. Try looking for other movies or actors

Query

Finding patterns within the graph.

1. Actors are people who acted in movies
2. Directors are people who directed a movie
3. What other relationships exist?

Create

To the right is a giant code block containing a single Cypher query statement composed of multiple CREATE clauses. This will create the movie graph.

1. Click on the code block
2. Notice it gets copied to the editor above ↑
3. Click the editor's play button to execute
4. Wait for the query to finish

Solve

You've heard of the classic "Six Degrees of Kevin Bacon"? That is simply a shortest path query called the "Bacon Path".

1. Variable length patterns
2. Built-in shortestPath() algorithm

Let's get started with Gephi!!

What is Gephi?

Gephi is an open-source network analysis and visualization software package written in Java on the NetBeans platform. Gephi is a visualization and exploration software for graphs and networks. Think Photoshop, but for graph data. Gephi is open-source, free to download, and runs on Windows, Mac OS X, and Linux.

1. Install Gephi from here.

2. Start Gephi and select a sample project for demo purpose.

3. Now you can see the graph window.

If you cannot see the graph window, you can enable it from Window->Graphs in menu bar.

4. Using direct selection tool you can highlight all the nodes directly connected to the selected node.

5. Explore different tools from the graph tool palette

6. Loading graph data in Gephi.

Gephi support CSV file format to import data. You can import nodes using node table, edges using edge table, nodes and edges using adjacency list and adjacency matrix.

For example :

The sample below shows a node table of of three nodes. The column of node identifiers must be named “Id”.
Id A B C

The sample below shows an edge table of of two edge. The columns must be named “Source” and “Target”.
Source,Target A,B C,A

The sample below shows an adjacency list of 6 edges and 4 nodes.
q,w,e,r w,q,e,r

The sample below shows an adjacency matrix of 2 edges and 2 nodes.
,Z,X Z,0,1 X,1,0

Now creating the above mentioned CSV file and importing it to Gephi.

Importing nodes

Importing edges

Importing adjacency list

Importing adjacency matrix

Final Graph

Final edge table

These are the various ways to store and load graph dataset in Gephi.

References:

https://neo4j.com/developer/graph-database/
https://gephi.org/users/supported-graph-formats/spreadsheet/

Practical 8

Aim: Perform Data Analytics using PowerBI using the given dataset. Generate a report that contains various visualizations.

Dataset Description:

This work tries to build a model to predict whether the policyholders (customers) from the past year will also be interested in Vehicle Insurance provided by the company.

An insurance policy is an arrangement by which a company undertakes to provide a guarantee of compensation for specified loss, damage, illness, or death in return for the payment of a specified premium. A premium is a sum of money that the customer needs to pay regularly to an insurance company for this guarantee.

This work using the Health insurance prediction classification. Predictor variable :

1. Gender
2. Age
3. Driving_License
4. Region_Code
5. Previously_Insured
6. Vehicle_Age
7. Vehicle_Damage
8. Annual_Premium
9. Policy_Sales_Channel
10. Vintage

Steps to Generate reports using Power BI

1. Load the data.

2. Explore your dashboard.

3. Choose the Filed 

4. Generate reports that contains various visualizations

Reference

Practical 7

Aim: Consume data with Power BI and How to build a simple dashboard.

Consumer personalization is a major operation of financial institutions. With the help of real-time analytics, data scientists are able to take insights from consumer behavior and are able to take appropriate business decisions. Financial institutions like insurance companies make use of consumer analytics to measure the customer lifetime value, increase their cross-sales as well as reduce the below zero customers for optimizing the losses.

Steps to create Power BI Dashboard.

1. Load the data.

2. Explore your dashboard.

3. Choose the Filed 

Reference

Practical 5

Data Pre-processing and text analytics using Orange

What is text analytics?

Text analysis is a machine learning technique that allows us to automatically extract and classify text data such as tweets, emails, support tickets, product reviews, and survey responses. It is often used to describe, interpret and understand texts. It often aims to connect the text to a broader social, political, cultural or artistic context.

What is Sentiment Analysis?

Sentiment analysis is a machine learning tool that analysis texts for polarity from positive to negative. By training machine learning tools with examples of emotions in text, machines automatically learn how to detect sentiment without human input.

WHY?

Sentiment analysis is extremely useful in social media monitoring as it allows us to gain an overview of the wider public opinion behind certain topics. Sentiment analysis is useful for quickly gaining insights using large volumes of text.

Preprocessing is a key component in Data Science. Orange tool has various ways to achieve the activities.

Discretization: 

It is the process of transferring continuous functions, models, variables, and equations into discrete counterparts. This process is usually carried out as a first step toward making them suitable for numerical evaluation and implementation data by the models.

Discretization replaces continuous features with the corresponding categorical features:

Sample Code:

import Orange

store = Orange.data.Table("Mart_Production.tab")

disc = Orange.preprocess.Discretize()

disc.method = Orange.preprocess.discretize.EqualFreq(n=3)

d_store = disc(store)

print("Original dataset:")

for e in store[:3]:

    print(e)

print("Discretized dataset:")

for e in d_store[:3]:

    print(e)

Continuization:

Given a data table, return a new table in which the discretize attributes are replaced with continuous or removed.

• binary variables are transformed into 0.0/1.0 or -1.0/1.0 indicator variables, depending upon the argument zero_based.
• multinomial variables are treated according to the argument multinomial_treatment.
• discrete attribute with only one possible value are removed

Sample Code:

import Orange

products = Orange.data.Table("Products")

continuizer = Orange.preprocess.Continuize()

products1 = continuizer(titanic)

Normalization: 

It is a systematic approach of decomposing tables to eliminate data redundancy(repetition) and undesirable characteristics like Insertion, Update and Deletion Anomalies.

Sample Code:

>>> from Orange.data import Table

>>> from Orange.preprocess import Normalize

>>> data = Table("Customers")

>>> normalizer = Normalize(norm_type=Normalize.NormalizeBySpan)

>>> normalized_data = normalizer(data)

Randomization:

A method based on chance alone by which study participants are assigned to a treatment group. Randomization minimizes the differences among groups by equally distributing people with particular characteristics among all the trial arms.

Sample Code:

>>> from Orange.data import Table

>>> from Orange.preprocess import Randomize

>>> data = Table("Returns")

>>> randomizer = Randomize(Randomize.RandomizeClasses)

>>> randomized_data = randomizer(data)

Dataset Description

The aim is to build a predictive model and find out the sales of each product at a particular store. Using this model, BigMart will try to understand the properties of products and stores which play a key role in increasing sales.

Total Rows:- 8523

Total Columns:- 13

Target Variable:- Item_Outlet_SalesEffect of Different kind of pre-processing on data

Work Flow for pre-processing

Assigning target variable

Applying Pre-processing to dataset

Data obtained after pre-processing

Work Flow for running python script in orange

Python script to get data according to target variable

Output of above python script

Python script to round all data in dataset

Output of above python script

Reference 

Practical 4

Visual programming with Orange tool

What is Visual Programming ?

Visual programming is a form of programming language that uses illustration to allow humans to represent processes. A visual programming language helps the programmer to explain the process in terms that make sense to humans, while a traditional text-based programming language lets the programmer think like a machine.

Now we will train and test our model using visual programming with orange tool.

Let's start digging!!

1. Load the dataset by adding file widget.

2. Now we will add data sampler widget and we will split the data into two parts, 85% of data for training and 15% for testing. We will send the first 85% onwards to build a model.

3. Add kNN Model widget to train the model.

4. Add a test and score widget.

Now exploring the results

Accuracy of at 85% sampling.

Accuracy of at 60% sampling.

Now changing the value of Number of folds in cross validation.

Number folds = 10

Number folds = 5

Number folds = 20

Question and Answers

1. Does this visual programing is enough for data analytics?

No, Visual programming may provide easy to implement environment, but it doesn't give developer the full control over the program.

2. How to get much more deeper insights of data?

There can be 3 simple steps to get much more deeper insights of data:

1. Use As Much Data As You Can

2. Automate Your Discovery of Insights

3. Leverage Machine Learning for the Masses

Reference