Data Science-Descriptive Statistics


https://www.youtube.com/watch?v=K4sObJbN4JU

 Sampling Error: Difference between sample and population averages (X bar and Mu)
Parameter:A number that describes population is called parameter
Statistics:A number that describes sample is called Statistics
Correlation does not prove Causation
Constructs: Difficult to measure like anger, hard work, intelligence, hunger, itchiness
Operational Definitions: Constructs can be converted into variables with the help of operational definitions.
Dependent/Independent variables: Variables that scientists are interested in measuring by varying the independent variable.
Extraneous or lurking variables: Are other variables that may or may not interfere with the experiment. Any Extraneous variable that impact the results we call them compounding experiments
Control variable: Variable that are kept same in each trail e.g. ball size, number of people
Moderator variables: Are variables that can increase or decrease the relationship between dependent or independent variables
Golden Arch Theory of Conflict Prevention
Response bias: If people do not understand the Q
Non Response bias: When some set of people refuse to answer questions in a survey
Blinding: Not letting the participants know which treatment they are receiving is known as blinding
Placebo: Inactive pill people receive are known as Placebo, or in other words, Placebo are fake treatments that are given to Control Group undergoing the experiment. People who are part of an experiment if told about the type of pill they are given, they will subconsciously be thinking about it all the time resulting in inaccurate survey or experiment results. A beneficial effect produced by a placebo drug or treatment, which cannot be attributed to the properties of the placebo itself, and must therefore be due to the patient's belief in that treatment.
Double-Blind: Neither participants nor the Researchers are aware which treatment (pill) participants are receiving
Hypothesis: Proposed explanation made on the basis of limited evidence as a starting point for further investigation. It is also called the the statements about the relationships between variables.
Frequency Distribution table creation: 
  • 0-19: Is called Interval or bin and bin size here is 20
  • Histogram vs Bar Graph: In case of Histogram, same category is represented on Axis and therefore are connected but in case of Bar Graph, categories are not same and are distinct hence separated. In Histogram often data on x axis is quantitative and in bar graph it is often qualitative.
  • Normal distribution, Left (Negative)/Right (Positive) skewed graphs: Green has a big left side tail hence it is left skewed and known as Negatively skewed
  • For Left skewed distributions, mean is less than median which is less than mode
  • For Right skewed distributions, mean is more than median which is more than mode
  • Area of all distributions is ONE
  • Mode, Median and Mean:

  • Mode: Occurs with highest frequency. Mode is not influenced by a outlier. It can be used to describe categorical data such as Gender or Country of Origin
  • Uniform distribution is a straight line with No modes
  • Bi modal distribution : Distribution with 2 modes (like shoe size of women and men e.g.)
  • Average/Mean: xbar. Mean is influenced by a outlier.
  • Median: Median is not influenced much by a outlier.
 
  • Range: Another way to visualize the data. At times it is not the correct representation of the data. So to find more accurate representation we divide data in Quartiles (Q1/Q2/Q3) and Q3-Q1 is called Interquartile: IQR=(Q3-Q1)
  • Outlier < Q1-1.5(IQR) and > Q3+1.5(IQR)
  • Boxplots
  • It is not necessary that mean will always be in between Q1 and Q3 though generally it will be
----------
  • Deviation: xi-xbar
  • Squared Deviation: (xi-xbar)^2
  • Sum of squares (ss): sum(xi-xbar)^2
  • Variance: ss/n
  • Std Deviation (sigma): sqrt(Variance)  




  • 68% of the data falls in 1 STD of the mean
  • 95% of the data falls in 2 STD of the mean
  • Bessel's correction; Normally samples are taken from middle of the distribution curve and hence are more than the population average. To compensate this, Bessel's correction is used both in STD and Variance (only in case estimation has to be done for a population)
  • Standard Deviation of sample (divided by n-1) vs standard deviation of population (divided by n) 
  • Categorical variables represent types of data which may be divided into groups. Examples of categorical variables are race, sex, age group, and educational level. While the latter two variables may also be considered in a numerical manner by using exact values for age and highest grade completed, it is often more informative to categorize such variables into a relatively small number of groups.
  • Frequency vs Relative Frequency: Changing frequency into % is known as relative frequency
  • Z=(x-Mu)/Delta: Z score is defined as "How many Standard Deviations is any value away from the mean"
  • Lesson13.19: 
    • If we standardize a distribution by converting every value to a Z score what will be the new mean of this standardised distrubution=ZERO
    • If we standardize a distribution by converting every value to a Z score what will be the new standard distribution of this standardised distrubution=1
  • Standard Normal Distribution: It is the distribution where Mean is Zero and Std deviation is 
  • PDF: Probability Distribution Function is the curve we see
  • Area under the curve (Relative Probability Distribution Curve): Probability of randomly selecting less than X (X being a point on X Axis)=Proportion in sample/population with score less than X


Mean of simple mean (M)
  • Relation between population standard deviation (Sigma) and standard deviation of distribution of sample means also known as sampling distribution (SE) 
  •  σ/SE = Sqrt (n)

Central Limit Theorem
Let us say there is a huge population with Standard Deviation as σ. Take a sample and note its mean. Take another sample and note its mean and keep on doing it. Now plot the mean of all the samples taken and you will see that if samples are large enough, it will be plotted as a normal distribution curve such that standard deviation (SE) of this distribution is equal to  σ/sqrt(n), where n is a sample size and SE=Standard Error.

So SE = σ/Sqrt (n)

  • As size of sample  increases, distribution of mean of mean becomes more skinnier and SE decreases. Mean of sampling distribution is same as population mean
  • If we Quadruple n then the SE will become half or measurement error becomes half 
  • If sample size increases, SE decreases
Simulation: http://onlinestatbook.com/stat_sim/sampling_dist/index.html
  • Good job! We'll describe the location of the sample mean by calculating how many standard errors it is away from the center of the sampling distribution. That will give us a z-score for our sample mean.
  • Why probability of randomly selecting a sample mean greater than 110 decrease when we used a sample of 25 rather than a sample of 4.
    • Bigger sample sample size resulted in bigger Z score for that sample mean
    • Bigger the Z score, less the proportion of sample means greater than that sample mean
    • Bigger sample sizes result in skinnier sampling distribution
---------------------------------------------------------------------------------------------

Essential Statistics for Data Analysis using Excel
https://courses.edx.org/courses/course-v1:Microsoft+DAT222x+2T2018/courseware/5bc836d5d8864b2ca446e7536a8208e0/daeb5466e8ed4c24bf96dedc7ee6aeac/?child=first
  1. There are 2 types of data: Categorical and numerical
    1. Numerical: It can be discrete or continuous
      1. Discrete: Number of children in a family, shoe sizes
      2. Continuous: Height or weight of a person, length of a leg
    2. Categorical: cannot be assigned a numerical value
      1. Nominal (categories which are just names): Some kind of Color of a car, fruits, genders, ethnicity 
      2. Ordinal (categories can be sorted in a logical fashion): Some kind of order is there e.g. BIG, SMALL, MEDIUM...GOOD, BAD, BEST...YES,MAYBE,NO...
  2. Difference between Histogram and bar chart
    1. Histogram: 
      1. Represents a frequency distribution
      2. Continuous values on x axis
      3. Shows Numerical data- distribution of non discrete variable
      4. Value represented by area of the column (height x bin size) 
    2. Bar Chart:
      1. Used to represent comparison of variables
      2. Gaps in the values on x axis
      3. Shows Categorical data - comparing discrete variable
      4. Value defined by  height of the bar
  3. Excel add in of Analysis Took pak  Data-->Data Analysis
  4. How to decide Typical value:
    1. While doing comparing analysis in case there is a lot of skewness, you should use Median else mean can be used
    2. Lot of skewness means.. skewness <-1 (negitivley skewed) or >+1 (positively skewed)
  5. How to calculate more spread 
    1. More the standard deviation, more the spread
  6. Excel formulas:
    1. Percentile.EXC: Returns a value from the data set
    2. PERCENTRANK: gives the rank of a data point in data set, i.e. probability of a number falling below this number
    3. SKEW
    4. KURT
    5. LARGE: If we want to know 2nd highest number in an array
    6. SMALL: If we want to know 2nd smallest number in an array
    7. RANK.EQ: Find rank of a particular score
    8. RANK.AVG: It gives an average score in case the scores are equal
    9. FORMULATEXT: Gives the formula used in the cell
    10. Geometric Mean: 
      1. (X1*X2*X3......Xn)^(1/n): Used when you are comparing 2 variables with different properties. e.g. when we want to compare 2 companies for their compensation packages and citizenship rating. Note both are having a very different scale so if we do an average by giving same weight to both variables, we will find that citizenship ratings have been over weighted in the calculation. 
      2. Formula in Excel: GEOMEAN
    11. Compound Annual Growth rate (CAGR): GEOMEAN (range) - 1
      1. Take 1 PLUS all the values
      2. Then take a GEOMEAN
      3. Then subtract 1 to get CAGR
    12. Box plot: If observations are concentrated on the low end of the box, then distribution is skewed right and if observations are concentrated on the upper end of the box, then distribution is skewed left
    13. Hierarchical data: Tree map and sunburst charts
    14. Pareto principle

Comments

Post a Comment