AroniSmartIntelligence™: Optimized Bayesian A/B Testing Capabilities - Implementation and Execution in the New Version 15.x

A/B Test analysis allows to make data-driven decisions about which Recipe or design is most effective or performs better. Hence, in digital marketing, A/B Statistical test can be used to compare marketing metrics like visits, clicks, conversion rates, engagements, drop-off rate, time spent on website, and more.

AroniSmartIntelligence™ A/B Test Capabilities Objectives.

A/B Test Approach in AroniSmartIntelligence™.

Given the required advanced knowledge of statistics probability distributions, A/B Bayesian Statistical Testing is included in AroniSmartIntelligence Module dedicated to Statistics Testing and Inference. However, it also requires understanding of Bayesian models, in Module 7 of AroniSmartIntelligence™.
A/B Bayesian Test is a significant improvement of the Frequentist A/B Testing, which is close to the regular Hypothesis testing implemented in the Statistical Inference.

GET ARONISMARTINTELLIGENCE on App Store

AroniSmartIntelligence, the leading tool for Advanced Analytics, Machine Learning & Data Science

Statisticians, Data Scientists, Business and Financial Analysts, Savvy Investors, Engineers, Researchers, Students, Teachers, Economists, Political Analysts, and most of the practitioners use Advanced Analytics to answer questions, to support informed decision making or to learn.

AroniSmartIntelligence™ is a leading Advanced Analytics, Machine Learning and Data Science tool, with optimized cutting edge Statistics models, Econometrics, Big Data and Text Analytics.

AronismartIntelligence™ includes modules covering Machine Learning, Big Data mining, Bayesian Statistics, Neural Network Models, Unstructured Text Analysis, Sentiment and Emotion Analytics, and other advanced analytics.

AroniSmartIntelligence™ A/B Testing Steps.

The new version 15.x of AroniSmartIntellligence™ app on Apple's AppStore®

The data requirements conditions for each model, approach, and module are documented through messages, tool tips, and documentation. FAQs and other relevant documents can be found on the AroniSoft (http://www.aronisoft.com ) Website.

1. Formulate and set the hypotheses

Before conducting an A/B testing, it is critical to formulate a null (H0) and an alternative hypothesis (H1),
Under the null hypothesis (H0), the assumptions are that there is no difference between the control and test group.
The null hypothesis assumes that there is no difference between the control and test groups.
The alternative hypothesis relies on that assumptions that the changes are real and will be confirmed by A/B test and that changes in the sample observations are not random. From an A/B test perspective, the alternative hypothesis assumes that there is a difference between the control and the test groups.

For example, an alternative hypothesis may be that adding features to a web page would increase visits or clicks.

2. Create your control group and test group

Once you determine your null and alternative hypothesis, the next step is to create your control and test (variant) group. There are two important concepts to consider in this step, random samplings and sample size.

3. Determine the sample size

The sample size will determine how long long the experiment will last in order to collect all the required data.

4. Fix the p value, before any analysis.

As a recommendation, A/B Testing needs to follow what is known as PICOT process.

Population: the group of people or other targets that participate in the experiment or study
Intervention: the new variant (Test) in the study
Comparison: a reference group (Control) to compare against your intervention (Test)
Outcome: the result to be measured
Time: the duration of the experiment, that is when and how long the data is collected.

A/B Testing: Frequentist vs Bayesian Approach AroniSmartIntelligence™.

Frequentist A/B Testing

The regular A/B testing, as in the Frequentist Hypothesis testing, has the following approach:

a) Null hypothesis:
The null hypothesis, or H0, assumes that there is no difference between Recipe A (Control) and Recipe B (Test or Treatment). For example, in digital marketing, a Control and Test marketing asset would have no impact on user conversions.

b) Alternative hypothesis :
An alternative hypothesis suggests the opposite of the null hypothesis. It assumes that there is a difference between Recipe A (Control) and Recipe B (Test) , and hence moving to Recipe B (Test) will impact user conversions.
c) Statistical significance
Statistical significance is calculated by measuring the p-value, or probability value. A low p-value points to the fact that the results of the A/B test were not random. In general , a p-value of 5% or lower means that the A/B test is statistically significant.
d) Confidence level
The confidence level is the inverse of the p-value. It indicates that the results of A/B experiment are most likely due to the changes in variables or assets.

A statistically significant test leads to a p-value less or equal to 5%, hence to the confidence level of 95%.

The Frequentist A/B Testing approach usually faces challenges related to errors. The errors include Type 1 and Type 2 errors. A Type 1 error leads to deciding that a worse Recipe is better when it really isn’t. A Type 2 error is finding that a better Recipe is equal or worse than a bad Recipe. The reason is that the Frequentist A/B Testing relies on just the available samples that may be very small or not representative.

Bayesian A/B Testing

Bayesian A/B Testing relies on the approach used in the Bayesian models as implemented in Module 7 of AroniSmartIntelligence™. With advances in Statistics and Machine Learning of Artificial Intellligence, Bayesian analytics framework offers a simpler, yet more open, reliable, and intuitive approach to A/B testing.
It takes three steps:
a) Selecting and Setting Inputs: Recipe Sample A (Control) and Recipe Sample B (Treatment/Test), Distributions and Priors
b) Running the Bayesian A/B Test Models and getting the results
c) Getting and Analyzing outputs and results, including the statistics, Probabilities of A is better that B or B is better than A, confidence interval and lift, expected loss (A-B)/B and plots

The input takes two vectors of data: Recipe A ( Control or Variation A ) and Recipe B (Test, Treatment or Variation B). Which Recipe is the control or treatment matters only for the analysis and interpretability of the final results, plots, and intervals/point estimates.

The following Probability Distributions are implemented for A/B Bayesian Test in AroniSmartIntelligence™: Binomial/Bernoulli, Exponential, Normal, LogNormal, Poisson, Uniform, along with their conjugates: Beta, Gamma, NormalInverseGamma, and Pareto.

For More, check the AroniSmartIntellligence™ app on Apple's AppStore®

Below are a few an example of A/B Bayesian Test and Regular Frequentist Hypothesis Testing, executed in AroniSmartIntelligence™.

Welcome -- Statistical Inference Module

Figure 1: Welcome -- Select the Statistical Inference Module

Descriptive Statistics

Figure 2: Statistical Inference Module: Descriptive Analytics

Hypothesis Testing

Figure 3 - a : Statistical Inference Module: Frequentist Hypothesis Testing Capability - T-Tests

Hypothesis Testing

Figure 3 - b : Statistical Inference Module: Frequentist Hypothesis Testing Capability - F Tests

AB Bayesian Test Capability

Figure 4: Statistical Inference Module: AB Bayesian Test Capability

AB Bayesian Test Inputs Analysis

Figure 5: AB Bayesian Test Inputs Analysis

AB Bayesian Test Inputs Analysis - Posteriors Analysis Results

Figure 6: AB Bayesian Test Analysis - Posteriors Analysis Results

AB Bayesian Test Inputs Analysis - Posteriors Analysis Results

Figure 6: AB Bayesian Test Analysis - (A-B)/B Samples Posteriors Analysis Results

For more visit here AroniSmartIntelligence™: Optimized Machine Learning, Advanced Analytics, and Data Science Capabilities Including Econometrics, Bayesian Models, Neural Network Models, Marketing Mix Models, and NLP Analytics

FOR QUICK DEMOS ON HANDBOOK and USER'S MANUALS, PLEASE CLICK ON THE LINKS BELOW TO DOWNLOAD THE PDF FILES:

AroniSmartIntelligence™ Handbook and Manual: Click here to download a presentation in pdf format
AroniSmartIntelligence™ Data loading and Descriptive statistics: Click here to download a presentation in pdf format
AroniSmartIntelligence™ Statistical Inference and Testing: Click here to download a presentation in pdf format
AroniSmartIntelligence™ Regression Analysis and Time Series : Click here to download a presentation in pdf format
AroniSmartInttelligence™ Big Data Text Mining: Click here to download a presentation in pdf format
AroniSmartInttelligence™ Segmentation and Mixture Models: Click here to download a presentation in pdf format
AroniSmartInttelligence™ Bayesian Models: Click here to download a presentation in pdf format
AroniSmartInttelligence™ Neural Network Models: Click here to download a presentation in pdf format

Advertising:

GET ARONISMARTINTELLIGENCE on App Store

What is next?

The new version of AroniSmartIntelligence™ that includes improved Advanced Analytics, Machine Learning and Data Science is available on Apple's App Store®.

More detailed information on the capabilities of AroniSmartIntelligence™ is available in the tool on Apple®'s App Store and on the AroniSoft web site(http://www.aronisoft.com).

Check the Frequently Asked Questions (FAQ) here.

More detailed use cases, real life applications, and AroniSmart Tools in actions can be found on on the AroniSmart web site (http://www.aronismart.com). Check the following tabs: ABOUT US, SOLUTIONS, SCITECH, MONEY.