Tutorial 1B: Machine Learning Experiment Scoring and Analysis Tutorial - Financial Focus

In this tutorial, we will be working with a subset of the Freddie Mac Single-Family Loan-Level Dataset to build a Classification model. You will be exploring how to evaluate a DAI model through tools like ROC, Prec-Recall, Gain and Lift Charts, K-S Chart as well as metrics such as AUC, F-Scores, GINI, MCC, and Log Loss.

Key:

Complete
Failed
Available
Locked
Objective
Select the "Read" button to begin.
Select the "Read" button to begin. Explore the goal of this tutorial
Prerequisites
Select the "Read" button to begin.
Select the "Read" button to begin. Find general information about what you need in order to complete this tutorial
Task 1: Launch Experiment
Select the "Read" button to begin.
Select the "Read" button to begin. Follow the next steps to upload your dataset, split it, drop columns and choose the target column.
Task 2: Explore Experiment Settings and Expert Settings
Select the "Read" button to begin.
Select the "Read" button to begin. Learn about the experiment settings of Accuracy, Time and Interpretability Knobs in Driverless AI, as well as the Expert Experiment settings
Task 3: Experiment Scoring and Analysis Concepts
Select the "Read" button to begin.
Select the "Read" button to begin. Review general concepts about scoring and analysis for a classification model. Some of the concepts that you will be reviewing are ROC, Prec-Recall, as well as metrics such as GINI, F1 among others.
Task 4: Experiment Results Summary
Select the "Read" button to begin.
Select the "Read" button to begin. Explore the Summary section to find out more about the scoring in Driverless AI
Task 5: Diagnostics Scores and Confusion Matrix
Select the "Read" button to begin.
Select the "Read" button to begin. Launch a diagnostic for your experiment, and check out the Confusion Matrix. You will learn about the labels used in the Confusion Matrix such as True Positive, True Negative, among others. You will also explore the scores for your model.
Task 6: ER: ROC
Select the "Read" button to begin.
Select the "Read" button to begin. Review the ROC curve, and launch a new model with the same parameters
Task 7: ER: Prec-Recall
Select the "Read" button to begin.
Select the "Read" button to begin. Review the Precision and Recall (P-R) curve and create another new model with the same parameters
Task 8: ER: Gains
Select the "Read" button to begin.
Select the "Read" button to begin. Review the Gains curve to measure the performance of your model
Task 9: ER: LIFT
Select the "Read" button to begin.
Select the "Read" button to begin. Review and learn about the Lift curve, another visual aid for measuring model performance
Task 10: Kolmogorov-Smirnov Chart
Select the "Read" button to begin.
Select the "Read" button to begin. Review and learn about the Kolmogorov-Smirnov chart
Task 11: Experiment AutoDocs
Select the "Read" button to begin.
Select the "Read" button to begin. Download the Experiment Summary to help you better understand and interpret your model
Next Steps
Select the "Read" button to begin.
Select the "Read" button to begin. Check out the next tutorial about Machine Learning Interpretability.
Quiz
25 Questions  |  2 attempts  |  20/25 points to pass
25 Questions  |  2 attempts  |  20/25 points to pass
ML Scoring & Analysis Badge
  |  Badge available
  |  Badge available After passing the quiz, you can get your badge by clicking on "Badge Earned". Please check your email address for instructions on how to view, manage, and share your new badge!
You must be logged in to post to the discussion