
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "auto_examples/model_selection/plot_grid_search_stats.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_auto_examples_model_selection_plot_grid_search_stats.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_auto_examples_model_selection_plot_grid_search_stats.py:


==================================================
Statistical comparison of models using grid search
==================================================

This example illustrates how to statistically compare the performance of models
trained and evaluated using :class:`~sklearn.model_selection.GridSearchCV`.

.. GENERATED FROM PYTHON SOURCE LINES 10-14

.. code-block:: Python


    # Authors: The scikit-learn developers
    # SPDX-License-Identifier: BSD-3-Clause








.. GENERATED FROM PYTHON SOURCE LINES 15-19

We will start by simulating moon shaped data (where the ideal separation
between classes is non-linear), adding to it a moderate degree of noise.
Datapoints will belong to one of two possible classes to be predicted by two
features. We will simulate 50 samples for each class:

.. GENERATED FROM PYTHON SOURCE LINES 19-32

.. code-block:: Python


    import matplotlib.pyplot as plt
    import seaborn as sns

    from sklearn.datasets import make_moons

    X, y = make_moons(noise=0.352, random_state=1, n_samples=100)

    sns.scatterplot(
        x=X[:, 0], y=X[:, 1], hue=y, marker="o", s=25, edgecolor="k", legend=False
    ).set_title("Data")
    plt.show()




.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_001.png
   :alt: Data
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_001.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 33-41

We will compare the performance of :class:`~sklearn.svm.SVC` estimators that
vary on their `kernel` parameter, to decide which choice of this
hyper-parameter predicts our simulated data best.
We will evaluate the performance of the models using
:class:`~sklearn.model_selection.RepeatedStratifiedKFold`, repeating 10 times
a 10-fold stratified cross validation using a different randomization of the
data in each repetition. The performance will be evaluated using
:class:`~sklearn.metrics.roc_auc_score`.

.. GENERATED FROM PYTHON SOURCE LINES 41-58

.. code-block:: Python


    from sklearn.model_selection import GridSearchCV, RepeatedStratifiedKFold
    from sklearn.svm import SVC

    param_grid = [
        {"kernel": ["linear"]},
        {"kernel": ["poly"], "degree": [2, 3]},
        {"kernel": ["rbf"]},
    ]

    svc = SVC(random_state=0)

    cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=10, random_state=0)

    search = GridSearchCV(estimator=svc, param_grid=param_grid, scoring="roc_auc", cv=cv)
    search.fit(X, y)






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <style>.sk-global {
      /* Definition of color scheme common for light and dark mode */
      --sklearn-color-text: #000;
      --sklearn-color-text-muted: #666;
      --sklearn-color-line: gray;
      /* Definition of color scheme for unfitted estimators */
      --sklearn-color-unfitted-level-0: #fff5e6;
      --sklearn-color-unfitted-level-1: #f6e4d2;
      --sklearn-color-unfitted-level-2: #ffe0b3;
      --sklearn-color-unfitted-level-3: chocolate;
      /* Definition of color scheme for fitted estimators */
      --sklearn-color-fitted-level-0: #f0f8ff;
      --sklearn-color-fitted-level-1: #d4ebff;
      --sklearn-color-fitted-level-2: #b3dbfd;
      --sklearn-color-fitted-level-3: cornflowerblue;
    }

    .sk-global.light {
      /* Specific color for light theme */
      --sklearn-color-text-on-default-background: black;
      --sklearn-color-background: white;
      --sklearn-color-border-box: black;
      --sklearn-color-icon: #696969;
    }

    .sk-global.dark {
      --sklearn-color-text-on-default-background: white;
      --sklearn-color-background: #111;
      --sklearn-color-border-box: white;
      --sklearn-color-icon: #878787;
    }

    .sk-global {
      color: var(--sklearn-color-text);
    }

    .sk-global pre {
      padding: 0;
    }

    .sk-global input.sk-hidden--visually {
      border: 0;
      clip-path: inset(100%);
      height: 1px;
      margin: -1px;
      overflow: hidden;
      padding: 0;
      position: absolute;
      width: 1px;
    }

    .sk-global div.sk-dashed-wrapped {
      border: 1px dashed var(--sklearn-color-line);
      margin: 0 0.4em 0.5em 0.4em;
      box-sizing: border-box;
      padding-bottom: 0.4em;
      background-color: var(--sklearn-color-background);
    }

    .sk-global div.sk-container {
      /* jupyter's `normalize.less` sets `[hidden] { display: none; }`
         but bootstrap.min.css set `[hidden] { display: none !important; }`
         so we also need the `!important` here to be able to override the
         default hidden behavior on the sphinx rendered scikit-learn.org.
         See: https://github.com/scikit-learn/scikit-learn/issues/21755 */
      display: inline-block !important;
      position: relative;
    }

    .sk-global div.sk-text-repr-fallback {
      display: none;
    }

    div.sk-parallel-item,
    div.sk-serial,
    div.sk-item {
      /* draw centered vertical line to link estimators */
      background-image: linear-gradient(var(--sklearn-color-text-on-default-background), var(--sklearn-color-text-on-default-background));
      background-size: 2px 100%;
      background-repeat: no-repeat;
      background-position: center center;
    }

    /* Parallel-specific style estimator block */

    .sk-global div.sk-parallel-item::after {
      content: "";
      width: 100%;
      border-bottom: 2px solid var(--sklearn-color-text-on-default-background);
      flex-grow: 1;
    }

    .sk-global div.sk-parallel {
      display: flex;
      align-items: stretch;
      justify-content: center;
      background-color: var(--sklearn-color-background);
      position: relative;
    }

    .sk-global div.sk-parallel-item {
      display: flex;
      flex-direction: column;
    }

    .sk-global div.sk-parallel-item:first-child::after {
      align-self: flex-end;
      width: 50%;
    }

    .sk-global div.sk-parallel-item:last-child::after {
      align-self: flex-start;
      width: 50%;
    }

    .sk-global div.sk-parallel-item:only-child::after {
      width: 0;
    }

    /* Serial-specific style estimator block */

    .sk-global div.sk-serial {
      display: flex;
      flex-direction: column;
      align-items: center;
      background-color: var(--sklearn-color-background);
      padding-right: 1em;
      padding-left: 1em;
    }


    /* Toggleable style: style used for estimator/Pipeline/ColumnTransformer box that is
    clickable and can be expanded/collapsed.
    - Pipeline and ColumnTransformer use this feature and define the default style
    - Estimators will overwrite some part of the style using the `sk-estimator` class
    */

    /* Pipeline and ColumnTransformer style (default) */

    .sk-global div.sk-toggleable {
      /* Default theme specific background. It is overwritten whether we have a
      specific estimator or a Pipeline/ColumnTransformer */
      background-color: var(--sklearn-color-background);
    }

    /* Toggleable label */
    .sk-global label.sk-toggleable__label {
      cursor: pointer;
      display: flex;
      width: 100%;
      margin-bottom: 0;
      padding: 0.5em;
      box-sizing: border-box;
      text-align: center;
      align-items: center;
      justify-content: center;
      gap: 0.5em;
    }

    .sk-global label.sk-toggleable__label .caption {
      font-size: 0.6rem;
      font-weight: lighter;
      color: var(--sklearn-color-text-muted);
    }

    .sk-global label.sk-toggleable__label-arrow:before {
      /* Arrow on the left of the label */
      content: "▸";
      float: left;
      margin-right: 0.25em;
      color: var(--sklearn-color-icon);
    }

    .sk-global label.sk-toggleable__label-arrow:hover:before {
      color: var(--sklearn-color-text);
    }

    /* Toggleable content - dropdown */

    .sk-global div.sk-toggleable__content {
      display: none;
      text-align: left;
      /* unfitted */
      background-color: var(--sklearn-color-unfitted-level-0);
    }

    .sk-global div.sk-toggleable__content.fitted {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-0);
    }

    .sk-global div.sk-toggleable__content pre {
      margin: 0.2em;
      border-radius: 0.25em;
      color: var(--sklearn-color-text);
      /* unfitted */
      background-color: var(--sklearn-color-unfitted-level-0);
    }

    .sk-global div.sk-toggleable__content.fitted pre {
      /* unfitted */
      background-color: var(--sklearn-color-fitted-level-0);
    }

    .sk-global input.sk-toggleable__control:checked~div.sk-toggleable__content {
      /* Expand drop-down */
      display: block;
      width: 100%;
      overflow: visible;
    }

    .sk-global input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {
      content: "▾";
    }

    /* Pipeline/ColumnTransformer-specific style */

    .sk-global div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {
      color: var(--sklearn-color-text);
      background-color: var(--sklearn-color-unfitted-level-2);
    }

    .sk-global div.sk-label.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {
      background-color: var(--sklearn-color-fitted-level-2);
    }

    /* Estimator-specific style */

    /* Colorize estimator box */
    .sk-global div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {
      /* unfitted */
      background-color: var(--sklearn-color-unfitted-level-2);
    }

    .sk-global div.sk-estimator.fitted input.sk-toggleable__control:checked~label.sk-toggleable__label {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-2);
    }

    .sk-global div.sk-label label.sk-toggleable__label,
    .sk-global div.sk-label label {
      /* The background is the default theme color */
      color: var(--sklearn-color-text-on-default-background);
    }

    /* On hover, darken the color of the background */
    .sk-global div.sk-label:hover label.sk-toggleable__label {
      color: var(--sklearn-color-text);
      background-color: var(--sklearn-color-unfitted-level-2);
    }

    /* Label box, darken color on hover, fitted */
    .sk-global div.sk-label.fitted:hover label.sk-toggleable__label.fitted {
      color: var(--sklearn-color-text);
      background-color: var(--sklearn-color-fitted-level-2);
    }

    /* Estimator label */

    .sk-global div.sk-label label {
      font-family: monospace;
      font-weight: bold;
      line-height: 1.2em;
    }

    .sk-global div.sk-label-container {
      text-align: center;
    }

    /* Estimator-specific */
    .sk-global div.sk-estimator {
      font-family: monospace;
      border: 1px dotted var(--sklearn-color-border-box);
      border-radius: 0.25em;
      box-sizing: border-box;
      margin-bottom: 0.5em;
      /* unfitted */
      background-color: var(--sklearn-color-unfitted-level-0);
    }

    .sk-global div.sk-estimator.fitted {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-0);
    }

    /* on hover */
    .sk-global div.sk-estimator:hover {
      /* unfitted */
      background-color: var(--sklearn-color-unfitted-level-2);
    }

    .sk-global div.sk-estimator.fitted:hover {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-2);
    }

    /* Specification for estimator info (e.g. "i" and "?") */

    /* Common style for "i" and "?" */

    .sk-estimator-doc-link,
    a:link.sk-estimator-doc-link,
    a:visited.sk-estimator-doc-link {
      float: right;
      font-size: smaller;
      line-height: 1em;
      font-family: monospace;
      background-color: var(--sklearn-color-unfitted-level-0);
      border-radius: 1em;
      height: 1em;
      width: 1em;
      text-decoration: none !important;
      margin-left: 0.5em;
      text-align: center;
      /* unfitted */
      border: var(--sklearn-color-unfitted-level-3) 1pt solid;
      color: var(--sklearn-color-unfitted-level-3);
    }

    .sk-estimator-doc-link.fitted,
    a:link.sk-estimator-doc-link.fitted,
    a:visited.sk-estimator-doc-link.fitted {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-0);
      border: var(--sklearn-color-fitted-level-3) 1pt solid;
      color: var(--sklearn-color-fitted-level-3);
    }

    /* On hover */
    div.sk-estimator:hover .sk-estimator-doc-link:hover,
    .sk-estimator-doc-link:hover,
    div.sk-label-container:hover .sk-estimator-doc-link:hover,
    .sk-estimator-doc-link:hover {
      /* unfitted */
      background-color: var(--sklearn-color-unfitted-level-3);
      border: var(--sklearn-color-fitted-level-0) 1pt solid;
      color: var(--sklearn-color-unfitted-level-0);
      text-decoration: none;
    }

    div.sk-estimator.fitted:hover .sk-estimator-doc-link.fitted:hover,
    .sk-estimator-doc-link.fitted:hover,
    div.sk-label-container:hover .sk-estimator-doc-link.fitted:hover,
    .sk-estimator-doc-link.fitted:hover {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-3);
      border: var(--sklearn-color-fitted-level-0) 1pt solid;
      color: var(--sklearn-color-fitted-level-0);
      text-decoration: none;
    }

    /* Span, style for the box shown on hovering the info icon */
    .sk-estimator-doc-link span {
      display: none;
      z-index: 9999;
      position: relative;
      font-weight: normal;
      right: .2ex;
      padding: .5ex;
      margin: .5ex;
      width: min-content;
      min-width: 20ex;
      max-width: 50ex;
      color: var(--sklearn-color-text);
      box-shadow: 2pt 2pt 4pt #999;
      /* unfitted */
      background: var(--sklearn-color-unfitted-level-0);
      border: .5pt solid var(--sklearn-color-unfitted-level-3);
    }

    .sk-estimator-doc-link.fitted span {
      /* fitted */
      background: var(--sklearn-color-fitted-level-0);
      border: var(--sklearn-color-fitted-level-3);
    }

    .sk-estimator-doc-link:hover span {
      display: block;
    }

    /* "?"-specific style due to the `<a>` HTML tag */

    .sk-global a.estimator_doc_link {
      float: right;
      font-size: 1rem;
      line-height: 1em;
      font-family: monospace;
      background-color: var(--sklearn-color-unfitted-level-0);
      border-radius: 1rem;
      height: 1rem;
      width: 1rem;
      text-decoration: none;
      /* unfitted */
      color: var(--sklearn-color-unfitted-level-1);
      border: var(--sklearn-color-unfitted-level-1) 1pt solid;
    }

    .sk-global a.estimator_doc_link.fitted {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-0);
      border: var(--sklearn-color-fitted-level-1) 1pt solid;
      color: var(--sklearn-color-fitted-level-1);
    }

    /* On hover */
    .sk-global a.estimator_doc_link:hover {
      /* unfitted */
      background-color: var(--sklearn-color-unfitted-level-3);
      color: var(--sklearn-color-background);
      text-decoration: none;
    }

    .sk-global a.estimator_doc_link.fitted:hover {
      /* fitted */
      background-color: var(--sklearn-color-fitted-level-3);
    }

    .sk-top-container.sk-global {
      /* pydata-sphinx-theme hides overflow, so scrolling is disabled.
       We need to set it to !important and add tabindex="0" in the HTML
       to allow keyboard-only users to navigate the display. */
      overflow-x: scroll !important;
      max-width: 100%;
    }

    .estimator-table {
        font-family: monospace;
    }

    .estimator-table summary {
        padding: .5rem;
        cursor: pointer;
    }

    .estimator-table summary::marker {
        font-size: 0.7rem;
    }

    .estimator-table details[open] {
        padding-left: 0.1rem;
        padding-right: 0.1rem;
        padding-bottom: 0.3rem;
    }

    .estimator-table .parameters-table {
        margin-left: auto !important;
        margin-right: auto !important;
        margin-top: 0;
    }

    .estimator-table .parameters-table tr:nth-child(odd) {
        background-color: #fff;
    }

    .estimator-table .parameters-table tr:nth-child(even) {
        background-color: #f6f6f6;
    }

    .estimator-table .parameters-table tr:hover td {
        background-color: #e0e0e0;
    }

    .estimator-table table :is(td, th) {
        border: 1px solid rgba(106, 105, 104, 0.232);
    }

    /*
        `table td`is set in notebook with right text-align.
        We need to overwrite it.
    */
    .estimator-table table td.param {
        text-align: left;
        position: relative;
        padding: 0;
    }

    .user-set td {
        color:rgb(255, 94, 0);
        text-align: left !important;
    }

    .user-set td.value {
        color:rgb(255, 94, 0);
        background-color: transparent;
    }

    .default td, .estimator-table th {
        color: black;
        text-align: left !important;
    }

    .user-set td i,
    .default td i {
        color: black;
    }

    td.fitted-att-type {
        white-space: preserve nowrap;
    }

    /*
        Styles for parameter documentation links
        We need styling for visited so jupyter doesn't overwrite it
    */
    a.param-doc-link,
    a.param-doc-link:link,
    a.param-doc-link:visited {
        text-decoration: underline dashed;
        text-underline-offset: .3em;
        color: inherit;
        display: block;
        padding: .5em;
    }

    @supports(anchor-name: --doc-link) {
        a.param-doc-link,
        a.param-doc-link:link,
        a.param-doc-link:visited {
        anchor-name: --doc-link;
        }
    }

    /* "hack" to make the entire area of the cell containing the link clickable */
    a.param-doc-link::before {
        position: absolute;
        content: "";
        inset: 0;
    }

    .param-doc-description {
        display: none;
        position: absolute;
        z-index: 9999;
        left: 0;
        padding: .5ex;
        margin-left: 1.5em;
        color: var(--sklearn-color-text);
        box-shadow: .3em .3em .4em #999;
        width: max-content;
        text-align: left;
        max-height: 10em;
        overflow-y: auto;

        /* unfitted */
        background: var(--sklearn-color-unfitted-level-0);
        border: thin solid var(--sklearn-color-unfitted-level-3);
    }

    @supports(position-area: center right) {
        .param-doc-description {
        position-area: center right;
        position: fixed;
        margin-left: 0;
        }
    }

    /* Fitted state for parameter tooltips */
    .fitted .param-doc-description {
        /* fitted */
        background: var(--sklearn-color-fitted-level-0);
        border: thin solid var(--sklearn-color-fitted-level-3);
    }

    .param-doc-link:hover .param-doc-description {
        display: block;
    }

    .copy-paste-icon {
        background-image: url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCA0NDggNTEyIj48IS0tIUZvbnQgQXdlc29tZSBGcmVlIDYuNy4yIGJ5IEBmb250YXdlc29tZSAtIGh0dHBzOi8vZm9udGF3ZXNvbWUuY29tIExpY2Vuc2UgLSBodHRwczovL2ZvbnRhd2Vzb21lLmNvbS9saWNlbnNlL2ZyZWUgQ29weXJpZ2h0IDIwMjUgRm9udGljb25zLCBJbmMuLS0+PHBhdGggZD0iTTIwOCAwTDMzMi4xIDBjMTIuNyAwIDI0LjkgNS4xIDMzLjkgMTQuMWw2Ny45IDY3LjljOSA5IDE0LjEgMjEuMiAxNC4xIDMzLjlMNDQ4IDMzNmMwIDI2LjUtMjEuNSA0OC00OCA0OGwtMTkyIDBjLTI2LjUgMC00OC0yMS41LTQ4LTQ4bDAtMjg4YzAtMjYuNSAyMS41LTQ4IDQ4LTQ4ek00OCAxMjhsODAgMCAwIDY0LTY0IDAgMCAyNTYgMTkyIDAgMC0zMiA2NCAwIDAgNDhjMCAyNi41LTIxLjUgNDgtNDggNDhMNDggNTEyYy0yNi41IDAtNDgtMjEuNS00OC00OEwwIDE3NmMwLTI2LjUgMjEuNS00OCA0OC00OHoiLz48L3N2Zz4=);
        background-repeat: no-repeat;
        background-size: 14px 14px;
        background-position: 0;
        display: inline-block;
        width: 14px;
        height: 14px;
        cursor: pointer;
    }

    .features {
      font-family: monospace;
      cursor: pointer;
      background-color: var(--sklearn-color-unfitted-level-0);
      border: 1px dotted var(--sklearn-color-border-box);
      border-radius: .20em;
      margin-bottom: 0.5em;
      font-size: inherit; /* Needed for jupyter */
    }

    .features.fitted {
      background-color: var(--sklearn-color-fitted-level-0);
    }

    .features summary {
      cursor: pointer;
      display: flex;
      margin-bottom: 0;
      text-align: center;
      align-items: center;
      justify-content: center;
      gap: 0.5em;
      padding: .25em;
    }

    .features details[open] > summary {
      color: var(--sklearn-color-text);
      background-color: var(--sklearn-color-unfitted-level-2);
      border-radius: .20em 0 0 0;
    }

    .features.fitted details[open] > summary {
      background-color: var(--sklearn-color-fitted-level-2);
      border-radius: .20em 0 0 0;
    }

    .features details > summary .arrow::before {
      content: "▸";
      color: grey;
    }

    .features details[open] > summary .arrow::before {
      content: "▾";
    }

    .features details:hover > summary {
      margin: 0;
      background-color: var(--sklearn-color-unfitted-level-2);
    }

    .features.fitted details:hover > summary {
      margin: 0;
      background-color: var(--sklearn-color-fitted-level-2);
    }

    .features .features-container {
      max-width: 15em;
      max-height: 10em;
      overflow: auto;
      scrollbar-width: thin;
      padding: .25em 0.1rem;
      background-color: var(--sklearn-color-unfitted-level-0);
      border-radius: 0 0 .5em .5em;
    }

    .features.fitted .features-container {
      background-color: var(--sklearn-color-fitted-level-0);
    }

    .features .image-container {
      block-size: 1em;
      inline-size: 1em;
      padding: 0;
      margin: 0%;
      display: flex;
      justify-content: center;
      align-items: center;
    }

    .features .copy-paste-icon {
      background-size: 1em 1em;
      width: 1em;
      height: 1em;
      filter: grayscale(100%) opacity(60%);
    }

    .features .features-container table {
      width: 100%;
      margin: 0.01em;
    }

    .features .features-container table tr:nth-child(odd) {
      background-color: #fff;
    }

    .features .features-container table tr:nth-child(even) {
      background-color: #f6f6f6;
    }

    .features .features-container table tr:hover {
      background-color: #e0e0e0;
    }

    .features .features-container table {
      table-layout: inherit;
    }

    .features .features-container table td {
      text-align: left;
      padding: 0 0.5em;
      border: 1px solid rgba(106, 105, 104, 0.232);
      white-space: nowrap;
      color: var(--sklearn-color-text);
    }

    .total_features {
      display: flex;
      justify-content: center;
      margin-top: 0.5em;
    }
    </style><body><div id="sk-container-id-41" tabindex="0" class="sk-top-container sk-global"><div class="sk-text-repr-fallback"><pre>GridSearchCV(cv=RepeatedStratifiedKFold(n_repeats=10, n_splits=10, random_state=0),
                 estimator=SVC(random_state=0),
                 param_grid=[{&#x27;kernel&#x27;: [&#x27;linear&#x27;]},
                             {&#x27;degree&#x27;: [2, 3], &#x27;kernel&#x27;: [&#x27;poly&#x27;]},
                             {&#x27;kernel&#x27;: [&#x27;rbf&#x27;]}],
                 scoring=&#x27;roc_auc&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class="sk-container" hidden><div class="sk-item sk-dashed-wrapped"><div class="sk-label-container"><div class="sk-label fitted sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually sk-global" id="sk-estimator-id-140" type="checkbox" ><label for="sk-estimator-id-140" class="sk-toggleable__label fitted sk-toggleable__label-arrow"><div><div>GridSearchCV</div></div><div><a class="sk-estimator-doc-link fitted" rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html">?<span>Documentation for GridSearchCV</span></a><span class="sk-estimator-doc-link fitted">i<span>Fitted</span></span></div></label><div class="sk-toggleable__content fitted" data-param-prefix="">
            <div class="estimator-table">
                <details>
                    <summary>Parameters</summary>
                    <table class="parameters-table">
                      <tbody>
                    
            <tr class="user-set">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('estimator',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-estimator;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=estimator,-estimator%20object">
                estimator
                <span class="param-doc-description"
                style="position-anchor: --doc-link-estimator;">
                estimator: estimator object<br><br>This is assumed to implement the scikit-learn estimator interface.<br>Either estimator needs to provide a ``score`` function,<br>or ``scoring`` must be passed.</span>
            </a>
        </td>
                <td class="value">SVC(random_state=0)</td>
            </tr>
    

            <tr class="user-set">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('param_grid',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-param_grid;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=param_grid,-dict%20or%20list%20of%20dictionaries">
                param_grid
                <span class="param-doc-description"
                style="position-anchor: --doc-link-param_grid;">
                param_grid: dict or list of dictionaries<br><br>Dictionary with parameters names (`str`) as keys and lists of<br>parameter settings to try as values, or a list of such<br>dictionaries, in which case the grids spanned by each dictionary<br>in the list are explored. This enables searching over any sequence<br>of parameter settings.</span>
            </a>
        </td>
                <td class="value">[{&#x27;kernel&#x27;: [&#x27;linear&#x27;]}, {&#x27;degree&#x27;: [2, 3], &#x27;kernel&#x27;: [&#x27;poly&#x27;]}, ...]</td>
            </tr>
    

            <tr class="user-set">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('scoring',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-scoring;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=scoring,-str%2C%20callable%2C%20list%2C%20tuple%20or%20dict%2C%20default%3DNone">
                scoring
                <span class="param-doc-description"
                style="position-anchor: --doc-link-scoring;">
                scoring: str, callable, list, tuple or dict, default=None<br><br>Strategy to evaluate the performance of the cross-validated model on<br>the test set.<br><br>If `scoring` represents a single score, one can use:<br><br>- a single string (see :ref:`scoring_string_names`);<br>- a callable (see :ref:`scoring_callable`) that returns a single value;<br>- `None`, the `estimator`&#x27;s<br>  :ref:`default evaluation criterion &lt;scoring_api_overview&gt;` is used.<br><br>If `scoring` represents multiple scores, one can use:<br><br>- a list or tuple of unique strings;<br>- a callable returning a dictionary where the keys are the metric<br>  names and the values are the metric scores;<br>- a dictionary with metric names as keys and callables as values.<br><br>See :ref:`multimetric_grid_search` for an example.</span>
            </a>
        </td>
                <td class="value">&#x27;roc_auc&#x27;</td>
            </tr>
    

            <tr class="user-set">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('cv',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-cv;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=cv,-int%2C%20cross-validation%20generator%20or%20an%20iterable%2C%20default%3DNone">
                cv
                <span class="param-doc-description"
                style="position-anchor: --doc-link-cv;">
                cv: int, cross-validation generator or an iterable, default=None<br><br>Determines the cross-validation splitting strategy.<br>Possible inputs for cv are:<br><br>- None, to use the default 5-fold cross validation,<br>- integer, to specify the number of folds in a `(Stratified)KFold`,<br>- :term:`CV splitter`,<br>- an iterable yielding (train, test) splits as arrays of indices.<br><br>For integer/None inputs, if the estimator is a classifier and ``y`` is<br>either binary or multiclass, :class:`StratifiedKFold` is used. In all<br>other cases, :class:`KFold` is used. These splitters are instantiated<br>with `shuffle=False` so the splits will be the same across calls.<br><br>Refer :ref:`User Guide &lt;cross_validation&gt;` for the various<br>cross-validation strategies that can be used here.<br><br>.. versionchanged:: 0.22<br>    ``cv`` default value if None changed from 3-fold to 5-fold.</span>
            </a>
        </td>
                <td class="value">RepeatedStrat...andom_state=0)</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('n_jobs',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-n_jobs;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=n_jobs,-int%2C%20default%3DNone">
                n_jobs
                <span class="param-doc-description"
                style="position-anchor: --doc-link-n_jobs;">
                n_jobs: int, default=None<br><br>Number of jobs to run in parallel.<br>``None`` means 1 unless in a :obj:`joblib.parallel_backend` context.<br>``-1`` means using all processors. See :term:`Glossary &lt;n_jobs&gt;`<br>for more details.<br><br>.. versionchanged:: v0.20<br>   `n_jobs` default changed from 1 to None</span>
            </a>
        </td>
                <td class="value">None</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('refit',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-refit;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=refit,-bool%2C%20str%2C%20or%20callable%2C%20default%3DTrue">
                refit
                <span class="param-doc-description"
                style="position-anchor: --doc-link-refit;">
                refit: bool, str, or callable, default=True<br><br>Refit an estimator using the best found parameters on the whole<br>dataset.<br><br>For multiple metric evaluation, this needs to be a `str` denoting the<br>scorer that would be used to find the best parameters for refitting<br>the estimator at the end.<br><br>Where there are considerations other than maximum score in<br>choosing a best estimator, ``refit`` can be set to a function which<br>returns the selected ``best_index_`` given ``cv_results_``. In that<br>case, the ``best_estimator_`` and ``best_params_`` will be set<br>according to the returned ``best_index_`` while the ``best_score_``<br>attribute will not be available.<br><br>The refitted estimator is made available at the ``best_estimator_``<br>attribute and permits using ``predict`` directly on this<br>``GridSearchCV`` instance.<br><br>Also for multiple metric evaluation, the attributes ``best_index_``,<br>``best_score_`` and ``best_params_`` will only be available if<br>``refit`` is set and all of them will be determined w.r.t this specific<br>scorer.<br><br>See ``scoring`` parameter to know more about multiple metric<br>evaluation.<br><br>See :ref:`sphx_glr_auto_examples_model_selection_plot_grid_search_digits.py`<br>to see how to design a custom selection strategy using a callable<br>via `refit`.<br><br>See :ref:`this example<br>&lt;sphx_glr_auto_examples_model_selection_plot_grid_search_refit_callable.py&gt;`<br>for an example of how to use ``refit=callable`` to balance model<br>complexity and cross-validated score.<br><br>.. versionchanged:: 0.20<br>    Support for callable added.</span>
            </a>
        </td>
                <td class="value">True</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('verbose',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-verbose;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=verbose,-int%2C%20default%3D0">
                verbose
                <span class="param-doc-description"
                style="position-anchor: --doc-link-verbose;">
                verbose: int, default=0<br><br>Controls the verbosity of information printed during fitting, with higher<br>values yielding more detailed logging.<br><br>- 0 : no messages are printed;<br>- &gt;=1 : summary of the total number of fits;<br>- &gt;=2 : computation time for each fold and parameter candidate;<br>- &gt;=3 : fold indices and scores;<br>- &gt;=10 : parameter candidate indices and START messages before each fit.</span>
            </a>
        </td>
                <td class="value">0</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('pre_dispatch',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-pre_dispatch;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=pre_dispatch,-int%2C%20or%20str%2C%20default%3D%272%2An_jobs%27">
                pre_dispatch
                <span class="param-doc-description"
                style="position-anchor: --doc-link-pre_dispatch;">
                pre_dispatch: int, or str, default=&#x27;2*n_jobs&#x27;<br><br>Controls the number of jobs that get dispatched during parallel<br>execution. Reducing this number can be useful to avoid an<br>explosion of memory consumption when more jobs get dispatched<br>than CPUs can process. This parameter can be:<br><br>- None, in which case all the jobs are immediately created and spawned. Use<br>  this for lightweight and fast-running jobs, to avoid delays due to on-demand<br>  spawning of the jobs<br>- An int, giving the exact number of total jobs that are spawned<br>- A str, giving an expression as a function of n_jobs, as in &#x27;2*n_jobs&#x27;</span>
            </a>
        </td>
                <td class="value">&#x27;2*n_jobs&#x27;</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('error_score',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-error_score;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=error_score,-%27raise%27%20or%20numeric%2C%20default%3Dnp.nan">
                error_score
                <span class="param-doc-description"
                style="position-anchor: --doc-link-error_score;">
                error_score: &#x27;raise&#x27; or numeric, default=np.nan<br><br>Value to assign to the score if an error occurs in estimator fitting.<br>If set to &#x27;raise&#x27;, the error is raised. If a numeric value is given,<br>FitFailedWarning is raised. This parameter does not affect the refit<br>step, which will always raise the error.</span>
            </a>
        </td>
                <td class="value">nan</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('return_train_score',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-return_train_score;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=return_train_score,-bool%2C%20default%3DFalse">
                return_train_score
                <span class="param-doc-description"
                style="position-anchor: --doc-link-return_train_score;">
                return_train_score: bool, default=False<br><br>If ``False``, the ``cv_results_`` attribute will not include training<br>scores.<br>Computing training scores is used to get insights on how different<br>parameter settings impact the overfitting/underfitting trade-off.<br>However computing the scores on the training set can be computationally<br>expensive and is not strictly required to select the parameters that<br>yield the best generalization performance.<br><br>.. versionadded:: 0.19<br><br>.. versionchanged:: 0.21<br>    Default value was changed from ``True`` to ``False``</span>
            </a>
        </td>
                <td class="value">False</td>
            </tr>
    
                      </tbody>
                    </table>
                </details>
            </div>
    
            <div class="estimator-table">
                <details>
                    <summary>Fitted attributes</summary>
                    <table class="parameters-table">
                        <tbody>
                            <tr>
                            <th>Name</th>
                            <th>Type</th>
                            <th>Value</th>
                            </tr>
                        
           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-best_estimator_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=best_estimator_,-estimator">
                best_estimator_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-best_estimator_;">
                best_estimator_: estimator<br><br>Estimator that was chosen by the search, i.e. estimator<br>which gave highest score (or smallest loss if specified)<br>on the left out data. Not available if ``refit=False``.<br><br>See ``refit`` parameter for more information on allowed values.</span>
            </a>
        </td>
               <td class="fitted-att-type">SVC</td>
               <td>SVC(random_state=0)</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-best_index_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=best_index_,-int">
                best_index_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-best_index_;">
                best_index_: int<br><br>The index (of the ``cv_results_`` arrays) which corresponds to the best<br>candidate parameter setting.<br><br>The dict at ``search.cv_results_[&#x27;params&#x27;][search.best_index_]`` gives<br>the parameter setting for the best model, that gives the highest<br>mean score (``search.best_score_``).<br><br>For multi-metric evaluation, this is present only if ``refit`` is<br>specified.</span>
            </a>
        </td>
               <td class="fitted-att-type">int64</td>
               <td>np.int64(3)</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-best_params_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=best_params_,-dict">
                best_params_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-best_params_;">
                best_params_: dict<br><br>Parameter setting that gave the best results on the hold out data.<br><br>For multi-metric evaluation, this is present only if ``refit`` is<br>specified.</span>
            </a>
        </td>
               <td class="fitted-att-type">dict</td>
               <td>{&#x27;kernel&#x27;: &#x27;rbf&#x27;}</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-best_score_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=best_score_,-float">
                best_score_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-best_score_;">
                best_score_: float<br><br>Mean cross-validated score of the best_estimator<br><br>For multi-metric evaluation, this is present only if ``refit`` is<br>specified.<br><br>This attribute is not available if ``refit`` is a function.</span>
            </a>
        </td>
               <td class="fitted-att-type">float64</td>
               <td>0.94</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-classes_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=classes_,-ndarray%20of%20shape%20%28n_classes%2C%29">
                classes_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-classes_;">
                classes_: ndarray of shape (n_classes,)<br><br>The classes labels. This is present only if ``refit`` is specified and<br>the underlying estimator is a classifier.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[int64](2,)</td>
               <td>[0,1]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-cv_results_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=cv_results_,-dict%20of%20numpy%20%28masked%29%20ndarrays">
                cv_results_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-cv_results_;">
                cv_results_: dict of numpy (masked) ndarrays<br><br>A dict with keys as column headers and values as columns, that can be<br>imported into a pandas ``DataFrame``.<br><br>For instance the below given table<br><br>+------------+-----------+------------+-----------------+---+---------+<br>|param_kernel|param_gamma|param_degree|split0_test_score|...|rank_t...|<br>+============+===========+============+=================+===+=========+<br>|  &#x27;poly&#x27;    |     --    |      2     |       0.80      |...|    2    |<br>+------------+-----------+------------+-----------------+---+---------+<br>|  &#x27;poly&#x27;    |     --    |      3     |       0.70      |...|    4    |<br>+------------+-----------+------------+-----------------+---+---------+<br>|  &#x27;rbf&#x27;     |     0.1   |     --     |       0.80      |...|    3    |<br>+------------+-----------+------------+-----------------+---+---------+<br>|  &#x27;rbf&#x27;     |     0.2   |     --     |       0.93      |...|    1    |<br>+------------+-----------+------------+-----------------+---+---------+<br><br>will be represented by a ``cv_results_`` dict of::<br><br>    {<br>    &#x27;param_kernel&#x27;: masked_array(data = [&#x27;poly&#x27;, &#x27;poly&#x27;, &#x27;rbf&#x27;, &#x27;rbf&#x27;],<br>                                 mask = [False False False False]...)<br>    &#x27;param_gamma&#x27;: masked_array(data = [-- -- 0.1 0.2],<br>                                mask = [ True  True False False]...),<br>    &#x27;param_degree&#x27;: masked_array(data = [2.0 3.0 -- --],<br>                                 mask = [False False  True  True]...),<br>    &#x27;split0_test_score&#x27;  : [0.80, 0.70, 0.80, 0.93],<br>    &#x27;split1_test_score&#x27;  : [0.82, 0.50, 0.70, 0.78],<br>    &#x27;mean_test_score&#x27;    : [0.81, 0.60, 0.75, 0.85],<br>    &#x27;std_test_score&#x27;     : [0.01, 0.10, 0.05, 0.08],<br>    &#x27;rank_test_score&#x27;    : [2, 4, 3, 1],<br>    &#x27;split0_train_score&#x27; : [0.80, 0.92, 0.70, 0.93],<br>    &#x27;split1_train_score&#x27; : [0.82, 0.55, 0.70, 0.87],<br>    &#x27;mean_train_score&#x27;   : [0.81, 0.74, 0.70, 0.90],<br>    &#x27;std_train_score&#x27;    : [0.01, 0.19, 0.00, 0.03],<br>    &#x27;mean_fit_time&#x27;      : [0.73, 0.63, 0.43, 0.49],<br>    &#x27;std_fit_time&#x27;       : [0.01, 0.02, 0.01, 0.01],<br>    &#x27;mean_score_time&#x27;    : [0.01, 0.06, 0.04, 0.04],<br>    &#x27;std_score_time&#x27;     : [0.00, 0.00, 0.00, 0.01],<br>    &#x27;params&#x27;             : [{&#x27;kernel&#x27;: &#x27;poly&#x27;, &#x27;degree&#x27;: 2}, ...],<br>    }<br><br>For an example of visualization and interpretation of GridSearch results,<br>see :ref:`sphx_glr_auto_examples_model_selection_plot_grid_search_stats.py`.<br><br>NOTE<br><br>The key ``&#x27;params&#x27;`` is used to store a list of parameter<br>settings dicts for all the parameter candidates.<br><br>The ``mean_fit_time``, ``std_fit_time``, ``mean_score_time`` and<br>``std_score_time`` are all in seconds.<br><br>For multi-metric evaluation, the scores for all the scorers are<br>available in the ``cv_results_`` dict at the keys ending with that<br>scorer&#x27;s name (``&#x27;_&lt;scorer_name&gt;&#x27;``) instead of ``&#x27;_score&#x27;`` shown<br>above. (&#x27;split0_test_precision&#x27;, &#x27;mean_train_precision&#x27; etc.)</span>
            </a>
        </td>
               <td class="fitted-att-type">dict</td>
               <td>{&#x27;me...me&#x27;: array([0., 0., 0., 0.]), &#x27;me...me&#x27;: array([0., 0., 0., 0.]), &#x27;me...re&#x27;: array([0.93, ..., 0.9 , 0.94]), &#x27;pa...ee&#x27;: masked_array(..._value=999999), ...}</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-multimetric_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=multimetric_,-bool">
                multimetric_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-multimetric_;">
                multimetric_: bool<br><br>Whether or not the scorers compute several metrics.</span>
            </a>
        </td>
               <td class="fitted-att-type">bool</td>
               <td>False</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-n_features_in_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=n_features_in_,-int">
                n_features_in_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-n_features_in_;">
                n_features_in_: int<br><br>Number of features seen during :term:`fit`. Only defined if<br>`best_estimator_` is defined (see the documentation for the `refit`<br>parameter for more details) and that `best_estimator_` exposes<br>`n_features_in_` when fit.<br><br>.. versionadded:: 0.24</span>
            </a>
        </td>
               <td class="fitted-att-type">int</td>
               <td>2</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-n_splits_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=n_splits_,-int">
                n_splits_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-n_splits_;">
                n_splits_: int<br><br>The number of cross-validation splits (folds/iterations).</span>
            </a>
        </td>
               <td class="fitted-att-type">int</td>
               <td>100</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-refit_time_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=refit_time_,-float">
                refit_time_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-refit_time_;">
                refit_time_: float<br><br>Seconds used for refitting the best model on the whole dataset.<br><br>This is present only if ``refit`` is not False.<br><br>.. versionadded:: 0.20</span>
            </a>
        </td>
               <td class="fitted-att-type">float</td>
               <td>0.00192</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-scorer_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.model_selection.GridSearchCV.html#:~:text=scorer_,-function%20or%20a%20dict">
                scorer_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-scorer_;">
                scorer_: function or a dict<br><br>Scorer function used on the held out data to choose the best<br>parameters for the model.<br><br>For multi-metric evaluation, this attribute holds the validated<br>``scoring`` dict which maps the scorer key to the scorer callable.</span>
            </a>
        </td>
               <td class="fitted-att-type">_Scorer</td>
               <td>make_scorer(r...edict_proba&#x27;))</td>


           </tr>
    
                        </tbody>
                    </table>
                </details>
            </div>
        </div></div></div><div class="sk-parallel"><div class="sk-parallel-item"><div class="sk-item"><div class="sk-label-container"><div class="sk-label fitted sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually sk-global" id="sk-estimator-id-141" type="checkbox" ><label for="sk-estimator-id-141" class="sk-toggleable__label fitted sk-toggleable__label-arrow"><div><div>best_estimator_: SVC</div></div></label><div class="sk-toggleable__content fitted" data-param-prefix="best_estimator___"><pre>SVC(random_state=0)</pre></div></div></div><div class="sk-serial"><div class="sk-item"><div class="sk-estimator fitted sk-toggleable"><input class="sk-toggleable__control sk-hidden--visually sk-global" id="sk-estimator-id-142" type="checkbox" ><label for="sk-estimator-id-142" class="sk-toggleable__label fitted sk-toggleable__label-arrow"><div><div>SVC</div></div><div><a class="sk-estimator-doc-link fitted" rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html">?<span>Documentation for SVC</span></a></div></label><div class="sk-toggleable__content fitted" data-param-prefix="best_estimator___">
            <div class="estimator-table">
                <details>
                    <summary>Parameters</summary>
                    <table class="parameters-table">
                      <tbody>
                    
            <tr class="user-set">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('random_state',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-random_state;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=random_state,-int%2C%20RandomState%20instance%20or%20None%2C%20default%3DNone">
                random_state
                <span class="param-doc-description"
                style="position-anchor: --doc-link-random_state;">
                random_state: int, RandomState instance or None, default=None<br><br>Controls the pseudo random number generation for shuffling the data for<br>probability estimates. Ignored when `probability` is False.<br>Pass an int for reproducible output across multiple function calls.<br>See :term:`Glossary &lt;random_state&gt;`.</span>
            </a>
        </td>
                <td class="value">0</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('C',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-C;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=C,-float%2C%20default%3D1.0">
                C
                <span class="param-doc-description"
                style="position-anchor: --doc-link-C;">
                C: float, default=1.0<br><br>Regularization parameter. The strength of the regularization is<br>inversely proportional to C. Must be strictly positive. The penalty<br>is a squared l2 penalty. For an intuitive visualization of the effects<br>of scaling the regularization parameter C, see<br>:ref:`sphx_glr_auto_examples_svm_plot_svm_scale_c.py`.</span>
            </a>
        </td>
                <td class="value">1.0</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('kernel',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-kernel;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=kernel,-%7B%27linear%27%2C%20%27poly%27%2C%20%27rbf%27%2C%20%27sigmoid%27%2C%20%27precomputed%27%7D%20or%20callable%2C%20%20%20%20%20%20%20%20%20%20default%3D%27rbf%27">
                kernel
                <span class="param-doc-description"
                style="position-anchor: --doc-link-kernel;">
                kernel: {&#x27;linear&#x27;, &#x27;poly&#x27;, &#x27;rbf&#x27;, &#x27;sigmoid&#x27;, &#x27;precomputed&#x27;} or callable,          default=&#x27;rbf&#x27;<br><br>Specifies the kernel type to be used in the algorithm. If<br>none is given, &#x27;rbf&#x27; will be used. If a callable is given it is used to<br>pre-compute the kernel matrix from data matrices; that matrix should be<br>an array of shape ``(n_samples, n_samples)``. For an intuitive<br>visualization of different kernel types see<br>:ref:`sphx_glr_auto_examples_svm_plot_svm_kernels.py`.</span>
            </a>
        </td>
                <td class="value">&#x27;rbf&#x27;</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('degree',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-degree;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=degree,-int%2C%20default%3D3">
                degree
                <span class="param-doc-description"
                style="position-anchor: --doc-link-degree;">
                degree: int, default=3<br><br>Degree of the polynomial kernel function (&#x27;poly&#x27;).<br>Must be non-negative. Ignored by all other kernels.</span>
            </a>
        </td>
                <td class="value">3</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('gamma',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-gamma;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=gamma,-%7B%27scale%27%2C%20%27auto%27%7D%20or%20float%2C%20default%3D%27scale%27">
                gamma
                <span class="param-doc-description"
                style="position-anchor: --doc-link-gamma;">
                gamma: {&#x27;scale&#x27;, &#x27;auto&#x27;} or float, default=&#x27;scale&#x27;<br><br>Kernel coefficient for &#x27;rbf&#x27;, &#x27;poly&#x27; and &#x27;sigmoid&#x27;.<br><br>- if ``gamma=&#x27;scale&#x27;`` (default) is passed then it uses<br>  1 / (n_features * X.var()) as value of gamma,<br>- if &#x27;auto&#x27;, uses 1 / n_features<br>- if float, must be non-negative.<br><br>.. versionchanged:: 0.22<br>   The default value of ``gamma`` changed from &#x27;auto&#x27; to &#x27;scale&#x27;.</span>
            </a>
        </td>
                <td class="value">&#x27;scale&#x27;</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('coef0',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-coef0;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=coef0,-float%2C%20default%3D0.0">
                coef0
                <span class="param-doc-description"
                style="position-anchor: --doc-link-coef0;">
                coef0: float, default=0.0<br><br>Independent term in kernel function.<br>It is only significant in &#x27;poly&#x27; and &#x27;sigmoid&#x27;.</span>
            </a>
        </td>
                <td class="value">0.0</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('shrinking',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-shrinking;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=shrinking,-bool%2C%20default%3DTrue">
                shrinking
                <span class="param-doc-description"
                style="position-anchor: --doc-link-shrinking;">
                shrinking: bool, default=True<br><br>Whether to use the shrinking heuristic.<br>See the :ref:`User Guide &lt;shrinking_svm&gt;`.</span>
            </a>
        </td>
                <td class="value">True</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('probability',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-probability;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=probability,-bool%2C%20default%3DFalse">
                probability
                <span class="param-doc-description"
                style="position-anchor: --doc-link-probability;">
                probability: bool, default=False<br><br>Whether to enable probability estimates. This must be enabled prior<br>to calling `fit`, will slow down that method as it internally uses<br>5-fold cross-validation, and `predict_proba` may be inconsistent with<br>`predict`. Read more in the :ref:`User Guide &lt;scores_probabilities&gt;`.<br><br>..deprecated:: 1.9<br>  The `probability` parameter is deprecated and will be removed in 1.11.<br>  Use `CalibratedClassifierCV(SVC(), ensemble=False)` instead of<br>  `SVC(probability=True)`.</span>
            </a>
        </td>
                <td class="value">&#x27;deprecated&#x27;</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('tol',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-tol;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=tol,-float%2C%20default%3D1e-3">
                tol
                <span class="param-doc-description"
                style="position-anchor: --doc-link-tol;">
                tol: float, default=1e-3<br><br>Tolerance for stopping criterion.</span>
            </a>
        </td>
                <td class="value">0.001</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('cache_size',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-cache_size;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=cache_size,-float%2C%20default%3D200">
                cache_size
                <span class="param-doc-description"
                style="position-anchor: --doc-link-cache_size;">
                cache_size: float, default=200<br><br>Specify the size of the kernel cache (in MB).</span>
            </a>
        </td>
                <td class="value">200</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('class_weight',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-class_weight;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=class_weight,-dict%20or%20%27balanced%27%2C%20default%3DNone">
                class_weight
                <span class="param-doc-description"
                style="position-anchor: --doc-link-class_weight;">
                class_weight: dict or &#x27;balanced&#x27;, default=None<br><br>Set the parameter C of class i to class_weight[i]*C for<br>SVC. If not given, all classes are supposed to have<br>weight one.<br>The &quot;balanced&quot; mode uses the values of y to automatically adjust<br>weights inversely proportional to class frequencies in the input data<br>as ``n_samples / (n_classes * np.bincount(y))``.</span>
            </a>
        </td>
                <td class="value">None</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('verbose',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-verbose;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=verbose,-bool%2C%20default%3DFalse">
                verbose
                <span class="param-doc-description"
                style="position-anchor: --doc-link-verbose;">
                verbose: bool, default=False<br><br>Enable verbose output. Note that this setting takes advantage of a<br>per-process runtime setting in libsvm that, if enabled, may not work<br>properly in a multithreaded context.</span>
            </a>
        </td>
                <td class="value">False</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('max_iter',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-max_iter;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=max_iter,-int%2C%20default%3D-1">
                max_iter
                <span class="param-doc-description"
                style="position-anchor: --doc-link-max_iter;">
                max_iter: int, default=-1<br><br>Hard limit on iterations within solver, or -1 for no limit.</span>
            </a>
        </td>
                <td class="value">-1</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('decision_function_shape',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-decision_function_shape;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=decision_function_shape,-%7B%27ovo%27%2C%20%27ovr%27%7D%2C%20default%3D%27ovr%27">
                decision_function_shape
                <span class="param-doc-description"
                style="position-anchor: --doc-link-decision_function_shape;">
                decision_function_shape: {&#x27;ovo&#x27;, &#x27;ovr&#x27;}, default=&#x27;ovr&#x27;<br><br>Whether to return a one-vs-rest (&#x27;ovr&#x27;) decision function of shape<br>(n_samples, n_classes) as all other classifiers, or the original<br>one-vs-one (&#x27;ovo&#x27;) decision function of libsvm which has shape<br>(n_samples, n_classes * (n_classes - 1) / 2). However, note that<br>internally, one-vs-one (&#x27;ovo&#x27;) is always used as a multi-class strategy<br>to train models; an ovr matrix is only constructed from the ovo matrix.<br>The parameter is ignored for binary classification.<br><br>.. versionchanged:: 0.19<br>    decision_function_shape is &#x27;ovr&#x27; by default.<br><br>.. versionadded:: 0.17<br>   *decision_function_shape=&#x27;ovr&#x27;* is recommended.<br><br>.. versionchanged:: 0.17<br>   Deprecated *decision_function_shape=&#x27;ovo&#x27; and None*.</span>
            </a>
        </td>
                <td class="value">&#x27;ovr&#x27;</td>
            </tr>
    

            <tr class="default">
                <td><i class="copy-paste-icon"
                     onclick="copyToClipboard('break_ties',
                              this.parentElement.nextElementSibling)"
                ></i></td>
                <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-break_ties;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=break_ties,-bool%2C%20default%3DFalse">
                break_ties
                <span class="param-doc-description"
                style="position-anchor: --doc-link-break_ties;">
                break_ties: bool, default=False<br><br>If true, ``decision_function_shape=&#x27;ovr&#x27;``, and number of classes &gt; 2,<br>:term:`predict` will break ties according to the confidence values of<br>:term:`decision_function`; otherwise the first class among the tied<br>classes is returned. Please note that breaking ties comes at a<br>relatively high computational cost compared to a simple predict. See<br>:ref:`sphx_glr_auto_examples_svm_plot_svm_tie_breaking.py` for an<br>example of its usage with ``decision_function_shape=&#x27;ovr&#x27;``.<br><br>.. versionadded:: 0.22</span>
            </a>
        </td>
                <td class="value">False</td>
            </tr>
    
                      </tbody>
                    </table>
                </details>
            </div>
    
            <div class="estimator-table">
                <details>
                    <summary>Fitted attributes</summary>
                    <table class="parameters-table">
                        <tbody>
                            <tr>
                            <th>Name</th>
                            <th>Type</th>
                            <th>Value</th>
                            </tr>
                        
           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-class_weight_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=class_weight_,-ndarray%20of%20shape%20%28n_classes%2C%29">
                class_weight_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-class_weight_;">
                class_weight_: ndarray of shape (n_classes,)<br><br>Multipliers of parameter C for each class.<br>Computed based on the ``class_weight`` parameter.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[float64](2,)</td>
               <td>[1.,1.]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-classes_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=classes_,-ndarray%20of%20shape%20%28n_classes%2C%29">
                classes_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-classes_;">
                classes_: ndarray of shape (n_classes,)<br><br>The classes labels.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[int64](2,)</td>
               <td>[0,1]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-dual_coef_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=dual_coef_,-ndarray%20or%20sparse%20array/matrix%20of%20shape%20%28n_classes%20-1%2C%20n_SV%29">
                dual_coef_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-dual_coef_;">
                dual_coef_: ndarray or sparse array/matrix of shape (n_classes -1, n_SV)<br><br>Dual coefficients of the support vector in the decision<br>function (see :ref:`sgd_mathematical_formulation`), multiplied by<br>their targets.<br>For multiclass, coefficient for all 1-vs-1 classifiers.<br>The layout of the coefficients in the multiclass case is somewhat<br>non-trivial. See the :ref:`multi-class section of the User Guide<br>&lt;svm_multi_class&gt;` for details.<br>If `X` is sparse, these will also be sparse.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[float64](1, 48)</td>
               <td>[[-1.  ,-0.09,-1.  ,..., 1.  , 1.  , 0.19]]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-fit_status_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=fit_status_,-int">
                fit_status_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-fit_status_;">
                fit_status_: int<br><br>0 if correctly fitted, 1 otherwise (will raise warning)</span>
            </a>
        </td>
               <td class="fitted-att-type">int</td>
               <td>0</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-intercept_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=intercept_,-ndarray%20of%20shape%20%28n_classes%20%2A%20%28n_classes%20-%201%29%20/%202%2C%29">
                intercept_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-intercept_;">
                intercept_: ndarray of shape (n_classes * (n_classes - 1) / 2,)<br><br>Constants in decision function.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[float64](1,)</td>
               <td>[0.01]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-n_features_in_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=n_features_in_,-int">
                n_features_in_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-n_features_in_;">
                n_features_in_: int<br><br>Number of features seen during :term:`fit`.<br><br>.. versionadded:: 0.24</span>
            </a>
        </td>
               <td class="fitted-att-type">int</td>
               <td>2</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-n_iter_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=n_iter_,-ndarray%20of%20shape%20%28n_classes%20%2A%20%28n_classes%20-%201%29%20//%202%2C%29">
                n_iter_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-n_iter_;">
                n_iter_: ndarray of shape (n_classes * (n_classes - 1) // 2,)<br><br>Number of iterations run by the optimization routine to fit the model.<br>The shape of this attribute depends on the number of models optimized<br>which in turn depends on the number of classes.<br><br>.. versionadded:: 1.1</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[int32](1,)</td>
               <td>[47]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-n_support_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=n_support_,-ndarray%20of%20shape%20%28n_classes%2C%29%2C%20dtype%3Dint32">
                n_support_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-n_support_;">
                n_support_: ndarray of shape (n_classes,), dtype=int32<br><br>Number of support vectors for each class.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[int32](2,)</td>
               <td>[24,24]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-probA_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=probA_,-ndarray%20of%20shape%20%28n_classes%20%2A%20%28n_classes%20-%201%29%20/%202%29">
                probA_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-probA_;">
                probA_: ndarray of shape (n_classes * (n_classes - 1) / 2)<br><br>If `probability=True`, it corresponds to the parameters learned in<br>Platt scaling to produce probability estimates from decision values.<br>If `probability=False`, it&#x27;s an empty array. Platt scaling uses the<br>logistic function</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[float64](0,)</td>
               <td>[]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-probB_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=probB_,-ndarray%20of%20shape%20%28n_classes%20%2A%20%28n_classes%20-%201%29%20/%202%29">
                probB_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-probB_;">
                probB_: ndarray of shape (n_classes * (n_classes - 1) / 2)<br><br>If `probability=True`, it corresponds to the parameters learned in<br>Platt scaling. Platt scaling uses the logistic function<br>``1 / (1 + exp(decision_value * probA_ + probB_))``<br>where ``probA_`` and ``probB_`` are learned from the dataset [2]_. For<br>more information on the multiclass case and training procedure see<br>section 8 of [1]_.<br><br>.. deprecated:: 1.9<br>    The attributes `probA_` and `probB_` are deprecated in version 1.9 and will<br>    be removed in 1.11.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[float64](0,)</td>
               <td>[]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-shape_fit_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=shape_fit_,-tuple%20of%20int%20of%20shape%20%28n_dimensions_of_X%2C%29">
                shape_fit_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-shape_fit_;">
                shape_fit_: tuple of int of shape (n_dimensions_of_X,)<br><br>Array dimensions of training vector ``X``.</span>
            </a>
        </td>
               <td class="fitted-att-type">tuple</td>
               <td>(100, 2)</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-support_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=support_,-ndarray%20of%20shape%20%28n_SV%29">
                support_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-support_;">
                support_: ndarray of shape (n_SV)<br><br>Indices of support vectors.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[int32](48,)</td>
               <td>[ 2, 6,20,...,89,93,97]</td>


           </tr>
    

           <tr class="default">
               <td class="param">
            <a class="param-doc-link"
                style="anchor-name: --doc-link-support_vectors_;"
                rel="noreferrer" target="_blank" href="https://scikit-learn.org/1.9/modules/generated/sklearn.svm.SVC.html#:~:text=support_vectors_,-ndarray%20or%20sparse%20array/matrix%20of%20shape%20%28n_SV%2C%20n_features%29">
                support_vectors_
                <span class="param-doc-description"
                style="position-anchor: --doc-link-support_vectors_;">
                support_vectors_: ndarray or sparse array/matrix of shape (n_SV, n_features)<br><br>Support vectors. An empty array if kernel is precomputed.<br>If `X` is sparse, these will also be sparse.</span>
            </a>
        </td>
               <td class="fitted-att-type">ndarray[float64](48, 2)</td>
               <td>[[-0.27, 0.07],
     [-1.14, 1.21],
     [ 0.45, 0.06],
     ...,
     [ 1.36,-0.15],
     [ 1.26,-0.14],
     [ 0.52,-1.45]]</td>


           </tr>
    
                        </tbody>
                    </table>
                </details>
            </div>
        </div></div></div></div></div></div></div></div></div></div><script>/*  Authors: The scikit-learn developers
     SPDX-License-Identifier: BSD-3-Clause
    */

    function copyToClipboard(text, element) {
        // Get the parameter prefix from the closest toggleable content
        const toggleableContent = element.closest('.sk-toggleable__content');
        const paramPrefix = toggleableContent ? toggleableContent.dataset.paramPrefix : '';
        const fullParamName = paramPrefix ? `${paramPrefix}${text}` : text;

        const originalStyle = element.style;
        const computedStyle = window.getComputedStyle(element);
        const originalWidth = computedStyle.width;
        const originalHTML = element.innerHTML.replace('Copied!', '');

        navigator.clipboard.writeText(fullParamName)
            .then(() => {
                element.style.width = originalWidth;
                element.style.color = 'green';
                element.innerHTML = "Copied!";

                setTimeout(() => {
                    element.innerHTML = originalHTML;
                    element.style = originalStyle;
                }, 2000);
            })
            .catch(err => {
                console.error('Failed to copy:', err);
                element.style.color = 'red';
                element.innerHTML = "Failed!";
                setTimeout(() => {
                    element.innerHTML = originalHTML;
                    element.style = originalStyle;
                }, 2000);
            });
        return false;
    }

    document.querySelectorAll('.copy-paste-icon').forEach(function(element) {
        const toggleableContent = element.closest('.sk-toggleable__content');
        const paramPrefix = toggleableContent ? toggleableContent.dataset.paramPrefix : '';

        const parent = element.parentElement;
        if (!parent || !parent.nextElementSibling) {
            console.warn('Expected copy-paste icon is missing from the DOM structure');
            return;
        }

        const paramName = element.parentElement.nextElementSibling
            .textContent.trim().split(' ')[0];
        const fullParamName = paramPrefix ? `${paramPrefix}${paramName}` : paramName;

        element.setAttribute('title', fullParamName);
    });

    /**
     * Copy the list of feature names formatted as a Python list.
     *
     * @param {HTMLElement} element - The copy button inside a `.features` block; its siblings
     *   contain a `details` element and a table containing feature named.
     * @returns {boolean} Always returns `false` so callers can prevent the default click behavior.
     */
    function copyFeatureNamesToClipboard(element) {
        var detailsElem = element.closest('.features').querySelector('details');
        var wasOpen = detailsElem.open;
        detailsElem.open = true;
        var content = element.closest('.features').querySelector('tbody')
                      .innerText.trim();
        if (!wasOpen) detailsElem.open = false;
        const rows = content.split('\n').map(row => `    "${row}"`);
        const formattedText = `[\n${rows.join(',\n')},\n]`;
        const originalHTML = element.innerHTML.replace('✔', '');
        const originalStyle = element.style;
        const copyMark = document.createElement('span');
        copyMark.innerHTML = '✔';
        copyMark.style.color = 'blue';
        copyMark.style.fontSize = '1em';

        navigator.clipboard.writeText(formattedText)
            .then(() => {
                element.style.display = 'none';
                element.parentElement.appendChild(copyMark);

                setTimeout(() => {
                    copyMark.remove();
                    element.innerHTML = originalHTML;
                    element.style = originalStyle;
                }, 1000);
            })
            .catch(err => {
                console.error('Failed to copy:', err);
                element.style.color = 'orange';
                element.innerHTML = "Failed!";
                setTimeout(() => {
                    element.innerHTML = originalHTML;
                    element.style = originalStyle;
                }, 1000);
            });
        return false;
    }
    /**
     * Adapted from Skrub
     * https://github.com/skrub-data/skrub/blob/403466d1d5d4dc76a7ef569b3f8228db59a31dc3/skrub/_reporting/_data/templates/report.js#L789
     * @returns "light" or "dark"
     */
    function detectTheme(element) {
        const body = document.querySelector('body');

        // Check VSCode theme
        const themeKindAttr = body.getAttribute('data-vscode-theme-kind');
        const themeNameAttr = body.getAttribute('data-vscode-theme-name');

        if (themeKindAttr && themeNameAttr) {
            const themeKind = themeKindAttr.toLowerCase();
            const themeName = themeNameAttr.toLowerCase();

            if (themeKind.includes("dark") || themeName.includes("dark")) {
                return "dark";
            }
            if (themeKind.includes("light") || themeName.includes("light")) {
                return "light";
            }
        }

        // Check Jupyter theme
        if (body.getAttribute('data-jp-theme-light') === 'false') {
            return 'dark';
        } else if (body.getAttribute('data-jp-theme-light') === 'true') {
            return 'light';
        }

        // Guess based on a parent element's color
        const color = window.getComputedStyle(element.parentNode, null).getPropertyValue('color');
        const match = color.match(/^rgb\s*\(\s*(\d+)\s*,\s*(\d+)\s*,\s*(\d+)\s*\)\s*$/i);
        if (match) {
            const [r, g, b] = [
                parseFloat(match[1]),
                parseFloat(match[2]),
                parseFloat(match[3])
            ];

            // https://en.wikipedia.org/wiki/HSL_and_HSV#Lightness
            const luma = 0.299 * r + 0.587 * g + 0.114 * b;

            if (luma > 180) {
                // If the text is very bright we have a dark theme
                return 'dark';
            }
            if (luma < 75) {
                // If the text is very dark we have a light theme
                return 'light';
            }
            // Otherwise fall back to the next heuristic.
        }

        // Fallback to system preference
        return window.matchMedia('(prefers-color-scheme: dark)').matches ? 'dark' : 'light';
    }


    function forceTheme(elementId) {
        const estimatorElement = document.querySelector(`#${elementId}`);
        if (estimatorElement === null) {
            console.error(`Element with id ${elementId} not found.`);
        } else {
            const theme = detectTheme(estimatorElement);
            estimatorElement.classList.add(theme);
        }
    }

    forceTheme('sk-container-id-41');</script></body>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 59-61

We can now inspect the results of our search, sorted by their
`mean_test_score`:

.. GENERATED FROM PYTHON SOURCE LINES 61-71

.. code-block:: Python


    import pandas as pd

    results_df = pd.DataFrame(search.cv_results_)
    results_df = results_df.sort_values(by=["rank_test_score"])
    results_df = results_df.set_index(
        results_df["params"].apply(lambda x: "_".join(str(val) for val in x.values()))
    ).rename_axis("kernel")
    results_df[["params", "rank_test_score", "mean_test_score", "std_test_score"]]






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>params</th>
          <th>rank_test_score</th>
          <th>mean_test_score</th>
          <th>std_test_score</th>
        </tr>
        <tr>
          <th>kernel</th>
          <th></th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>rbf</th>
          <td>{'kernel': 'rbf'}</td>
          <td>1</td>
          <td>0.9400</td>
          <td>0.079297</td>
        </tr>
        <tr>
          <th>linear</th>
          <td>{'kernel': 'linear'}</td>
          <td>2</td>
          <td>0.9300</td>
          <td>0.077846</td>
        </tr>
        <tr>
          <th>3_poly</th>
          <td>{'degree': 3, 'kernel': 'poly'}</td>
          <td>3</td>
          <td>0.9044</td>
          <td>0.098776</td>
        </tr>
        <tr>
          <th>2_poly</th>
          <td>{'degree': 2, 'kernel': 'poly'}</td>
          <td>4</td>
          <td>0.6852</td>
          <td>0.169106</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 72-95

We can see that the estimator using the `'rbf'` kernel performed best,
closely followed by `'linear'`. Both estimators with a `'poly'` kernel
performed worse, with the one using a two-degree polynomial achieving a much
lower performance than all other models.

Usually, the analysis just ends here, but half the story is missing. The
output of :class:`~sklearn.model_selection.GridSearchCV` does not provide
information on the certainty of the differences between the models.
We don't know if these are **statistically** significant.
To evaluate this, we need to conduct a statistical test.
Specifically, to contrast the performance of two models we should
statistically compare their AUC scores. There are 100 samples (AUC
scores) for each model as we repreated 10 times a 10-fold cross-validation.

However, the scores of the models are not independent: all models are
evaluated on the **same** 100 partitions, increasing the correlation
between the performance of the models.
Since some partitions of the data can make the distinction of the classes
particularly easy or hard to find for all models, the models scores will
co-vary.

Let's inspect this partition effect by plotting the performance of all models
in each fold, and calculating the correlation between models across folds:

.. GENERATED FROM PYTHON SOURCE LINES 95-117

.. code-block:: Python


    # create df of model scores ordered by performance
    model_scores = results_df.filter(regex=r"split\d*_test_score")

    # plot 30 examples of dependency between cv fold and AUC scores
    fig, ax = plt.subplots()
    sns.lineplot(
        data=model_scores.transpose().iloc[:30],
        dashes=False,
        palette="Set1",
        marker="o",
        alpha=0.5,
        ax=ax,
    )
    ax.set_xlabel("CV test fold", size=12, labelpad=10)
    ax.set_ylabel("Model AUC", size=12)
    ax.tick_params(bottom=True, labelbottom=False)
    plt.show()

    # print correlation of AUC scores across folds
    print(f"Correlation of models:\n {model_scores.transpose().corr()}")




.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_002.png
   :alt: plot grid search stats
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_002.png
   :class: sphx-glr-single-img


.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Correlation of models:
     kernel       rbf    linear    3_poly    2_poly
    kernel                                        
    rbf     1.000000  0.882561  0.783392  0.351390
    linear  0.882561  1.000000  0.746492  0.298688
    3_poly  0.783392  0.746492  1.000000  0.355440
    2_poly  0.351390  0.298688  0.355440  1.000000




.. GENERATED FROM PYTHON SOURCE LINES 118-129

We can observe that the performance of the models highly depends on the fold.

As a consequence, if we assume independence between samples we will be
underestimating the variance computed in our statistical tests, increasing
the number of false positive errors (i.e. detecting a significant difference
between models when such does not exist) [1]_.

Several variance-corrected statistical tests have been developed for these
cases. In this example we will show how to implement one of them (the so
called Nadeau and Bengio's corrected t-test) under two different statistical
frameworks: frequentist and Bayesian.

.. GENERATED FROM PYTHON SOURCE LINES 131-167

Comparing two models: frequentist approach
------------------------------------------

We can start by asking: "Is the first model significantly better than the
second model (when ranked by `mean_test_score`)?"

To answer this question using a frequentist approach we could
run a paired t-test and compute the p-value. This is also known as
Diebold-Mariano test in the forecast literature [5]_.
Many variants of such a t-test have been developed to account for the
'non-independence of samples problem'
described in the previous section. We will use the one proven to obtain the
highest replicability scores (which rate how similar the performance of a
model is when evaluating it on different random partitions of the same
dataset) while maintaining a low rate of false positives and false negatives:
the Nadeau and Bengio's corrected t-test [2]_ that uses a 10 times repeated
10-fold cross validation [3]_.

This corrected paired t-test is computed as:

.. math::
   t=\frac{\frac{1}{k \cdot r}\sum_{i=1}^{k}\sum_{j=1}^{r}x_{ij}}
   {\sqrt{(\frac{1}{k \cdot r}+\frac{n_{test}}{n_{train}})\hat{\sigma}^2}}

where :math:`k` is the number of folds,
:math:`r` the number of repetitions in the cross-validation,
:math:`x` is the difference in performance of the models,
:math:`n_{test}` is the number of samples used for testing,
:math:`n_{train}` is the number of samples used for training,
and :math:`\hat{\sigma}^2` represents the variance of the observed
differences.

Let's implement a corrected right-tailed paired t-test to evaluate if the
performance of the first model is significantly better than that of the
second model. Our null hypothesis is that the second model performs at least
as good as the first model.

.. GENERATED FROM PYTHON SOURCE LINES 167-225

.. code-block:: Python


    import numpy as np
    from scipy.stats import t


    def corrected_std(differences, n_train, n_test):
        """Corrects standard deviation using Nadeau and Bengio's approach.

        Parameters
        ----------
        differences : ndarray of shape (n_samples,)
            Vector containing the differences in the score metrics of two models.
        n_train : int
            Number of samples in the training set.
        n_test : int
            Number of samples in the testing set.

        Returns
        -------
        corrected_std : float
            Variance-corrected standard deviation of the set of differences.
        """
        # kr = k times r, r times repeated k-fold crossvalidation,
        # kr equals the number of times the model was evaluated
        kr = len(differences)
        corrected_var = np.var(differences, ddof=1) * (1 / kr + n_test / n_train)
        corrected_std = np.sqrt(corrected_var)
        return corrected_std


    def compute_corrected_ttest(differences, df, n_train, n_test):
        """Computes right-tailed paired t-test with corrected variance.

        Parameters
        ----------
        differences : array-like of shape (n_samples,)
            Vector containing the differences in the score metrics of two models.
        df : int
            Degrees of freedom.
        n_train : int
            Number of samples in the training set.
        n_test : int
            Number of samples in the testing set.

        Returns
        -------
        t_stat : float
            Variance-corrected t-statistic.
        p_val : float
            Variance-corrected p-value.
        """
        mean = np.mean(differences)
        std = corrected_std(differences, n_train, n_test)
        t_stat = mean / std
        p_val = t.sf(np.abs(t_stat), df)  # right-tailed t-test
        return t_stat, p_val









.. GENERATED FROM PYTHON SOURCE LINES 226-239

.. code-block:: Python

    model_1_scores = model_scores.iloc[0].values  # scores of the best model
    model_2_scores = model_scores.iloc[1].values  # scores of the second-best model

    differences = model_1_scores - model_2_scores

    n = differences.shape[0]  # number of test sets
    df = n - 1
    n_train = len(next(iter(cv.split(X, y)))[0])
    n_test = len(next(iter(cv.split(X, y)))[1])

    t_stat, p_val = compute_corrected_ttest(differences, df, n_train, n_test)
    print(f"Corrected t-value: {t_stat:.3f}\nCorrected p-value: {p_val:.3f}")





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Corrected t-value: 0.750
    Corrected p-value: 0.227




.. GENERATED FROM PYTHON SOURCE LINES 240-241

We can compare the corrected t- and p-values with the uncorrected ones:

.. GENERATED FROM PYTHON SOURCE LINES 241-250

.. code-block:: Python


    t_stat_uncorrected = np.mean(differences) / np.sqrt(np.var(differences, ddof=1) / n)
    p_val_uncorrected = t.sf(np.abs(t_stat_uncorrected), df)

    print(
        f"Uncorrected t-value: {t_stat_uncorrected:.3f}\n"
        f"Uncorrected p-value: {p_val_uncorrected:.3f}"
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Uncorrected t-value: 2.611
    Uncorrected p-value: 0.005




.. GENERATED FROM PYTHON SOURCE LINES 251-260

Using the conventional significance alpha level at `p=0.05`, we observe that
the uncorrected t-test concludes that the first model is significantly better
than the second.

With the corrected approach, in contrast, we fail to detect this difference.

In the latter case, however, the frequentist approach does not let us
conclude that the first and second model have an equivalent performance. If
we wanted to make this assertion we need to use a Bayesian approach.

.. GENERATED FROM PYTHON SOURCE LINES 262-303

Comparing two models: Bayesian approach
---------------------------------------
We can use Bayesian estimation to calculate the probability that the first
model is better than the second. Bayesian estimation will output a
distribution followed by the mean :math:`\mu` of the differences in the
performance of two models.

To obtain the posterior distribution we need to define a prior that models
our beliefs of how the mean is distributed before looking at the data,
and multiply it by a likelihood function that computes how likely our
observed differences are, given the values that the mean of differences
could take.

Bayesian estimation can be carried out in many forms to answer our question,
but in this example we will implement the approach suggested by Benavoli and
colleagues [4]_.

One way of defining our posterior using a closed-form expression is to select
a prior conjugate to the likelihood function. Benavoli and colleagues [4]_
show that when comparing the performance of two classifiers we can model the
prior as a Normal-Gamma distribution (with both mean and variance unknown)
conjugate to a normal likelihood, to thus express the posterior as a normal
distribution.
Marginalizing out the variance from this normal posterior, we can define the
posterior of the mean parameter as a Student's t-distribution. Specifically:

.. math::
   St(\mu;n-1,\overline{x},(\frac{1}{n}+\frac{n_{test}}{n_{train}})
   \hat{\sigma}^2)

where :math:`n` is the total number of samples,
:math:`\overline{x}` represents the mean difference in the scores,
:math:`n_{test}` is the number of samples used for testing,
:math:`n_{train}` is the number of samples used for training,
and :math:`\hat{\sigma}^2` represents the variance of the observed
differences.

Notice that we are using Nadeau and Bengio's corrected variance in our
Bayesian approach as well.

Let's compute and plot the posterior:

.. GENERATED FROM PYTHON SOURCE LINES 303-309

.. code-block:: Python


    # initialize random variable
    t_post = t(
        df, loc=np.mean(differences), scale=corrected_std(differences, n_train, n_test)
    )








.. GENERATED FROM PYTHON SOURCE LINES 310-311

Let's plot the posterior distribution:

.. GENERATED FROM PYTHON SOURCE LINES 311-322

.. code-block:: Python


    x = np.linspace(t_post.ppf(0.001), t_post.ppf(0.999), 100)

    plt.plot(x, t_post.pdf(x))
    plt.xticks(np.arange(-0.04, 0.06, 0.01))
    plt.fill_between(x, t_post.pdf(x), 0, facecolor="blue", alpha=0.2)
    plt.ylabel("Probability density")
    plt.xlabel(r"Mean difference ($\mu$)")
    plt.title("Posterior distribution")
    plt.show()




.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_003.png
   :alt: Posterior distribution
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_003.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 323-328

We can calculate the probability that the first model is better than the
second by computing the area under the curve of the posterior distribution
from zero to infinity. And also the reverse: we can calculate the probability
that the second model is better than the first by computing the area under
the curve from minus infinity to zero.

.. GENERATED FROM PYTHON SOURCE LINES 328-340

.. code-block:: Python


    better_prob = 1 - t_post.cdf(0)

    print(
        f"Probability of {model_scores.index[0]} being more accurate than "
        f"{model_scores.index[1]}: {better_prob:.3f}"
    )
    print(
        f"Probability of {model_scores.index[1]} being more accurate than "
        f"{model_scores.index[0]}: {1 - better_prob:.3f}"
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Probability of rbf being more accurate than linear: 0.773
    Probability of linear being more accurate than rbf: 0.227




.. GENERATED FROM PYTHON SOURCE LINES 341-347

In contrast with the frequentist approach, we can compute the probability
that one model is better than the other.

Note that we obtained similar results as those in the frequentist approach.
Given our choice of priors, we are essentially performing the same
computations, but we are allowed to make different assertions.

.. GENERATED FROM PYTHON SOURCE LINES 349-368

Region of Practical Equivalence
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sometimes we are interested in determining the probabilities that our models
have an equivalent performance, where "equivalent" is defined in a practical
way. A naive approach [4]_ would be to define estimators as practically
equivalent when they differ by less than 1% in their accuracy. But we could
also define this practical equivalence taking into account the problem we are
trying to solve. For example, a difference of 5% in accuracy would mean an
increase of $1000 in sales, and we consider any quantity above that as
relevant for our business.

In this example we are going to define the
Region of Practical Equivalence (ROPE) to be :math:`[-0.01, 0.01]`. That is,
we will consider two models as practically equivalent if they differ by less
than 1% in their performance.

To compute the probabilities of the classifiers being practically equivalent,
we calculate the area under the curve of the posterior over the ROPE
interval:

.. GENERATED FROM PYTHON SOURCE LINES 368-377

.. code-block:: Python


    rope_interval = [-0.01, 0.01]
    rope_prob = t_post.cdf(rope_interval[1]) - t_post.cdf(rope_interval[0])

    print(
        f"Probability of {model_scores.index[0]} and {model_scores.index[1]} "
        f"being practically equivalent: {rope_prob:.3f}"
    )





.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    Probability of rbf and linear being practically equivalent: 0.432




.. GENERATED FROM PYTHON SOURCE LINES 378-379

We can plot how the posterior is distributed over the ROPE interval:

.. GENERATED FROM PYTHON SOURCE LINES 379-391

.. code-block:: Python


    x_rope = np.linspace(rope_interval[0], rope_interval[1], 100)

    plt.plot(x, t_post.pdf(x))
    plt.xticks(np.arange(-0.04, 0.06, 0.01))
    plt.vlines([-0.01, 0.01], ymin=0, ymax=(np.max(t_post.pdf(x)) + 1))
    plt.fill_between(x_rope, t_post.pdf(x_rope), 0, facecolor="blue", alpha=0.2)
    plt.ylabel("Probability density")
    plt.xlabel(r"Mean difference ($\mu$)")
    plt.title("Posterior distribution under the ROPE")
    plt.show()




.. image-sg:: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_004.png
   :alt: Posterior distribution under the ROPE
   :srcset: /auto_examples/model_selection/images/sphx_glr_plot_grid_search_stats_004.png
   :class: sphx-glr-single-img





.. GENERATED FROM PYTHON SOURCE LINES 392-396

As suggested in [4]_, we can further interpret these probabilities using the
same criteria as the frequentist approach: is the probability of falling
inside the ROPE bigger than 95% (alpha value of 5%)?  In that case we can
conclude that both models are practically equivalent.

.. GENERATED FROM PYTHON SOURCE LINES 398-408

The Bayesian estimation approach also allows us to compute how uncertain we
are about our estimation of the difference. This can be calculated using
credible intervals. For a given probability, they show the range of values
that the estimated quantity, in our case the mean difference in
performance, can take.
For example, a 50% credible interval [x, y] tells us that there is a 50%
probability that the true (mean) difference of performance between models is
between x and y.

Let's determine the credible intervals of our data using 50%, 75% and 95%:

.. GENERATED FROM PYTHON SOURCE LINES 408-421

.. code-block:: Python


    cred_intervals = []
    intervals = [0.5, 0.75, 0.95]

    for interval in intervals:
        cred_interval = list(t_post.interval(interval))
        cred_intervals.append([interval, cred_interval[0], cred_interval[1]])

    cred_int_df = pd.DataFrame(
        cred_intervals, columns=["interval", "lower value", "upper value"]
    ).set_index("interval")
    cred_int_df






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>lower value</th>
          <th>upper value</th>
        </tr>
        <tr>
          <th>interval</th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0.50</th>
          <td>0.000977</td>
          <td>0.019023</td>
        </tr>
        <tr>
          <th>0.75</th>
          <td>-0.005422</td>
          <td>0.025422</td>
        </tr>
        <tr>
          <th>0.95</th>
          <td>-0.016445</td>
          <td>0.036445</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 422-426

As shown in the table, there is a 50% probability that the true mean
difference between models will be between 0.000977 and 0.019023, 70%
probability that it will be between -0.005422 and 0.025422, and 95%
probability that it will be between -0.016445 and 0.036445.

.. GENERATED FROM PYTHON SOURCE LINES 428-444

Pairwise comparison of all models: frequentist approach
-------------------------------------------------------

We could also be interested in comparing the performance of all our models
evaluated with :class:`~sklearn.model_selection.GridSearchCV`. In this case
we would be running our statistical test multiple times, which leads us to
the `multiple comparisons problem
<https://en.wikipedia.org/wiki/Multiple_comparisons_problem>`_.

There are many possible ways to tackle this problem, but a standard approach
is to apply a `Bonferroni correction
<https://en.wikipedia.org/wiki/Bonferroni_correction>`_. Bonferroni can be
computed by multiplying the p-value by the number of comparisons we are
testing.

Let's compare the performance of the models using the corrected t-test:

.. GENERATED FROM PYTHON SOURCE LINES 444-470

.. code-block:: Python


    from itertools import combinations
    from math import factorial

    n_comparisons = factorial(len(model_scores)) / (
        factorial(2) * factorial(len(model_scores) - 2)
    )
    pairwise_t_test = []

    for model_i, model_k in combinations(range(len(model_scores)), 2):
        model_i_scores = model_scores.iloc[model_i].values
        model_k_scores = model_scores.iloc[model_k].values
        differences = model_i_scores - model_k_scores
        t_stat, p_val = compute_corrected_ttest(differences, df, n_train, n_test)
        p_val *= n_comparisons  # implement Bonferroni correction
        # Bonferroni can output p-values higher than 1
        p_val = 1 if p_val > 1 else p_val
        pairwise_t_test.append(
            [model_scores.index[model_i], model_scores.index[model_k], t_stat, p_val]
        )

    pairwise_comp_df = pd.DataFrame(
        pairwise_t_test, columns=["model_1", "model_2", "t_stat", "p_val"]
    ).round(3)
    pairwise_comp_df






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>model_1</th>
          <th>model_2</th>
          <th>t_stat</th>
          <th>p_val</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>rbf</td>
          <td>linear</td>
          <td>0.750</td>
          <td>1.000</td>
        </tr>
        <tr>
          <th>1</th>
          <td>rbf</td>
          <td>3_poly</td>
          <td>1.657</td>
          <td>0.302</td>
        </tr>
        <tr>
          <th>2</th>
          <td>rbf</td>
          <td>2_poly</td>
          <td>4.565</td>
          <td>0.000</td>
        </tr>
        <tr>
          <th>3</th>
          <td>linear</td>
          <td>3_poly</td>
          <td>1.111</td>
          <td>0.807</td>
        </tr>
        <tr>
          <th>4</th>
          <td>linear</td>
          <td>2_poly</td>
          <td>4.276</td>
          <td>0.000</td>
        </tr>
        <tr>
          <th>5</th>
          <td>3_poly</td>
          <td>2_poly</td>
          <td>3.851</td>
          <td>0.001</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 471-476

We observe that after correcting for multiple comparisons, the only model
that significantly differs from the others is `'2_poly'`.
`'rbf'`, the model ranked first by
:class:`~sklearn.model_selection.GridSearchCV`, does not significantly
differ from `'linear'` or `'3_poly'`.

.. GENERATED FROM PYTHON SOURCE LINES 478-486

Pairwise comparison of all models: Bayesian approach
----------------------------------------------------

When using Bayesian estimation to compare multiple models, we don't need to
correct for multiple comparisons (for reasons why see [4]_).

We can carry out our pairwise comparisons the same way as in the first
section:

.. GENERATED FROM PYTHON SOURCE LINES 486-509

.. code-block:: Python


    pairwise_bayesian = []

    for model_i, model_k in combinations(range(len(model_scores)), 2):
        model_i_scores = model_scores.iloc[model_i].values
        model_k_scores = model_scores.iloc[model_k].values
        differences = model_i_scores - model_k_scores
        t_post = t(
            df, loc=np.mean(differences), scale=corrected_std(differences, n_train, n_test)
        )
        worse_prob = t_post.cdf(rope_interval[0])
        better_prob = 1 - t_post.cdf(rope_interval[1])
        rope_prob = t_post.cdf(rope_interval[1]) - t_post.cdf(rope_interval[0])

        pairwise_bayesian.append([worse_prob, better_prob, rope_prob])

    pairwise_bayesian_df = pd.DataFrame(
        pairwise_bayesian, columns=["worse_prob", "better_prob", "rope_prob"]
    ).round(3)

    pairwise_comp_df = pairwise_comp_df.join(pairwise_bayesian_df)
    pairwise_comp_df






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>model_1</th>
          <th>model_2</th>
          <th>t_stat</th>
          <th>p_val</th>
          <th>worse_prob</th>
          <th>better_prob</th>
          <th>rope_prob</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>rbf</td>
          <td>linear</td>
          <td>0.750</td>
          <td>1.000</td>
          <td>0.068</td>
          <td>0.500</td>
          <td>0.432</td>
        </tr>
        <tr>
          <th>1</th>
          <td>rbf</td>
          <td>3_poly</td>
          <td>1.657</td>
          <td>0.302</td>
          <td>0.018</td>
          <td>0.882</td>
          <td>0.100</td>
        </tr>
        <tr>
          <th>2</th>
          <td>rbf</td>
          <td>2_poly</td>
          <td>4.565</td>
          <td>0.000</td>
          <td>0.000</td>
          <td>1.000</td>
          <td>0.000</td>
        </tr>
        <tr>
          <th>3</th>
          <td>linear</td>
          <td>3_poly</td>
          <td>1.111</td>
          <td>0.807</td>
          <td>0.063</td>
          <td>0.750</td>
          <td>0.187</td>
        </tr>
        <tr>
          <th>4</th>
          <td>linear</td>
          <td>2_poly</td>
          <td>4.276</td>
          <td>0.000</td>
          <td>0.000</td>
          <td>1.000</td>
          <td>0.000</td>
        </tr>
        <tr>
          <th>5</th>
          <td>3_poly</td>
          <td>2_poly</td>
          <td>3.851</td>
          <td>0.001</td>
          <td>0.000</td>
          <td>1.000</td>
          <td>0.000</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 510-523

Using the Bayesian approach we can compute the probability that a model
performs better, worse or practically equivalent to another.

Results show that the model ranked first by
:class:`~sklearn.model_selection.GridSearchCV` `'rbf'`, has approximately a
6.8% chance of being worse than `'linear'`, and a 1.8% chance of being worse
than `'3_poly'`.
`'rbf'` and `'linear'` have a 43% probability of being practically
equivalent, while `'rbf'` and `'3_poly'` have a 10% chance of being so.

Similarly to the conclusions obtained using the frequentist approach, all
models have a 100% probability of being better than `'2_poly'`, and none have
a practically equivalent performance with the latter.

.. GENERATED FROM PYTHON SOURCE LINES 525-544

Take-home messages
------------------
- Small differences in performance measures might easily turn out to be
  merely by chance, but not because one model predicts systematically better
  than the other. As shown in this example, statistics can tell you how
  likely that is.
- When statistically comparing the performance of two models evaluated in
  GridSearchCV, it is necessary to correct the calculated variance which
  could be underestimated since the scores of the models are not independent
  from each other.
- A frequentist approach that uses a (variance-corrected) paired t-test can
  tell us if the performance of one model is better than another with a
  degree of certainty above chance.
- A Bayesian approach can provide the probabilities of one model being
  better, worse or practically equivalent than another. It can also tell us
  how confident we are of knowing that the true differences of our models
  fall under a certain range of values.
- If multiple models are statistically compared, a multiple comparisons
  correction is needed when using the frequentist approach.

.. GENERATED FROM PYTHON SOURCE LINES 546-570

.. rubric:: References

.. [1] Dietterich, T. G. (1998). `Approximate statistical tests for
       comparing supervised classification learning algorithms
       <http://web.cs.iastate.edu/~jtian/cs573/Papers/Dietterich-98.pdf>`_.
       Neural computation, 10(7).
.. [2] Nadeau, C., & Bengio, Y. (2000). `Inference for the generalization
       error
       <https://papers.nips.cc/paper/1661-inference-for-the-generalization-error.pdf>`_.
       In Advances in neural information processing systems.
.. [3] Bouckaert, R. R., & Frank, E. (2004). `Evaluating the replicability
       of significance tests for comparing learning algorithms
       <https://www.cms.waikato.ac.nz/~ml/publications/2004/bouckaert-frank.pdf>`_.
       In Pacific-Asia Conference on Knowledge Discovery and Data Mining.
.. [4] Benavoli, A., Corani, G., Demšar, J., & Zaffalon, M. (2017). `Time
       for a change: a tutorial for comparing multiple classifiers through
       Bayesian analysis
       <http://www.jmlr.org/papers/volume18/16-305/16-305.pdf>`_.
       The Journal of Machine Learning Research, 18(1). See the Python
       library that accompanies this paper `here
       <https://github.com/janezd/baycomp>`_.
.. [5] Diebold, F.X. & Mariano R.S. (1995). `Comparing predictive accuracy
       <http://www.est.uc3m.es/esp/nueva_docencia/comp_col_get/lade/tecnicas_prediccion/Practicas0708/Comparing%20Predictive%20Accuracy%20(Dielbold).pdf>`_
       Journal of Business & economic statistics, 20(1), 134-144.


.. rst-class:: sphx-glr-timing

   **Total running time of the script:** (0 minutes 3.004 seconds)


.. _sphx_glr_download_auto_examples_model_selection_plot_grid_search_stats.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_grid_search_stats.ipynb <plot_grid_search_stats.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_grid_search_stats.py <plot_grid_search_stats.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_grid_search_stats.zip <plot_grid_search_stats.zip>`


.. include:: plot_grid_search_stats.recommendations


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_
