Поговорим про performance-тестирование

name: template-default
layout: true

<div id="slide-controller">
 <div id="slide-prev" style="display: inline-block;" onclick="slideshow.gotoPreviousSlide()">◀</div>
 <div id="slide-next" style="display: inline-block;" onclick="slideshow.gotoNextSlide()">▶</div>
</div>
---
name: template-title
layout: true
template: template-default
class: slide-title, center
---
name: template-section
layout: true
template: template-default
class: slide-section, center, middle
---
name: template-page
layout: true
template: template-default
class: slide-page
---
name: template-comics
layout: true
template: template-default
class: slide-comics
---
template: template-title

# Поговорим про перфоманс-анализ
## Андрей Акиньшин, JetBrains
### DotNext Piter, 16.06.2020

---
layout: true
template: template-page

---
class: normal

### Типичный день перфоманс-инженера

.up7[]
.center-wide[
![:scale 99%](img/artworks/dino-fire.gif)
]

---
class: normal
count: false

### Типичный день любого инженера

.up7[]
.center-wide[
![:scale 99%](img/artworks/dino-fire.gif)
]

---
class: normal

### Четыре главные задачи перфоманс-анализа

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/problems-explore.svg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Анализ распределений</td>
 <td></td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Четыре главные задачи перфоманс-анализа

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/problems-explore.svg" /></td>
 <td><img src="img/ggplot/problems-compare.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Анализ распределений</td>
 <td>Сравнение бенчмарков</td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Четыре главные задачи перфоманс-анализа

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/problems-explore.svg" /></td>
 <td><img src="img/ggplot/problems-compare.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Анализ распределений</td>
 <td>Сравнение бенчмарков</td>
 </tr>
 <tr>
 <td><img src="img/ggplot/problems-history.svg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Анализ истории</td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Четыре главные задачи перфоманс-анализа

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/problems-explore.svg" /></td>
 <td><img src="img/ggplot/problems-compare.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Анализ распределений</td>
 <td>Сравнение бенчмарков</td>
 </tr>
 <tr>
 <td><img src="img/ggplot/problems-history.svg" /></td>
 <td><img src="img/ggplot/problems-test.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Анализ истории</td>
 <td>Перфоманс-тесты</td>
 </tr>
</table>

]

---
class: normal

### Источники вдохновения

.up[]

.center[![:scale 100%](img/brand/bdn.svg)]

</div>

.center[
<img src="img/brand/perfolizer.svg" style="width: 70%; padding-left: 140px; padding-top: 10px;" />
]

.center[
<img src="img/brand/rider.svg" style="width: 50%; padding-right: 99px; margin-top: -31px;" />
]

---
template: template-section

## Часть 1
## Изучаем сырые данные

---
layout: true
template: template-page

<div>.footer-note[(1) Изучаем сырые данные]</div>

---
class: normal

### Анализируем глазами

.up[]
.center[![:scale 60%](img/ggplot/eyes.svg)]

**Задачка:** если у нас есть 10'000 тестов, то какова вероятность получить такую картинку хотя бы в одном из них?

---
class: normal

### Считаем вероятности

.up[]
.center[Предположим, что форма распределения известна:]
.pull-left[![:scale 100%](img/ggplot/eyes.svg)]
.pull-right[![:scale 100%](img/ggplot/eyes-density.svg)]

.center[Вероятность появления "плохой" картинки:]
$$
2^{-10} = 0.0009765625
$$

.center[Вероятность появления хотя бы одной "плохой" картинки из 10'000:]

$$
1 - (1-2^{-10})^{10000} = 0.9999429
$$

---
class: normal

### График вероятности

.up[]
.center[![:scale 100%](img/ggplot/fp-probability.svg)]

---
class: normal

### Закон малых чисел

.up[]
.center[![:scale 100%](img/ggplot/small-numbers.svg)]
.bottom-hint-huge[Tversky, Amos, and Daniel Kahneman. ["Belief in the law of small numbers."](http://www.stats.org.uk/statistical-inference/TverskyKahneman1971.pdf) Psychological bulletin 76, no. 2 (1971): 105]

---
class: normal

### Кто лучше всех разбирается в перфоманс-анализе?

---
class: normal
count: false

### Кто лучше всех разбирается в перфоманс-анализе?

<table class="invisible-table" style="width: 100%;">
 <tr>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-me.jpg" /></td>
 <td style="width: 50%;text-align: center;"></td>
 </tr>
 <tr>
 <td style="width: 50%;text-align: center;">Процент угадываний: 20-30%</td>
 <td style="width: 50%;text-align: center;"></td>
 </tr>
</table>

???

И тогда я пошёл к руководителю всего нашего .NET-отдела и попросил порешать те же самые перфомансные задачки (драматическая пауза) его кошку.

---
class: normal
count: false

### Кто лучше всех разбирается в перфоманс-анализе?

<table class="invisible-table" style="width: 100%;">
 <tr>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-me.jpg" /></td>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-cat1.jpg" /></td>
 </tr>
 <tr>
 <td style="width: 50%;text-align: center;">Процент угадываний: 20-30%</td>
 <td style="width: 50%;text-align: center;"></td>
 </tr>
</table>

---
class: normal
count: false

### Кто лучше всех разбирается в перфоманс-анализе?

<table class="invisible-table" style="width: 100%;">
 <tr>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-me.jpg" /></td>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-cat2.jpg" /></td>
 </tr>
 <tr>
 <td style="width: 50%;text-align: center;">Процент угадываний: 20-30%</td>
 <td style="width: 50%;text-align: center;"></td>
 </tr>
</table>

---
class: normal
count: false

### Кто лучше всех разбирается в перфоманс-анализе?

<table class="invisible-table" style="width: 100%;">
 <tr>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-me.jpg" /></td>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-cat3.jpg" /></td>
 </tr>
 <tr>
 <td style="width: 50%;text-align: center;">Процент угадываний: 20-30%</td>
 <td style="width: 50%;text-align: center;"></td>
 </tr>
</table>

---
class: normal
count: false

### Кто лучше всех разбирается в перфоманс-анализе?

<table class="invisible-table" style="width: 100%;">
 <tr>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-me.jpg" /></td>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-cat3.jpg" /></td>
 </tr>
 <tr>
 <td style="width: 50%;text-align: center;">Процент угадываний: 20-30%</td>
 <td style="width: 50%;text-align: center;">Процент угадываний: 45-55%</td>
 </tr>
</table>

---
class: normal
count: false

### Кто лучше всех разбирается в перфоманс-анализе?

<table class="invisible-table" style="width: 100%;">
 <tr>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-me.jpg" /></td>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-cat4.jpg" /></td>
 </tr>
 <tr>
 <td style="width: 50%;text-align: center;">Процент угадываний: 20-30%</td>
 <td style="width: 50%;text-align: center;">Процент угадываний: 45-55%</td>
 </tr>
</table>

---
class: normal
count: false

### Кто лучше всех разбирается в перфоманс-анализе?

<table class="invisible-table" style="width: 100%;">
 <tr>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-me.jpg" /></td>
 <td style="width: 50%;text-align: center;"><img class="engineer-photo" src="img/photos/engineer-coins.jpg" /></td>
 </tr>
 <tr>
 <td style="width: 50%;text-align: center;">Процент угадываний: 20-30%</td>
 <td style="width: 50%;text-align: center;">Процент угадываний: 45-55%</td>
 </tr>
</table>

---
class: normal

### Нужно запомнить

.memorize[Сырые данные контринтуитивны и обманчивы]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
 Но если пользоваться интуицией опасно, то как правильно изучать данные?
]

---
template: template-section

## Часть 2
## Изучаем сводные метрики

---
layout: true
template: template-page

<div>.footer-note[(2) Изучаем сводные метрики]</div>

---
class: normal

### Нормальное распределение

.up[]
.center[![:scale 85%](img/ggplot/normal.svg)]

.up[]
<div style="font-size: 70%">
.c[
> *Нормальность — это миф; никогда не было и никогда не будет ни одного нормального распределения.* 
> ["Testing for normality"](http://webspace.ship.edu/pgmarr/Geo441/Readings/Geary%201947%20-%20Testing%20for%20Normality.pdf), R.C. Geary, 1947
]
</div>

---
class: normal

### Проведём эксперимент

.up[]
.size-75[
.center[
<pre><code class="md hljs markdown remark-code"><div class="remark-code-line">| Метод | Среднее | Стандартное отклонение | Медиана |</div>
<div class="remark-code-line">|-------|----------|------------------------|----------|</div>
<div class="remark-code-line">| A | 136.2 мс | 56.92 мс | 107.0 мс |</div>
<div class="remark-code-line">| B | 133.7 мс | 12.20 мс | 130.2 мс |</div>
</code></pre>
]
]

.up1[]
.pull-left[.center[![:scale 110%](img/ggplot/misleading1.svg)]]

.pull-right[.center[![:scale 110%](img/ggplot/misleading2.svg)]]

---
class: normal

### Квартет Энскомба

</td><td style="vertical-align:top;">

</td></tr>
</table>

.bottom-hint-huge[Anscombe, Francis J. ["Graphs in statistical analysis."](http://ww.w.lithoguru.com/scientist/statistics/Anscombe_Graphs%20in%20Statistical%20Analysis_1973.pdf) The american statistician 27, no. 1 (1973): 17-21.]

---
class: normal
count: false

### Квартет Энскомба

</td><td style="vertical-align:top;">

<table class="visible-table" style="text-align: left">
 <tr><th>Характеристика</th><th>Значение</th></tr>
 <tr><td>Среднее x</td><td>9.0</td></tr>
 <tr><td>Дисперсия x</td><td>11.0</td></tr>
 <tr><td>Среднее y</td><td>7.5</td></tr>
 <tr><td>Дисперсия y</td><td>4.125</td></tr>
 <tr><td>Корреляция rxy</td><td>0.816</td></tr>
 <tr><td>Линейная регрессия</td><td>y=3+0.5x</td></tr>
</table>

</td></tr>
</table>

---
class: normal

### Квартет Энскомба

.up[]
.center[![:scale 100%](img/ggplot/anscombe.svg)]

---
class: normal

### Обманчивые среднее и дисперсия

.up[]
.center[![:scale 70%](img/screenshots/samestats1.png)]
.bottom-hint-huge[Justin Matejka, George Fitzmaurice (2017), ["Same Stats, Different Graphs: Generating Datasets with Varied Appearance and Identical Statistics through Simulated Annealing"](https://www.autodeskresearch.com/publications/samestats), CHI 2017 Conference proceedings: ACM SIGCHI Conference on Human Factors in Computing Systems]

---
class: normal

### Обманчивые среднее и дисперсия

.center[![:scale 100%](img/screenshots/samestats2.gif)]
.bottom-hint-huge[Justin Matejka, George Fitzmaurice (2017), ["Same Stats, Different Graphs: Generating Datasets with Varied Appearance and Identical Statistics through Simulated Annealing"](https://www.autodeskresearch.com/publications/samestats), CHI 2017 Conference proceedings: ACM SIGCHI Conference on Human Factors in Computing Systems]

---
class: normal

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Интервал     |
|-------|--------------|
| A     | `100мс..200мс` |
| B     | `120мс..150мс` |
| C     | `400мс..900мс` |
```
]

---
class: normal
count: false

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Интервал     | Заметки             |
|-------|--------------|---------------------|
| A     | 100мс..200мс | `Большие выбросы`     |
| B     | 120мс..150мс |                     |
| C     | 400мс..900мс |                     |
```
]

---
class: normal
count: false

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Интервал     | Заметки             |
|-------|--------------|---------------------|
| A     | 100мс..200мс | Большие выбросы`^1`   |
| B     | 120мс..150мс |                     |
| C     | 400мс..900мс |                     |

`^1 Выбросы: 327мс, 364мс, 396мс`
```
]

---
class: normal
count: false

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Интервал     | Заметки             |
|-------|--------------|---------------------|
| A     | 100мс..200мс | Большие выбросы^1   |
| B     | 120мс..150мс | `Маленькая выборка`   |
| C     | 400мс..900мс |                     |

^1 Выбросы: 327мс, 364мс, 396мс
```
]

---
class: normal
count: false

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Интервал     | Заметки             |
|-------|--------------|---------------------|
| A     | 100мс..200мс | Большие выбросы^1   |
| B     | 120мс..150мс | Маленькая выборка`^2` |
| C     | 400мс..900мс |                     |

^1 Выбросы: 327мс, 364мс, 396мс
`^2 Размер выборки: 3 замера`
```
]

---
class: normal
count: false

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Интервал     | Заметки             |
|-------|--------------|---------------------|
| A     | 100мс..200мс | Большие выбросы^1   |
| B     | 120мс..150мс | Маленькая выборка^2 |
| C     | 400мс..900мс | `Мультимодальность`   |

^1 Выбросы: 327мс, 364мс, 396мс
^2 Размер выборки: 3 замера
```
]

---
class: normal
count: false

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Интервал     | Заметки             |
|-------|--------------|---------------------|
| A     | 100мс..200мс | Большие выбросы^1   |
| B     | 120мс..150мс | Маленькая выборка^2 |
| C     | 400мс..900мс | Мультимодальность`^3` |

^1 Выбросы: 327мс, 364мс, 396мс
^2 Размер выборки: 3 замера
`^3 Интервалы: [400мс..450мс] и [850мс..900мс]`
```
]

---
class: normal

### Источники шума и мультимодальности

.center-wide[
.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/artworks/noise-scheduler.svg" /></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Планировщик ОС</td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 <td></td>
 </tr>
</table>

]
]

---
class: normal
count: false

### Источники шума и мультимодальности

.center-wide[
.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/artworks/noise-scheduler.svg" /></td>
 <td><img src="img/artworks/noise-threads.svg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Планировщик ОС</td>
 <td>Многопоточность</td>
 <td</td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 <td></td>
 </tr>
</table>

]
]

---
class: normal
count: false

### Источники шума и мультимодальности

.center-wide[
.dashboard-table[

]
]

---
class: normal
count: false

### Источники шума и мультимодальности

.center-wide[
.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/artworks/noise-scheduler.svg" /></td>
 <td><img src="img/artworks/noise-threads.svg" /></td>
 <td><img src="img/artworks/noise-alignment.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Планировщик ОС</td>
 <td>Многопоточность</td>
 <td>Выравнивание</td>
 </tr>
 <tr>
 <td><img src="img/artworks/noise-cache.svg" /></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Кэш процессора</td>
 <td></td>
 <td></td>
 </tr>
</table>

]
]

---
class: normal
count: false

### Источники шума и мультимодальности

.center-wide[
.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/artworks/noise-scheduler.svg" /></td>
 <td><img src="img/artworks/noise-threads.svg" /></td>
 <td><img src="img/artworks/noise-alignment.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Планировщик ОС</td>
 <td>Многопоточность</td>
 <td>Выравнивание</td>
 </tr>
 <tr>
 <td><img src="img/artworks/noise-cache.svg" /></td>
 <td><img src="img/artworks/noise-freq.svg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Кэш процессора</td>
 <td>Изменение частоты CPU</td>
 <td></td>
 </tr>
</table>

]
]

---
class: normal
count: false

### Источники шума и мультимодальности

.center-wide[
.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/artworks/noise-scheduler.svg" /></td>
 <td><img src="img/artworks/noise-threads.svg" /></td>
 <td><img src="img/artworks/noise-alignment.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Планировщик ОС</td>
 <td>Многопоточность</td>
 <td>Выравнивание</td>
 </tr>
 <tr>
 <td><img src="img/artworks/noise-cache.svg" /></td>
 <td><img src="img/artworks/noise-freq.svg" /></td>
 <td><img src="img/artworks/noise-throttling.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Кэш процессора</td>
 <td>Изменение частоты CPU</td>
 <td>Термальный троттлинг</td>
 </tr>
</table>

]
]

---
class: normal

### Термальный троттлинг

.up[]

---
class: normal
count: false

### Термальный троттлинг

.up[]

---
class: normal
count: false

### Термальный троттлинг

.up[]

---
class: normal
count: false

### Термальный троттлинг

.up[]

<table class="invisible-table" style="width: 100%">
 <tr>
 <td style="width: 33%"><img src="img/photos/mac-freezer.jpg" class="mac-photo" /></td>
 <td style="width: 33%"><img src="img/photos/mac-freezer-thermal.jpg" class="mac-photo" /></td>
 <td style="width: 34%;vertical-align: top;">
 <pre style="margin: 0"><code class="md hljs markdown remark-code">(В морозилке)
Частота процессора: 3.5ГГц
Итерация 1: 103мс
Итерация 2: 104мс
Итерация 3: 102мс
Итерация 4: 101мс
Итерация 5: 106мс
Итерация 6: 104мс
</code></pre>
 </td>
 </tr>
 <tr>
 <td style="width: 33%"><img src="img/photos/mac-blanket.jpg" class="mac-photo" /></td>
 <td style="width: 33%"><img src="img/photos/mac-blanket-thermal.jpg" class="mac-photo" /></td>
 <td style="width: 34%;vertical-align: top;">
 <pre style="margin: 0"><code class="md hljs markdown remark-code">(В одеялке)
Частота процессора: 2.5ГГц
Итерация 1: 145мс
Итерация 2: 142мс
Итерация 3: 147мс
Итерация 4: 139мс
Итерация 5: 144мс
Итерация 6: 142мс
</code></pre>
 </td>
 </tr>
</table>

---
class: normal

### Нужно запомнить

.memorize[Статистика должна быть дружелюбной]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
 Допустим я разобрался со всеми этими метриками. 
 Как мне два бенчмарка-то сравнить?
]

---
template: template-section

## Часть 3
## Применяем статистические тесты

---
layout: true
template: template-page

<div>.footer-note[(3) Применяем статистические тесты]</div>

---
class: normal

### Типичная деградация

.up[]
.center[![:scale 55%](img/ggplot/compare1.svg)]

---
class: normal

### Медитируем на список статистических тестов

.up3[]
.center[![:scale 59.5%](img/screenshots/statistics-tests-overview.png)]

---
class: normal

### Общая схема

<div style="display:block; margin-top:-22%;"></div>
.center[![:scale 65%](img/flowcharts/hypotheses1.svg)]

---
class: normal
count: false

### Общая схема

<div style="display:block; margin-top:-22%;"></div>
.center[![:scale 65%](img/flowcharts/hypotheses2.svg)]

---
class: normal
count: false

### Общая схема

<div style="display:block; margin-top:-22%;"></div>
.center[![:scale 65%](img/flowcharts/hypotheses3.svg)]

---
class: normal
count: false

### Общая схема

<div style="display:block; margin-top:-22%;"></div>
.center[![:scale 65%](img/flowcharts/hypotheses3.svg)]

<div style="display:block; margin-top:-32%;"></div>
.center[![:scale 70%](img/flowcharts/pvalue1.svg)]

---
class: normal
count: false

### Общая схема

<div style="display:block; margin-top:-22%;"></div>
.center[![:scale 65%](img/flowcharts/hypotheses3.svg)]

<div style="display:block; margin-top:-32%;"></div>
.center[![:scale 70%](img/flowcharts/pvalue2.svg)]

---
class: normal
count: false

### Общая схема

<div style="display:block; margin-top:-22%;"></div>
.center[![:scale 65%](img/flowcharts/hypotheses3.svg)]

<div style="display:block; margin-top:-32%;"></div>
.center[![:scale 70%](img/flowcharts/pvalue3.svg)]

---
class: normal

### Статистические ошибки

.center[**Самые главные ошибки**]

* *Ошибка первого рода   :* деградации нет, а мы решили, что есть

* *Ошибка второго рода   :* деградация есть, а мы решили, что нет

* *Ошибка третьего рода.superscript[1] :* мы получили ответ на неправильный вопрос

.bottom-hint-huge[.superscript[1]Mitroff, Ian I., and Tom R. Featheringham. ["On systemic problem solving and the error of the third kind."](https://doi.org/10.1002/bs.3830190605) Behavioral Science 19, no. 6 (1974): 383-393.]

.center[**Вероятности**]

* *Ошибка первого рода   :* α (значение по умолчанию = 0.05)

* *Ошибка второго рода   :* β (значение по умолчанию = 0.20)

* *Ошибка третьего рода  :* большая вероятность

---
class: normal

### Уровень статистической значимости: α=0.05

.bottom-hint-huge[Fisher RA. [The Design of Experiments](https://archive.org/details/in.ernet.dli.2015.502684), Edinburgh: Oliver and Boyd. 1935. Pages 15–16]

---
class: normal
count: false

### Уровень статистической значимости: α=0.05

.bottom-hint-huge[Fisher RA. [The Design of Experiments](https://archive.org/details/in.ernet.dli.2015.502684), Edinburgh: Oliver and Boyd. 1935. Pages 15–16]

---
class: normal
count: false

### Уровень статистической значимости: α=0.05

<div style="position: absolute;left: 51%;top:15%;font-size: 80%;">
<pre><code class="cs hljs remark-code">class StatisticalTest
{
 // Магическое число,
 // которые мы выбрали случайным образом
 const double DefaultAlpha = 0.05;

// Никогда не используйте умолчание!
 public StatisticalTest(
 double alpha = DefaultAlpha)
 {
 // ...
 }
}
</code></pre>
</div>

.bottom-hint-huge[Fisher RA. [The Design of Experiments](https://archive.org/details/in.ernet.dli.2015.502684), Edinburgh: Oliver and Boyd. 1935. Pages 15–16]

---
class: normal

### Статистическая мощность теста: 1-β=0.80

.bottom-hint-huge[Cohen, Jacob. [Statistical power analysis for the behavioral sciences.](http://www.utstat.toronto.edu/~brunner/oldclass/378f16/readings/CohenPower.pdf) No. 300.72 C6. 1988. Page 56]

---
class: normal
count: false

### Статистическая мощность теста: 1-β=0.80

.bottom-hint-huge[Cohen, Jacob. [Statistical power analysis for the behavioral sciences.](http://www.utstat.toronto.edu/~brunner/oldclass/378f16/readings/CohenPower.pdf) No. 300.72 C6. 1988. Page 56]

---
class: normal
count: false

### Статистическая мощность теста: 1-β=0.80

<div style="position: absolute;left: 51%;top:15%;font-size: 80%;">
<pre><code class="cs hljs remark-code">class StatisticalTest
{
 // Магические числа,
 // которые мы выбрали случайным образом
 const double DefaultAlpha = 0.05;
 const double DefaultBeta = DefaultAlpha * 4;

// Никогда не используйте умолчания!
 public StatisticalTest(
 double alpha = DefaultAlpha,
 double beta = DefaultBeta)
 {
 // ...
 }
}
</code></pre>
</div>

.bottom-hint-huge[Cohen, Jacob. [Statistical power analysis for the behavioral sciences.](http://www.utstat.toronto.edu/~brunner/oldclass/378f16/readings/CohenPower.pdf) No. 300.72 C6. 1988. Page 56]

---
class: normal

### Эвфемизмы для неудачных результатов

<div id="not-significant" style="z-index: 0">
<pre>
(barely) not statistically significant p=0.052
a barely detectable statistically significant difference p=0.073
a borderline significant trend p=0.09
a certain trend toward significance p=0.08
a clear tendency to significance p=0.052
a clear trend p<0.09
a clear, strong trend p=0.09
a considerable trend toward significance p=0.069
a decreasing trend p=0.09
a definite trend p=0.08
a distinct trend toward significance p=0.07
a favorable trend p=0.09
a favourable statistical trend p=0.09
a little significant p<0.1
a margin at the edge of significance p=0.0608
a marginal trend p=0.09
a marginal trend toward significance p=0.052
a marked trend p=0.07
a mild trend p<0.09
a moderate trend toward significance p=0.068
a near-significant trend p=0.07
a negative trend p=0.09
a nonsignificant trend p<0.1
a nonsignificant trend toward significance p=0.1
a notable trend p<0.1
a numerical increasing trend p=0.09
a numerical trend p=0.09
a positive trend p=0.09
a possible trend p=0.09
a possible trend toward significance p=0.052
a pronounced trend p=0.09
a reliable trend p=0.058
a robust trend toward significance p=0.0503
a significant trend p=0.09
a slight slide towards significance p<0.20
a slight tendency toward significance p<0.08
a slight trend p<0.09
a slight trend toward significance p=0.098
a slightly increasing trend p=0.09
a small trend p=0.09
a statistical trend p=0.09
a statistical trend toward significance p=0.09
a strong tendency towards statistical significance p=0.051
a strong trend p=0.077
a strong trend toward significance p=0.08
a substantial trend toward significance p=0.068
a suggestive trend p=0.06
a trend close to significance p=0.08
a trend significance level p=0.08
a trend that approached significance p<0.06
a very slight trend toward significance p=0.20
a weak trend p=0.09
a weak trend toward significance p=0.12
a worrying trend p=0.07
all but significant p=0.055
almost achieved significance p=0.065
almost approached significance p=0.065
almost attained significance p<0.06
almost became significant p=0.06
almost but not quite significant p=0.06
almost clinically significant p<0.10
almost insignificant p<0.065
almost marginally significant p>0.05
almost non-significant p=0.083
almost reached statistical significance p=0.06
almost significant p=0.06
almost significant tendency p=0.06
almost statistically significant p=0.06
an adverse trend p=0.10
an apparent trend p=0.286
an associative trend p=0.09
an elevated trend p<0.05
an encouraging trend p<0.1
an established trend p<0.10
an evident trend p=0.13
an expected trend p=0.08
an important trend p=0.066
an increasing trend p<0.09
an interesting trend p=0.1
an inverse trend toward signiﬁcance p=0.06
an observed trend p=0.06
an obvious trend p=0.06
an overall trend p=0.2
an unexpected trend p=0.09
an unexplained trend p=0.09
an unfavorable trend p<0.10
appeared to be marginally significant p<0.10
approached acceptable levels of statistical significance p=0.054
approached but did not quite achieve significance p>0.05
approached but fell short of significance p=0.07
approached conventional levels of significance p<0.10
approached near significance p=0.06
approached our criterion of significance p>0.08
approached significant p=0.11
approached the borderline of significance p=0.07
approached the level of signiﬁcance p=0.09
approached trend levels of significance p=0.05
approached, but did reach, significance p=0.065
approaches statistical significance p>0.06
approaching a level of significance p=0.089
approaching an acceptable significance level p=0.056
approaching borderline significance p=0.08
approaching borderline statistical significance p=0.07
approaching but not reaching significance p=0.53
approaching clinical significance p=0.07
approaching close to significance p<0.1
approaching conventional significance levels p=0.06
approaching conventional statistical significance p=0.06
approaching formal significance p=0.1052
approaching independent prognostic significance p=0.08
approaching marginal levels of significance p<0.107
approaching marginal significance p=0.064
approaching more closely significance p=0.06
approaching our preset significance level p=0.076
approaching prognostic significance p=0.052
approaching significance p=0.09
approaching the traditional significance level p=0.06
approaching to statistical significance p=0.075
approaching, although not reaching, significance p=0.08
approaching, but not reaching, significance p<0.09
approximately significant p=0.053
approximating significance p=0.09
arguably significant p=0.07
as good as significant p=0.0502
at the brink of significance p=0.06
at the cusp of significance p=0.06
at the edge of significance p=0.055
at the limit of significance p=0.054
at the limits of significance p=0.053
at the margin of significance p=0.056
at the margin of statistical significance p<0.07
at the verge of significance p=0.058
at the very edge of significance p=0.053
barely below the level of significance p=0.06
barely escaped statistical significance p=0.07
barely failed to attain statistical significance p=0.067
barely fails to attain statistical significance at conventional levels p<0.10
barely insignificant p=0.075
barely missed statistical significance p=0.051
barely missed the commonly acceptable significance level p<0.053
barely outside the range of significance p=0.06
barely significant p=0.07
below (but verging on) the statistical significant level p>0.05
better trends of improvement p=0.056
bordered on a statistically significant value p=0.06
bordered on being significant p>0.07
bordered on being statistically significant p=0.0502
bordered on but was not less than the accepted level of significance p>0.05
bordered on significant p=0.09
borderline conventional significance p=0.051
borderline level of statistical significance p=0.053
borderline signiﬁcant p=0.09
borderline significant trends p=0.099
close to a marginally significant level p=0.06
close to being significant p=0.06
close to being statistically signiﬁcant p=0.055
close to borderline signiﬁcance p=0.072
close to the boundary of significance p=0.06
close to the level of significance p=0.07
close to the limit of significance p=0.17
close to the margin of significance p=0.055
close to the margin of statistical significance p=0.075
closely approaches the brink of signiﬁcance p=0.07
closely approaches the statistical significance p=0.0669
closely approximating significance p>0.05
closely not significant p=0.06
closely significant p=0.058
close-to-signiﬁcant p=0.09
did not achieve conventional threshold levels of statistical significancp=0.08
did not exceed the conventional level of statistical significance p<0.08
did not quite achieve acceptable levels of statistical significance p=0.054
did not quite achieve significance p=0.076
did not quite achieve the conventional levels of significance p=0.052
did not quite achieve the threshold for statistical significance p=0.08
did not quite attain conventional levels of significance p=0.07
did not quite reach a statistically significant level p=0.108
did not quite reach conventional levels of statistical significance p=0.079
did not quite reach statistical significance p=0.063
did not reach the traditional level of signiﬁcance p=0.10
did not reach the usually accepted level of clinical significance p=0.07
difference was apparent p=0.07
direction heading towards significance p=0.10
does not appear to be sufficiently significant p>0.05
does not narrowly reach statistical significance p=0.06
does not reach the conventional significance level p=0.098
effectively significant p=0.051
equivocal significance p=0.06
essentially significant p=0.10
extremely close to signiﬁcance p=0.07
failed to reach significance on this occasion p=0.09
failed to reach statistical significance p=0.06
fairly close to significance p=0.065
fairly significant p=0.09
falls just short of standard levels of statistical significance p=0.06
fell (just) short of significance p=0.08
fell barely short of significance p=0.08
fell just short of significance p=0.07
fell just short of statistical significance p=0.12
fell marginally short of significance p=0.07
fell narrowly short of significance p=0.0623
fell only marginally short of significance p=0.0879
fell only short of significance p=0.06
fell short of significance p=0.07
fell slightly short of significance p=0.0167
fell somewhat short of significance p=0.138
felt short of significance p=0.07
flirting with conventional levels of significance p>0.1
heading towards significance p=0.086
highly significant p=0.09
hint of significance p>0.05
hovered around signiﬁcance p=0.061
hovered at nearly a significant level p=0.058
hovering closer to statistical significance p=0.076
hovers on the brink of significance p=0.055
in the edge of significance p=0.059
in the verge of significance p=0.06
inconclusively significant p=0.070
indeterminate significance p=0.08
indicative significance p=0.08
just about significant p=0.051
just above the arbitrary level of signiﬁcance p=0.07
just above the margin of significance p=0.053
just at the conventional level of significance p=0.05001
just barely below the level of significance p=0.06
just barely failed to reach significance p=<0.06
just barely insignificant p=0.11
just barely statistically signiﬁcant p=0.054
just beyond significance p=0.06
just borderline significant p=0.058
just escaped significance p=0.07
just failed significance p=0.057
just failed to be significant p=0.072
just failed to reach statistical significance p=0.06
just failing to reach statistical significance p=0.06
just fails to reach conventional levels of statistical significance p=0.07
just lacked significance p=0.053
just marginally significant p=0.0562
just missed being statistically significant p=0.06
just missing significance p=0.07
just on the verge of significance p=0.06
just outside accepted levels of significance p=0.06
just outside levels of significance p<0.08
just outside the bounds of significance p=0.06
just outside the conventional levels of significance p=0.1076
just outside the level of significance p=0.0683
just outside the limits of significance p=0.06
just outside the traditional bounds of significance p=0.06
just over the limits of statistical significance p=0.06
just short of significance p=0.07
just shy of significance p=0.053
just skirting the boundary of significance p=0.052
just tendentially signiﬁcant p=0.056
just very slightly missed the significance level p=0.086
leaning towards significance p=0.15
leaning towards statistical significance p=0.06
likely to be significant p=0.054
loosely significant p=0.10
marginal significance p=0.07
marginally and negatively significant p=0.08
marginally insignificant p=0.08
marginally nonsignificant p=0.096
marginally significant p>=0.1
marginally significant tendency p=0.08
marginally statistically significant p=0.08
may not be signiﬁcant p=0.06
medium level of significance p=0.051
mildly signiﬁcant p=0.07
missed narrowly statistical significance p=0.054
moderately significant p>0.11
modestly significant p=0.09
narrowly avoided significance p=0.052
narrowly eluded statistical significance p=0.0789
narrowly escaped significance p=0.08
narrowly evaded statistical significance p>0.05
narrowly failed significance p=0.054
narrowly missed achieving significance p=0.055
narrowly missed overall significance p=0.06
narrowly missed significance p=0.051
narrowly missed standard significance levels p<0.07
narrowly missed the significance level p=0.07
narrowly missing conventional significance p=0.054
near limit significance p=0.073
near miss of statistical significance p>0.1
near nominal significance p=0.064
near significance p=0.07
near to statistical significance p=0.056
near/possible significance p=0.0661
near-borderline significance p=0.10
near-certain signiﬁcance p=0.07
nearing significance p>0.051
nearly acceptable level of significance p=0.06
nearly approaches statistical significance p=0.079
nearly borderline significance p=0.052
nearly negatively significant p<0.1
nearly positively significant p=0.063
nearly reached a significant level p=0.07
nearly reaching the level of significance p<0.06
nearly significant p=0.06
nearly significant tendency p=0.06
nearly, but not quite significant p>0.06
near-marginal significance p=0.18
near-significant p=0.09
near-to-significance p=0.093
near-trend significance p=0.11
nominally significant p=0.08
non-insignificant result p=0.500
non-significant in the statistical sense p>0.05
not absolutely significant but very probably so p>0.05
not as significant p=0.06
not clearly significant p=0.08
not completely significant p=0.07
not completely statistically signiﬁcant p=0.0811
not currently significant p=0.06
not decisively significant p=0.106
not entirely significant p=0.10
not especially significant p>0.05
not exactly significant p=0.052
not extremely significant p<0.06
not formally significant p=0.06
not fully significant p=0.085
not globally significant p=0.11
not highly significant p=0.089
not insignificant p=0.056
not markedly significant p=0.06
not moderately significant p>0.20
not non-significant p>0.1
not numerically significant p>0.05
not obviously signiﬁcant p>0.3
not overly significant p>0.08
not quite borderline significance p=0.089
not quite reach the level of significance p=0.07
not quite significant p=0.118
not quite within the conventional bounds of statistical significance p=0.12
not reliably signiﬁcant p=0.091
not remarkably signiﬁcant p=0.236
not significant by common standards p=0.099
not significant by conventional standards p=0.10
not significant by traditional standards p<0.1
not significant in the formal statistical sense p=0.08
not significant in the narrow sense of the word p=0.29
not significant in the normally accepted statistical sense p=0.064
not significantly significant but..clinically meaningful p=0.072
not statistically quite significant p<0.06
not strictly significant p=0.06
not strictly speaking significant p=0.057
not technically significant p=0.06
not that significant p=0.08
not to an extent that was fully statistically signiﬁcant p=0.06
not totally significant p=0.09
not unequivocally significant p=0.055
not very definitely significant p=0.08
not very definitely significant from the statistical point of view p=0.08
not very far from significance p<0.092
not very significant p=0.1
not very statistically significant p=0.10
not wholly significant p>0.1
not yet significant p=0.09
not strongly significant p=0.08
noticeably signiﬁcant p=0.055
on the border of significance p=0.063
on the borderline of significance p=0.0699
on the borderlines of significance p=0.08
on the boundaries of signiﬁcance p=0.056
on the boundary of signiﬁcance p=0.055
on the brink of significance p=0.052
on the cusp of conventional statistical significance p=0.054
on the cusp of significance p=0.058
on the edge of significance p>0.08
on the limit to significant p=0.06
on the margin of significance p=0.051
on the threshold of significance p=0.059
on the verge of significance p=0.053
on the very fringes of signiﬁcance p=0.099
only a little short of significance p>0.05
only just failed to meet statistical significance p=0.051
only just insignificant p>0.10
only marginally fails to be significant at the 95% level p=0.06
only marginally nearly insignificant p=0.059
only marginally significant p=0.9
only slightly less than significant p=0.08
only slightly missed the conventional threshold of significance p=0.062
only slightly missed the level of significance p=0.058
only slightly missed the significance level p=0.0556
only slightly non-signiﬁcant p=0.0738
only slightly significant p=0.08
partial significance p>0.09
partially significant p=0.08
partly significant p=0.08
perceivable statistical significance p=0.0501
possible significance p>0.098
possibly marginally significant p=0.116
possibly statistically significant p=0.10
potentially significant p>0.1
practically significant p=0.06
probably not experimentally significant p=0.2
probably not significant p>0.25
probably not statistically significant p=0.14
probably significant p=0.06
provisionally significant p=0.073
quasi-significant p=0.09
questionably significant p=0.13
quite close to significance at the 10% level p=0.104
quite significant p=0.07
rather marginal significance p>0.10
reached borderline significance p=0.0509
reached near significance p=0.07
reasonably significant p=0.07
remarkably close to significance p=0.05009
resides on the edge of significance p=0.10
roughly significant p>0.1
significant tendency p=0.09
significant, or close to significant effects p=0.05
significantly better overall p=0.051
significantly significant p=0.065
similar but not nonsigniﬁcant trends p>0.05
slight non-significance p=0.06
slight significance p=0.128
slight tendency toward significance p=0.086
slightly above the level of signiﬁcance p=0.06
slightly below the level of signiﬁcance p=0.068
slightly exceeded signiﬁcance level p=0.06
slightly failed to reach statistical signiﬁcance p=0.061
slightly insignificant p=0.07
slightly less than needed for significance p=0.08
slightly marginally significant p=0.06
slightly missed being of statistical significance p=0.08
slightly missed statistical significance p=0.059
slightly missed the conventional level of significance p=0.061
slightly missed the level of statistical significance p<0.10
slightly missed the margin of significance p=0.051
slightly not significant p=0.06
slightly outside conventional statistical significance p=0.051
slightly outside the margins of significance p=0.08
slightly outside the range of significance p=0.09
slightly outside the significance level p=0.077
slightly outside the statistical significance level p=0.053
slightly significant p=0.09
somewhat marginally significant p<0.055
somewhat short of significance p=0.07
somewhat significant p=0.23
somewhat statistically significant p=0.092
strong trend toward significance p=0.08
sufficiently close to significance p=0.07
suggestive but not quite significant p=0.061
suggestive of a significant trend p=0.08
suggestive of statistical significance p=0.06
suggestively significant p=0.064
tailed to insignificance p=0.1
tantalisingly close to significance p=0.104
technically not significant p=0.06
teetering on the brink of significance p=0.06
tend to significant p>0.1
tended to approach significance p=0.09
tended to be significant p=0.06
tended toward significance p=0.13
tendency toward statistical significance p=0.07
tends to approach signiﬁcance p=0.12
tentatively signiﬁcant p=0.107
too far from signiﬁcance p=0.12
trend bordering on statistical significance p=0.066
trend in a significant direction p=0.09
trend in the direction of significance p=0.089
trend significance level p=0.06
trend toward p>0.07
trending towards significance p>0.15
trending towards significant p=0.099
uncertain significance p>0.07
vaguely significant p>0.2
verged on being significant p=0.11
verging on significance p=0.056
verging on the statistically significant p<0.1
verging-on-significant p=0.06
very close to approaching significance p=0.060
very close to significant p=0.11
very close to the conventional level of significance p=0.055
very close to the cut-off for significance p=0.07
very close to the established statistical significance level of p=0.05 p=0.065
very close to the threshold of significance p=0.07
very closely approaches the conventional significance level p=0.055
very closely brushed the limit of statistical significance p=0.051
very narrowly missed significance p<0.06
very nearly significant p=0.0656
very slightly non-significant p=0.10
very slightly significant p<0.1
virtually significant p=0.059
weak significance p>0.10
weakly non-significant p=0.07
weakly significant p=0.11
weakly statistically significant p=0.0557
well-nigh signiﬁcant p=0.11
</pre>
Hankins, Matthew. "Still Not Significant." (2013) 
<a href="https://mchankins.wordpress.com/2013/04/21/still-not-significant-2/">https://mchankins.wordpress.com/2013/04/21/still-not-significant-2/</a>
</div>

---
class: normal

### Попытка использования статистических тестов №1

.up[]
```cs
// Перед изменениями
Итерация 0000:  1.011 мин
Итерация 0001:  1.014 мин
Итерация 0002:  1.021 мин
Итерация 0003:  1.017 мин
...
Итерация 9999:  1.012 мин
```

.up1[]
```cs
// После изменений
Итерация 0000: 60.127 мин
```

```md
🐙Менеджер             : У нас есть перфомансная деградация?
```
--
.up1[]
```md
🐵Перфоманс-инженер    : Думаю, что есть, но это не точно.
```
--
.up1[]
```cs
👻Статистический тест  : throw new DivideByZeroException("N should be > 1")
```

---
class: normal

### Попытка использования статистических тестов №2

.up1[]
```cs
// После изменений
Итерация 0000: 60.127 мин
Итерация 0001: 60.279 мин
Итерация 0002: 60.241 мин
```

```md
🐙Менеджер              : У нас есть перфомансная деградация?
```
--
.up1[]
```md
🐵Перфоманс-инженер     : Скорее всего.
```
--
.up1[]
```md
💩Статистический тест   : Нельзя отвергнуть нулевую гипотезу.
   
```

---
class: normal

### Попытка использования статистических тестов №3

.size-80[
.c[
.cl[
```cs
// Перед изменениями
public void Foo(object x)
{

// Некоторый код
}
```
]
.cr[
```cs
// После изменений
public void Foo(object x)
{
  `if (x == null)`
    `throw new NullReferenceException("x");`
  // Некоторый код
}
```
]
]
]

```md
🐙Менеджер              : У нас есть перфомансная деградация?
```
--
.up1[]
```md
🐵Перфоманс-инженер     : Конечно! Мы же нового кода добавили.
```
--
.up1[]
```md
💩Статистический тест   : Нельзя отвергнуть нулевую гипотезу.
```

.center[**Мы задали неправильный вопрос!**]

---
class: normal

### Попытка использования статистических тестов №4

.size-80[
.c[
.cl[
```cs
// Перед изменениями
public void Foo(object x)
{

```md
🐙Менеджер             : Насколько велика эта деградация?
```
--
.up1[]
```md
🙈Перфоманс-инженер    : Очень маленькая, можно не обращать внимания.
```
--
.up1[]
```cs
👻Статистический тест  : throw new InvalidOperationException(":(")
```
---
class: normal

### Награда по неврологии в 2012 году

<table class="invisible-table">
 <tr>
 <td><img class="scientist-photo" src="img/photos/bennett.jpg" /></td>
 <td><img class="scientist-photo" src="img/photos/baird.jpg" /></td>
 <td><img class="scientist-photo" src="img/photos/miller.jpg" /></td>
 <td><img class="scientist-photo" src="img/photos/wolford.jpg" /></td>
 </tr>
 <tr style="font-size: 90%;">
 <td align="center">Craig M. Bennett1</td>
 <td align="center">Abigail A. Baird2</td>
 <td align="center">Michael B. Miller1</td>
 <td align="center">George L. Wolford3</td>
 </tr>
</table>

<div class="bottom-hint-huge" style="text-align: left">
 .superscript[1]Department of Psychology, University of California at Santa Barbara, Santa Barbara, CA 93106 
 .superscript[2]Department of Psychology, Blodgett Hall, Vassar College, Poughkeepsie, NY 12604 
 .superscript[3]Department of Psychological and Brain Sciences, Moore Hall, Dartmouth College, Hanover, NH03755
</ul>

---
class: normal

### Функциональная МРТ мёртвого атлантического лосося

.up2[]
.center[![:scale 60%](img/photos/salmon1.png)]
.up1[]
.center[![:scale 60%](img/photos/salmon2.png)]
.bottom-hint-huge[Bennet, C., A. Baird, M. Miller, and G. Wolford. ["Neural correlates of interspecies perspective taking in the post-mortem Atlantic salmon: An argument for proper multiple comparisons correction."](https://teenspecies.github.io/pdfs/NeuralCorrelates.pdf) Journal of Serendipitous and Unexpected Results 1, no. 1 (2010): 1-5.]

---
class: normal

### Нужно запомнить

.memorize[Статистические тесты 
сложны для восприятия и использования ]

---
class: normal
count: false

### Нужно запомнить

.memorize[Статистические тесты 
сложны для восприятия и использования; 
обладают множеством скрытых ограничений ]

---
class: normal
count: false

### Нужно запомнить

.memorize[Статистические тесты 
сложны для восприятия и использования; 
обладают множеством скрытых ограничений; 
не воспроизводимы ]

---
class: normal
count: false

### Нужно запомнить

.memorize[Статистические тесты 
сложны для восприятия и использования; 
обладают множеством скрытых ограничений; 
не воспроизводимы; 
отвечают не на тот вопрос ]

---
class: normal
count: false

### Нужно запомнить

---
class: normal

### Нужно запомнить

.memorize[Статистические тесты не нужны в перфоманс-анализе]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
 Но если тесты на статистическую значимость не нужны, то как распределения-то сравнивать?
]

---
template: template-section

## Часть 4
## Сравниваем перфомансные распределения

---
layout: true
template: template-page

<div>.footer-note[(4) Сравниваем перфомансные распределения]</div>

---
class: normal

### Ключевые слова для поиска

.up[]
.center[![:scale 100%](img/screenshots/google-compare.png)]

---
class: normal

### Сдвиг всего распределения

.up[]
.center[![:scale 55%](img/ggplot/compare1.svg)]

---
class: normal

### Сдвиг одной моды

.up[]
.center[![:scale 55%](img/ggplot/compare2.svg)]

---
class: normal

### Сдвиг двух мод

.up[]
.center[![:scale 55%](img/ggplot/compare3.svg)]

---
class: normal

### Функция сдвига

.up[]
.center[![:scale 100%](img/ggplot/shift-intro1.svg)]

---
class: normal
count: false

### Функция сдвига

.up[]
.center[![:scale 100%](img/ggplot/shift-intro2.svg)]

---
class: normal
count: false

### Функция сдвига

.up[]
.center[![:scale 100%](img/ggplot/shift-intro3.svg)]

---
class: normal
count: false

### Функция сдвига

.up[]
.center[![:scale 100%](img/ggplot/shift-intro4.svg)]

---
class: normal
count: false

### Функция сдвига

.up[]
.center[![:scale 100%](img/ggplot/shift-intro5.svg)]

---
class: normal
count: false

### Функция сдвига

.up[]
.center[![:scale 100%](img/ggplot/shift-intro6.svg)]

---
class: normal
count: false

### Функция сдвига

.up[]
.center[![:scale 100%](img/ggplot/shift-intro7.svg)]

---
class: normal

### Функция сдвига

.up2[]
.center[![:scale 78%](img/ggplot/shift.svg)]

---
class: normal

### Функция пропорции

.up[]
.center[![:scale 78%](img/ggplot/ratio.svg)]

---
class: normal

### Сдвиг средней моды

.up[]
.center[![:scale 55%](img/ggplot/compare4.svg)]

---
class: normal

### Статистическое безобразие

.up[]
.center-wide[
.pull-left[![:scale 96%](img/ggplot/compare5.svg)]
.pull-right[![:scale 96%](img/ggplot/compare6.svg)]
]

---
class: normal

### Метрики в текстовом виде

.size-150[
```md
| Метод | Пропорция |
|-------|-----------|
| A     | Эталон    |
| B     | `1.5-3.0`   |
| C     | `1.2-1.4`   |
| D     | `1.2-1.3`   |
```
]

```md
🐙Менеджер             : У нас точно есть все эти деградации?
```
--
.up1[]
```md
🐵Перфоманс-инженер    : (B,C) Скорее всего, данных довольно много.
```
--
.up1[]
```md
🐵Перфоманс-инженер    : (D) Непонятно, надо бы собрать больше данных.
```

---
class: normal

### Статистика должна быть дружелюбной

.size-150[
```md
| Метод | Пропорция | Точно ли есть деградация? |
|-------|-----------|---------------------------|
| A     | Эталон    |                           |
| B     | 1.5-3.0   | `Скорее всего`              |
| C     | 1.2-1.4   | `Скорее всего`              |
| D     | 1.2-1.3   | `🤷‍♀️ Нужно больше данных`     |
```
]

---
class: normal

### Нужно запомнить

.memorize[Не "деградировали ли мы?", а "насколько мы деградировали?"]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
 А если у меня есть история замеров? 
 Как в ней проблемы искать?
]

---
template: template-section

## Часть 5
## Анализируем историю замеров

---
layout: true
template: template-page

<div>.footer-note[(5) Анализируем историю замеров]</div>

---
class: normal

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes0.svg)]

---
class: normal
count: false

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes1.svg)]

---
class: normal
count: false

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes2.svg)]

---
class: normal
count: false

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes3.svg)]

---
class: normal
count: false

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes4.svg)]

---
class: normal
count: false

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes5.svg)]

---
class: normal
count: false

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes6.svg)]

---
class: normal
count: false

### Изменения бывают разными

.up[]
.center[![:scale 100%](img/ggplot/various-changes7.svg)]

---
class: normal

### Много разных алгоритмов

.up[]
.center[![:scale 95%](img/screenshots/cpd-overview.png)]

.bottom-hint-huge[Truong, Charles, Laurent Oudre, and Nicolas Vayatis. ["Selective review of offline change point detection methods."](https://arxiv.org/pdf/1801.00718.pdf) Signal Processing 167 (2020): 107299.]

---
class: normal

### Хороший алгоритм: ED-PELT

.center[![:scale 90%](img/screenshots/edpelt.png)]

.bottom-hint-huge[Haynes, Kaylea, Paul Fearnhead, and Idris A. Eckley. ["A computationally efficient nonparametric approach for changepoint detection."](https://link.springer.com/article/10.1007/s11222-016-9687-5) Statistics and Computing 27, no. 5 (2017): 1293-1305.]

---
class: normal

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic0.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic1.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic2.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic3.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic4.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic5.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic6.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic7.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic8.svg)]

---
class: normal
count: false

### Динамическое программирование

.up[]
.center[![:scale 100%](img/diagrams/dynamic9.svg)]

---
class: normal

### Pruned exact linear time (PELT)

.up[]
.center[![:scale 100%](img/diagrams/pelt0.svg)]

.bottom-hint-huge[Killick, R., Fearnhead, P., Eckley, I.A. ["Optimal detection of changepoints with a linear computational cost."](https://arxiv.org/pdf/1101.1438.pdf) J. Am. Stat. Assoc. 107(500), 1590–1598 (2012)]

---
class: normal
count: false

### Pruned exact linear time (PELT)

.up[]
.center[![:scale 100%](img/diagrams/pelt1.svg)]

---
class: normal

### Функция стоимости на эмпирическом распределении

.center[Вводим обозначения:]

$$
\{x_j\}_n - \textrm{данные}, \quad \{\tau_i\} - \textrm{индексы точек разладки}
$$

.center[Задаём эмпирическую функцию распределения:]

$$
\hat{F}_i(t) {=}\frac{1}{\tau _i-\tau _{i-1}} \times \left( \sum _{j=\tau _{i-1}+1}^{\tau _i} \mathbf {1}\{x_j < t\} + 0.5 \times \mathbf {1}\{x_j {=} t\} \right)
$$

.center[Задаём непараметрическую функцию максимального правдоподобия:]

\begin{aligned}&\mathcal {L}_{np}(x_{\tau _{i-1}+1:\tau _i}|t) = (\tau _{i} - \tau _{i-1})\times [\hat{F}_i(t) \log \hat{F}_i(t) \nonumber +\,(1-\hat{F}_i(t)) \log (1-\hat{F}_i(t))] \end{aligned}

---
class: normal

### Функция стоимости на эмпирическом распределении

.center[Считаем K квантилей по всей выборке:]

$$
t_k = (1 + (2n - 1) \cdot e^{\frac{c}{K}(2k-1)})^{-1}, \quad c = -log(2n - 1)
$$

.center[Назначаем штраф (модифицированный Байесовский информационный критерий):]

$$
\textit{penalty} = 3 \cdot log(n)
$$

.center[**Определяем функцию стоимости:**]

$$
w(j, i) = \frac{-2c}{K} \sum_{k=1}^{K} \mathcal {L}_{np}(x_{\tau_j+1:\tau _i}|t_k) + \textit{penalty}
$$

---
class: normal

### Запускаем ED-PELT короткой истории

.up[]
.center[![:scale 100%](img/ggplot/check-edpelt1.svg)]

---
class: normal

### Запускаем ED-PELT на длинной истории

.up[]
.center[![:scale 100%](img/ggplot/check-edpelt2.svg)]

---
class: normal

### Функция стоимости на пересечениях

.up[]
.center[![:scale 100%](img/ggplot/overlap-intro1.svg)]

---
class: normal
count: false

### Функция стоимости на пересечениях

.up[]
.center[![:scale 100%](img/ggplot/overlap-intro2.svg)]

---
class: normal
count: false

### Функция стоимости на пересечениях

.up[]
.center[![:scale 100%](img/ggplot/overlap-intro3.svg)]

---
class: normal
count: false

### Функция стоимости на пересечениях

.up[]
.center[![:scale 100%](img/ggplot/overlap-intro4.svg)]

---
class: normal
count: false

### Функция стоимости на пересечениях

.up[]
.center[![:scale 100%](img/ggplot/overlap-intro5.svg)]

---
class: normal
count: false

### Функция стоимости на пересечениях

.up[]
.center[![:scale 100%](img/ggplot/overlap-intro6.svg)]

---
class: normal

### Функция стоимости на пересечениях

.overlap-table[

]

---
class: normal
count: false

### Функция стоимости на пересечениях

.overlap-table[

]

---
class: normal
count: false

### Функция стоимости на пересечениях

.overlap-table[

]

---
class: normal
count: false

### Функция стоимости на пересечениях

.overlap-table[

]

---
class: normal

### Функция стоимости на пересечениях

.up[]
.center[![:scale 100%](img/ggplot/overlap-quantile.svg)]

---
class: normal

### Поиск квантилей на отрезке (RQQ)

.up[]
.center[![:scale 60%](img/screenshots/rqq.png)]
.bottom-hint-huge[Gagie, Travis, Simon J. Puglisi, and Andrew Turpin. ["Range quantile queries: Another virtue of wavelet trees."](https://arxiv.org/pdf/0903.4726.pdf) In International Symposium on String Processing and Information Retrieval, pp. 1-6. Springer, Berlin, Heidelberg, 2009.]

---
class: normal

### Сбалансированное вейвлет-дерево

.up3[]

.size-75[
.center[
<pre><code class="md hljs markdown remark-code"> ┌─────────────────────┐ 
 │ 6 2 0 7 9 3 1 8 5 4 │ 
 │ │ 
 └─────────────────────┘

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

.size-75[
.center[
<pre><code class="md hljs markdown remark-code"> ┌─────────────────────┐ 
 │ 6 2 0 7 9 3 1 8 5 4 │ 
 │ R L L R R L L R R L │ 
 └┬───────────────────┬┘ 
 │ │ 
 ┌──────────┴┐ ┌┴──────────┐ 
 │ 2 0 3 1 4 │ │ 6 7 9 8 5 │ 
 │ │ │ │ 
 └───────────┘ └───────────┘

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

.size-75[
.center[
<pre><code class="md hljs markdown remark-code"> ┌─────────────────────┐ 
 │ 6 2 0 7 9 3 1 8 5 4 │ 
 │ R L L R R L L R R L │ 
 └┬───────────────────┬┘ 
 │ │ 
 ┌──────────┴┐ ┌┴──────────┐ 
 │ 2 0 3 1 4 │ │ 6 7 9 8 5 │ 
 │ L L R L R │ │ L L R R L │ 
 └───────────┘ └───────────┘

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

.size-75[
.center[
<pre><code class="md hljs markdown remark-code"> ┌─────────────────────┐ 
 │ 6 2 0 7 9 3 1 8 5 4 │ 
 │ R L L R R L L R R L │ 
 └┬───────────────────┬┘ 
 │ │ 
 ┌──────────┴┐ ┌┴──────────┐ 
 │ 2 0 3 1 4 │ │ 6 7 9 8 5 │ 
 │ L L R L R │ │ L L R R L │ 
 └┬─────────┬┘ └┬─────────┬┘ 
 │ │ │ │ 
 ┌─────┴─┐ ┌─┴───┐ ┌─────┴─┐ ┌─┴───┐ 
 │ 2 0 1 │ │ 3 4 │ │ 6 7 5 │ │ 9 8 │ 
 │ │ │ │ │ │ │ │ 
 └───────┘ └─────┘ └───────┘ └─────┘

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

.size-75[
.center[
<pre><code class="md hljs markdown remark-code"> ┌─────────────────────┐ 
 │ 6 2 0 7 9 3 1 8 5 4 │ 
 │ R L L R R L L R R L │ 
 └┬───────────────────┬┘ 
 │ │ 
 ┌──────────┴┐ ┌┴──────────┐ 
 │ 2 0 3 1 4 │ │ 6 7 9 8 5 │ 
 │ L L R L R │ │ L L R R L │ 
 └┬─────────┬┘ └┬─────────┬┘ 
 │ │ │ │ 
 ┌─────┴─┐ ┌─┴───┐ ┌─────┴─┐ ┌─┴───┐ 
 │ 2 0 1 │ │ 3 4 │ │ 6 7 5 │ │ 9 8 │ 
 │ R L L │ │ L R │ │ L R L │ │ R L │ 
 └┬─────┬┘ └┬───┬┘ └┬─────┬┘ └┬───┬┘ 
 │ │ │ │ │ │ │ │ 
 ┌────┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐ ┌────┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐
 │ 0 1 │ │ 2 │ │ 3 │ │ 4 │ │ 6 5 │ │ 7 │ │ 8 │ │ 9 │
 │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
 └─────┘ └───┘ └───┘ └───┘ └─────┘ └───┘ └───┘ └───┘

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

.size-75[
.center[
<pre><code class="md hljs markdown remark-code"> ┌─────────────────────┐ 
 │ 6 2 0 7 9 3 1 8 5 4 │ 
 │ R L L R R L L R R L │ 
 └┬───────────────────┬┘ 
 │ │ 
 ┌──────────┴┐ ┌┴──────────┐ 
 │ 2 0 3 1 4 │ │ 6 7 9 8 5 │ 
 │ L L R L R │ │ L L R R L │ 
 └┬─────────┬┘ └┬─────────┬┘ 
 │ │ │ │ 
 ┌─────┴─┐ ┌─┴───┐ ┌─────┴─┐ ┌─┴───┐ 
 │ 2 0 1 │ │ 3 4 │ │ 6 7 5 │ │ 9 8 │ 
 │ R L L │ │ L R │ │ L R L │ │ R L │ 
 └┬─────┬┘ └┬───┬┘ └┬─────┬┘ └┬───┬┘ 
 │ │ │ │ │ │ │ │ 
 ┌────┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐ ┌────┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐
 │ 0 1 │ │ 2 │ │ 3 │ │ 4 │ │ 6 5 │ │ 7 │ │ 8 │ │ 9 │
 │ L R │ │ │ │ │ │ │ │ R L │ │ │ │ │ │ │
 └┬───┬┘ └───┘ └───┘ └───┘ └┬───┬┘ └───┘ └───┘ └───┘
 │ │ │ │ 
┌──┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐ 
│ 0 │ │ 1 │ │ 5 │ │ 6 │ 
│ │ │ │ │ │ │ │ 
└───┘ └───┘ └───┘ └───┘

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

</code></pre>
]
]

---
class: normal
count: false

### Сбалансированное вейвлет-дерево

.up3[]

.size-75[
.center[
<pre><code class="md hljs markdown remark-code"> ┌─────────────────────┐ 
 │ `6 2 0 7 9 3 1 8 5 4` │ 
 │ R L L R R L L R R L │ 
 └┬───────────────────┬┘ 
 │ │ 
 ┌──────────┴┐ ┌┴──────────┐ 
 │ 2 0 3 1 4 │ │ 6 7 9 8 5 │ 
 │ L L R L R │ │ L L R R L │ 
 └┬─────────┬┘ └┬─────────┬┘ 
 │ │ │ │ 
 ┌─────┴─┐ ┌─┴───┐ ┌─────┴─┐ ┌─┴───┐ 
 │ 2 0 1 │ │ 3 4 │ │ 6 7 5 │ │ 9 8 │ 
 │ R L L │ │ L R │ │ L R L │ │ R L │ 
 └┬─────┬┘ └┬───┬┘ └┬─────┬┘ └┬───┬┘ 
 │ │ │ │ │ │ │ │ 
 ┌────┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐ ┌────┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐
 │ 0 1 │ │ `2` │ │ `3` │ │ `4` │ │ 6 5 │ │ `7` │ │ `8` │ │ `9` │
 │ L R │ │ │ │ │ │ │ │ R L │ │ │ │ │ │ │
 └┬───┬┘ └───┘ └───┘ └───┘ └┬───┬┘ └───┘ └───┘ └───┘
 │ │ │ │ 
┌──┴┐ ┌┴──┐ ┌──┴┐ ┌┴──┐ 
│ `0` │ │ `1` │ │ `5` │ │ `6` │ 
│ │ │ │ │ │ │ │ 
└───┘ └───┘ └───┘ └───┘

</code></pre>
]
]

---
class: normal

### Запускаем ED-PELT и RQQ-PELT

.up[]
.center[![:scale 100%](img/ggplot/check-pelts.svg)]

---
class: normal

### Как начать всё это использовать?

.center[**Фаза 1: Просто попробуйте**]

* R : https://cran.r-project.org/web/packages/changepoint.np/index.html
* Command-line, C#: https://perfolizer.com

.center[**Фаза 2: Анализируйте ограничения**]

* Сколько у вас тестов?

* Сколько у вас замеров для каждого теста?

* Какие изменения вы хотите находить?

* Сколько времени вы готовы тратить на поиск?

---
class: normal

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget1.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget2.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget3.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget4.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget5.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget6.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget7.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget8.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal
count: false

### История про медленный NuGet-сервер

.up[]
.center[![:scale 90%](img/ggplot/nuget9.svg)]

.bottom-hint-huge[Andrey Akinshin. "A story about slow NuGet package browsing." (2018) https://aakinshin.net/posts/nuget-package-browsing/]

---
class: normal

### Нужно запомнить

.memorize[Анализировать историю полезно]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
 У меня нашлось 600 точек разладки. 
 Что с ними делать?
]

---
template: template-section

## Часть 6
## Анализируем точки разладки

---
layout: true
template: template-page

<div>.footer-note[(6) Анализируем точки разладки]</div>

---
class: normal

### Сводная таблица

.up[]
.center[**Худшие деградации**]
.up1[]

| Тест | Сдвиг (секунды) | Пропорция |
| ------------------- | ---------------------: | ----------------------------------------: |
| .color-bad[Test472] | .color-bad[19.7..22.4] | 35.3..48.9 |
| .color-bad[Test982] | .color-bad[0.8.. 1.5] | 10.0..12.0 |
| Test872 | 0.2.. 0.3 | 1.1.. 1.2 |
| Test375 | 0.1.. 0.2 | 1.0.. 1.1 |
| Test184 | -0.1.. 0.0 | 0.9.. 1.0 |
| Test592 | -0.1.. 0.0 | 0.9.. 1.0 |
| Test824 | -0.1.. 0.0 | 0.9.. 1.0 |
| Test294 | -0.1.. 0.0 | 0.9.. 1.0 |
| Test235 | -0.1.. 0.0 | 0.9.. 1.0 |
| Test948 | -0.1.. 0.0 | 0.9.. 1.0 |

---
class: normal

### Сводные таблицы по целому распределению

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/dashboard-variance.svg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Дисперсия</td>
 <td></td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Сводные таблицы по целому распределению

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/dashboard-variance.svg" /></td>
 <td><img src="img/ggplot/dashboard-multimodal.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Дисперсия</td>
 <td>Мультимодальность</td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Сводные таблицы по целому распределению

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/dashboard-variance.svg" /></td>
 <td><img src="img/ggplot/dashboard-multimodal.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Дисперсия</td>
 <td>Мультимодальность</td>
 </tr>
 <tr>
 <td><img src="img/ggplot/dashboard-percentile.svg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Процентиль</td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Сводные таблицы по целому распределению

.dashboard-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/ggplot/dashboard-variance.svg" /></td>
 <td><img src="img/ggplot/dashboard-multimodal.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Дисперсия</td>
 <td>Мультимодальность</td>
 </tr>
 <tr>
 <td><img src="img/ggplot/dashboard-percentile.svg" /></td>
 <td><img src="img/ggplot/dashboard-outliers.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Процентиль</td>
 <td>Выбросы</td>
 </tr>
</table>

]

---
class: normal

### Ищем разладки в реальной жизни

.up[]
.center[![:scale 72%](img/ggplot/weekend1.svg)]

???

В теории нет разницы между теорией и практикой. А на практике есть.

---
class: normal
count: false

### Ищем разладки в реальной жизни

.up[]
.center[![:scale 72%](img/ggplot/weekend2.svg)]

.up[]
```cs
var changePoints = GetChangePoints(durations);
for (var cp in changePoints)
  if (IsNotReported(cp))
    SendAlarm(cp);
```

---
class: normal
count: false

### Ищем разладки в реальной жизни

.up[]
.center[![:scale 72%](img/ggplot/weekend3.svg)]

.up[]
```cs
var changePoints = GetChangePoints(durations);
for (var cp in changePoints)
  if (IsNotReported(cp))
    SendAlarm(cp);
```

---
class: normal
count: false

### Ищем разладки в реальной жизни

.up[]
.center[![:scale 72%](img/ggplot/weekend4.svg)]

.up[]
```cs
var changePoints = GetChangePoints(durations);
for (var cp in changePoints)
  if (IsNotReported(cp))
    SendAlarm(cp);
```

---
class: normal
count: false

### Ищем разладки в реальной жизни

.up[]
.center[![:scale 72%](img/ggplot/weekend4.svg)]

.up[]
```cs
var changePoints = GetChangePoints(durations);
for (var cp in changePoints)
  if (IsNotReported(cp) && `cp.Date.DayOfWeek != DayOfWeek.Saturday &&`
                           `cp.Date.DayOfWeek != DayOfWeek.Monday`)
    SendAlarm(cp);
```

---
class: normal
count: false

### Ищем разладки в реальной жизни

.up[]
.center[![:scale 72%](img/ggplot/weekend5.svg)]

.up[]
```cs
var changePoints = GetChangePoints(durations);
for (var cp in changePoints)
  if (IsNotReported(cp) && cp.Date.DayOfWeek != DayOfWeek.Saturday &&
                           cp.Date.DayOfWeek != DayOfWeek.Monday)
    SendAlarm(cp);
```

---
class: normal
count: false

### Ищем разладки в реальной жизни

.up[]
.center[![:scale 72%](img/ggplot/weekend6.svg)]

---
class: normal

### Смотрим на железо

.up[]
.center[![:scale 100%](img/artworks/hardware1.png)]

---
class: normal
count: false

### Смотрим на железо

.up[]
.center[![:scale 100%](img/artworks/hardware2.png)]

---
class: normal
count: false

### Смотрим на железо

.up[]
.center[![:scale 100%](img/artworks/hardware3.png)]

---
class: normal
count: false

### Смотрим на железо

.up[]
.center[![:scale 100%](img/artworks/hardware4.png)]

---
class: normal
count: false

### Смотрим на железо

.up[]
.center[![:scale 100%](img/artworks/hardware5.png)]

---
class: normal

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress0.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress1.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress2.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress3.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress4.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress5.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress6.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress7.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress8.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress9.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress10.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress11.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress12.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress13.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress14.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress15.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress16.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress17.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress18.svg)]

---
class: normal
count: false

### Строим модель нагрузки

.up[]
.center[![:scale 100%](img/ggplot/memory-stress19.svg)]

---
class: normal

### Нужно запомнить

.memorize[Не используйте магические числа, используйте сортировку]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
  Не хочу узнавать о деградациях спустя неделю, хочу узнавать о них сразу!
]

---
template: template-section

## Часть 7
## Пишем перфомансные тесты

---
layout: true
template: template-page

<div>.footer-note[(7) Пишем перфомансные тесты]</div>

---
class: normal

### Абсолютное условие

.up[]
```cs
[Test, Timeout(`2000`)]
public void Benchmark()
{
    // ...
}
```

---
class: normal

### Относительное условие

.up[]
```cs
private long baseline;

public void Baseline()
{
    var sw = Stopwatch.StartNew();
    // ...
    sw.Stop();
    baseline = sw.ElapsedMilliseconds;
}
```

```cs
public void Benchmark()
{
    var sw = Stopwatch.StartNew();
    // ...
    sw.Stop();
    if (sw.ElapsedMilliseconds / baseline > `2`)
        Assert.Fail();
}
```

---
class: normal

### Адаптивное условие

```cs
public void Benchmark(string name)
{
⠀⠀⠀⠀

⠀⠀⠀⠀
⠀⠀⠀⠀
⠀⠀⠀⠀

⠀⠀⠀⠀
⠀⠀⠀⠀
}
```

---
class: normal
count: false

### Адаптивное условие

```cs
public void Benchmark(string name)
{
    var history = LoadHistoryFromDatabase(name);

⠀⠀⠀⠀
⠀⠀⠀⠀
⠀⠀⠀⠀

⠀⠀⠀⠀
⠀⠀⠀⠀
}
```

---
class: normal
count: false

### Адаптивное условие

```cs
public void Benchmark(string name)
{
    var history = LoadHistoryFromDatabase(name);

var current = new List<double>();
 for (int i = 0; i < `N`; i++)
 current.Add(Measure(name));

⠀⠀⠀⠀
⠀⠀⠀⠀
}
```

---
class: normal
count: false

### Адаптивное условие

```cs
public void Benchmark(string name)
{
    var history = LoadHistoryFromDatabase(name);

var current = new List<double>();
 for (int i = 0; i < `N`; i++)
 current.Add(Measure(name));
 
 if (HasDegradation(history, current))
 Assert.Fail();
}
```

---
class: normal

### Статистический последовательный анализ

.center[<object data="img/flowcharts/sequential1.svg" type="image/svg+xml"></object>]

---
class: normal
count: false

### Статистический последовательный анализ

.center[<object data="img/flowcharts/sequential2.svg" type="image/svg+xml"></object>]

---
class: normal
count: false

### Статистический последовательный анализ

.center[<object data="img/flowcharts/sequential3.svg" type="image/svg+xml"></object>]

---
class: normal
count: false

### Статистический последовательный анализ

.center[<object data="img/flowcharts/sequential4.svg" type="image/svg+xml"></object>]

---
class: normal
count: false

### Статистический последовательный анализ

.center[<object data="img/flowcharts/sequential5.svg" type="image/svg+xml"></object>]

---
class: normal
count: false

### Статистический последовательный анализ

.center[<object data="img/flowcharts/sequential6.svg" type="image/svg+xml"></object>]

---
class: normal
count: false

### Статистический последовательный анализ

.center[<object data="img/flowcharts/sequential7.svg" type="image/svg+xml"></object>]

---
class: normal

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision0.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision1.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision2.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision3.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision4.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision5.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision6.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision7.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision8.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decision9.svg)]

---
class: normal
count: false

### Как принять решение? (Пример)

.up[]
.center[![:scale 100%](img/ggplot/decisionF.svg)]

---
class: normal

### Реальная жизнь

.up[]
.center[![:scale 100%](img/ggplot/bimodal1.svg)]

---
class: normal
count: false

### Реальная жизнь

.up[]
.center[![:scale 100%](img/ggplot/bimodal2.svg)]

---
class: normal
count: false

### Реальная жизнь

.up[]
.center[![:scale 100%](img/ggplot/bimodal3.svg)]

---
class: normal
count: false

### Реальная жизнь

.up[]
.center[![:scale 100%](img/ggplot/bimodal4.svg)]

---
class: normal

### Нужно запомнить

.memorize[Перфоманс-тесты должны быть адаптивными]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
  Звучит сложно, вряд ли сходу получится это внедрить на нашем проекте.
]

---
template: template-section

## Часть 8
## Допускаем много ошибок

---
layout: true
template: template-page

<div>.footer-note[(8) Допускаем много ошибок]</div>

---
class: normal

### Томас Уотсон старший

.up4[]
.center-wide[![:scale 99%](img/photos/watson.jpg)]

---
class: normal

### Ошибки были допущены

.up3[]
.center[![:scale 40%](img/books/tavris.jpg)]

---
class: normal

### Написание постмортемов

.up4[]
.center-wide[![:scale 95%](img/artworks/postmortem.svg)]

---
class: normal

### Кросс-бенчмаркинг

.up3[]
.center-wide[![:scale 100%](img/artworks/ikea1.svg)]

---
class: normal
count: false

### Кросс-бенчмаркинг

.up3[]
.center-wide[![:scale 100%](img/artworks/ikea2.svg)]

---
class: normal

### Полезные советы

* Не стесняйтесь допускать ошибки

* Не стесняйтесь признавать ошибки

* Пишите постмортемы

* Используйте кросс-бенчмаркинг

* Получайте удовольствие от ошибок

.vspace5[]

> Самая волнующая фраза, какую можно услышать в науке,  
> вовсе не *«Эврика!»*, а *«Это забавно…»*  
>  
> — Айзек Азимов.superscript[1]

.bottom-hint-huge[.superscript[1]https://quoteinvestigator.com/2015/03/02/eureka-funny/]

---
class: normal

### Карл Поппер

.up3[]
.center-wide[![:scale 138%](img/photos/popper.jpg)]

???
Однажды у Карла Поппера спросили, можно ли «фальсифицировать фальсификацию».
То есть можно ли проявить скептицизм по отношению к самой концепции скептицизма.
Карл Поппер ответил, что выгонял студентов с лекций и за менее тупые вопросы.

---
class: normal

### Нужно запомнить

.memorize[Допускать ошибки — это круто!]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
 Я одну половину доклада не запомнил, а другую не понял. 
 Что бы мне почитать на тему перфоманс-анализа?
]

---
template: template-section

## Часть 9
## Читаем методическую литературу

---
layout: true
template: template-page

<div>.footer-note[(9) Читаем методическую литературу]</div>

---
class: normal

### Хорошие книжки

.up[]

---
class: normal

### Про способность находить закономерности

.cite-block[

]

---
class: normal

### Про статистические тесты

.cite-block[

* Cohen, Jacob. "The earth is round (p<. 05)." In What if there were no significance tests?, pp. 69-82. Routledge, 2016. 
 https://pdfs.semanticscholar.org/fa63/cbf9b514a9bc4991a0ef48542b689e2fa08d.pdf
* Amrhein, Valentin, Fränzi Korner-Nievergelt, and Tobias Roth. "The earth is flat (p> 0.05): significance thresholds and the crisis of unreplicable research." PeerJ 5 (2017): e3544. 
 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5502092/
* Wasserstein, Ronald L., Allen L. Schirm, and Nicole A. Lazar. "Moving to a world beyond “p< 0.05”." (2019): 1-19. 
 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913
* Winder, W. C. "What you always wanted to know about testing but were afraid to ask." American dairy review (1973). 
 https://www.researchgate.net/publication/241372934
* Grieve, Andrew P. "How to test hypotheses if you must." Pharmaceutical statistics 14, no. 2 (2015): 139-150. 
 http://doi.wiley.com/10.1002/pst.1667
* Krawczyk, Michał. "The search for significance: a few peculiarities in the distribution of P values in experimental psychology literature." PloS one 10, no. 6 (2015). 
 https://dx.plos.org/10.1371/journal.pone.0127872

]

---
class: normal

### Про cтатистический последовательный анализ

.cite-block[

* Lakens, Daniël. "Performing high‐powered studies efficiently with sequential analyses." European Journal of Social Psychology 44, no. 7 (2014): 701-710.  
  http://doi.wiley.com/10.1002/ejsp.2023

]

---
class: normal

### Про размер эффекта

.cite-block[

* Peng, Chao-Ying Joanne, and Li-Ting Chen. "Beyond Cohen's d: Alternative effect size measures for between-subject designs." The Journal of Experimental Education 82, no. 1 (2014): 22-50.  
  http://www.tandfonline.com/doi/abs/10.1080/00220973.2012.745471
* Wilcox, Rand. "A Robust Nonparametric Measure of Effect Size Based on an Analog of Cohen's d, Plus Inferences About the Median of the Typical Difference." Journal of Modern Applied Statistical Methods 17, no. 2 (2019): 1.  
  https://digitalcommons.wayne.edu/jmasm/vol17/iss2/1
* Tomczak, Maciej, and Ewa Tomczak. "The need to report effect size estimates revisited. An overview of some recommended measures of effect size." (2014).  
  https://www.researchgate.net/publication/303919832

]

---
class: normal

### Про функцию сдвига

.cite-block[

* Doksum, Kjell. "Empirical probability plots and statistical inference for nonlinear models in the two-sample case." The annals of statistics (1974): 267-277.  
  https://doi.org/10.1214/aos/1176342662
* Doksum, Kjell A., and Gerald L. Sievers. "Plotting with confidence: Graphical comparisons of two populations." Biometrika 63, no. 3 (1976): 421-434.  
  https://doi.org/10.2307/2335720
* Rousselet, Guillaume. "the shift function: a powerful tool to compare two entire distributions." 2016  
  https://garstats.wordpress.com/2016/07/12/shift-function/
* Rousselet, Guillaume. "Bayesian shift function." (2018)  
  https://garstats.wordpress.com/2018/04/06/bsf/
* Akinshin, Andrey. "Distribution comparison via the shift and ratio functions." (2019)  
  https://aakinshin.net/posts/shift-and-ratio-functions/

]

---
class: normal

### Про разладку

.cite-block[

* Haynes, Kaylea, Paul Fearnhead, and Idris A. Eckley. "A computationally efficient nonparametric approach for changepoint detection." Statistics and Computing 27, no. 5 (2017): 1293-1305.  
  https://link.springer.com/article/10.1007/s11222-016-9687-5
* Akinshin, Andrey. "Implementation of efficient algorithm for changepoint detection: ED-PELT." (2019)  
  https://aakinshin.net/posts/edpelt/

]

---
class: normal

### Про k-й наименьший элемент

.cite-block[

* Gagie, Travis, Simon J. Puglisi, and Andrew Turpin. "Range quantile queries: Another virtue of wavelet trees." In International Symposium on String Processing and Information Retrieval, pp. 1-6. Springer, Berlin, Heidelberg, 2009.  
  https://arxiv.org/pdf/0903.4726.pdf
* Alexandrescu, Andrei. "Fast deterministic selection." arXiv preprint arXiv:1606.00484 (2016).  
  http://erdani.com/research/sea2017.pdf

]

---
class: normal

### Про квантильные оценки

.cite-block[

* Harrell, Frank E., and C. E. Davis. "A new distribution-free quantile estimator." Biometrika 69, no. 3 (1982): 635-640.  
  http://biomet.oxfordjournals.org/content/69/3/635.abstract

]

---
class: normal

### Как не надо делать

.up[]
.center[[![:scale 60%](img/screenshots/scihub.png)](https://sci-hub.se/)]

.center[[![:scale 45%](img/screenshots/scihubtw.png)](https://twitter.com/Protohedgehog/status/1005837592275881984)]

---
class: normal

### Нужно запомнить

.memorize[Никогда не переставайте изучать]

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
 Я устал. 
 Давай заканчивать, нужно передохнуть.
]

---
template: template-section

## Часть 10
## Резюмируем изученную информацию

---
layout: true
template: template-page

<div>.footer-note[(10) Резюмируем изученную информацию]</div>

---
class: normal

### Советы перфоманс-инженерам

.result-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/screenshots/samestats3.gif" /></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Образовывайтесь</td>
 <td></td>
 <td></td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Советы перфоманс-инженерам

.result-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/screenshots/samestats3.gif" /></td>
 <td><img src="img/ggplot/problems-history-square.svg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Образовывайтесь</td>
 <td>Анализируйте</td>
 <td></td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Советы перфоманс-инженерам

.result-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/screenshots/samestats3.gif" /></td>
 <td><img src="img/ggplot/problems-history-square.svg" /></td>
 <td><img src="img/ggplot/dashboard-variance-square.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Образовывайтесь</td>
 <td>Анализируйте</td>
 <td>Адаптируйтесь</td>
 </tr>
 <tr>
 <td></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td></td>
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Советы перфоманс-инженерам

.result-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/screenshots/samestats3.gif" /></td>
 <td><img src="img/ggplot/problems-history-square.svg" /></td>
 <td><img src="img/ggplot/dashboard-variance-square.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Образовывайтесь</td>
 <td>Анализируйте</td>
 <td>Адаптируйтесь</td>
 </tr>
 <tr>
 <td><img src="img/photos/engineer-cat4.jpg" /></td>
 <td></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Сомневайтесь</td>
 <td></td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Советы перфоманс-инженерам

.result-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/screenshots/samestats3.gif" /></td>
 <td><img src="img/ggplot/problems-history-square.svg" /></td>
 <td><img src="img/ggplot/dashboard-variance-square.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Образовывайтесь</td>
 <td>Анализируйте</td>
 <td>Адаптируйтесь</td>
 </tr>
 <tr>
 <td><img src="img/photos/engineer-cat4.jpg" /></td>
 <td><img src="img/photos/mac-freezer-thermal.jpg" /></td>
 <td></td>
 </tr>
 <tr class="hint-row">
 <td>Сомневайтесь</td>
 <td>Думайте</td>
 <td></td>
 </tr>
</table>

]

---
class: normal
count: false

### Советы перфоманс-инженерам

.result-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/screenshots/samestats3.gif" /></td>
 <td><img src="img/ggplot/problems-history-square.svg" /></td>
 <td><img src="img/ggplot/dashboard-variance-square.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Образовывайтесь</td>
 <td>Анализируйте</td>
 <td>Адаптируйтесь</td>
 </tr>
 <tr>
 <td><img src="img/photos/engineer-cat4.jpg" /></td>
 <td><img src="img/photos/mac-freezer-thermal.jpg" /></td>
 <td><img src="img/photos/popper-square.jpg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Сомневайтесь</td>
 <td>Думайте</td>
 <td>Ошибайтесь</td>
 </tr>
</table>

]

---
class: normal
count: false

### Советы любым инженерам

.result-table[

<table class="invisible-table">
 <tr>
 <td><img src="img/screenshots/samestats3.gif" /></td>
 <td><img src="img/ggplot/problems-history-square.svg" /></td>
 <td><img src="img/ggplot/dashboard-variance-square.svg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Образовывайтесь</td>
 <td>Анализируйте</td>
 <td>Адаптируйтесь</td>
 </tr>
 <tr>
 <td><img src="img/photos/engineer-cat4.jpg" /></td>
 <td><img src="img/photos/mac-freezer-thermal.jpg" /></td>
 <td><img src="img/photos/popper-square.jpg" /></td>
 </tr>
 <tr class="hint-row">
 <td>Сомневайтесь</td>
 <td>Думайте</td>
 <td>Ошибайтесь</td>
 </tr>
</table>

]

---
layout: true
template: template-page

---
template: template-comics

.center[![:scale 100%](img/brand/dino-comics.svg)]
.comics[
  Пишите быстрые программы!
]

<div style="position: absolute;right: 2em;bottom: 2em;">
<img src="img/favicons/home.svg" class="favicon" /><a href="https://aakinshin.net">https://aakinshin.net</a> 
<img src="img/favicons/github.svg" class="favicon" /><a href="https://github.com/AndreyAkinshin">https://github.com/AndreyAkinshin</a> 
<img src="img/favicons/twitter.svg" class="favicon" /><a href="https://twitter.com/andrey_akinshin">https://twitter.com/andrey_akinshin</a> 
<img src="img/favicons/mail.svg" class="favicon" /><a href="mailto:andrey.akinshin@gmail.com">mailto:andrey.akinshin@gmail.com<a/>
</div>

---
template: template-comics

.credits-header1[Титры]

.credits-header2[Главные герои]
<table class="credits-table">
 <tr><td>Перфолозавр</td><td><img src="img/brand/dino.svg" height="25px" /></td></tr>
 <tr><td>Перфолитка</td><td><img src="img/brand/snail.svg" height="25px" /></td></tr>
 <tr><td>Менеджер</td><td>🐙</td></tr>
 <tr><td>Перфоманс-инженер</td><td>🐵</td></tr>
 <tr><td>Статистические тесты</td><td>👻💩</td></tr>
</table>

.credits-header2[Визуальное оформление]
.credits-block[

* Beamer theme gallery: Pittsburgh 
 http://deic.uab.es/~iblanes/beamer_gallery/individual/Pittsburgh-default-default.html 
 *(Оформление слайдов)*
* Okabe, Masataka, and Kei Ito. "Color universal design (cud)-how to make figures and presentations that are friendly to colorblind people." J Fly: Data Depository for Drosophila Researchers (2008). 
 https://jfly.uni-koeln.de/color/ 
 *(Цветовая палитра графиков адаптированная для дальтоников)*

]

.credits-header2[Создание презентации]

.credits-block[

* remark: A simple, in-browser, markdown-driven slideshow tool  
  https://github.com/gnab/remark  
  *(Движок для этой презентации)*
]

.credits-header2[Графики и иллюстрации]
.credits-block[

* Wickham, Hadley. "ggplot2." Wiley Interdisciplinary Reviews: Computational Statistics 3, no. 2 (2011): 180-185. 
 https://ggplot2.tidyverse.org/ 
 *(Графики)*
* R package with functions for drawing density and frequency trail waterfall plots 
 https://github.com/AndreyAkinshin/ggwaterfall 
 *(Каскадные графики)*
* Sketchviz 
 https://sketchviz.com/ 
 *(Блок-схемы)*
* Grapholite: State-of-the-art diagrams editor for Desktop, iPad and Windows 10 tablets 
 https://grapholite.com/ 
 (Диаграммы)
* Adobe Illustrator 
 https://www.adobe.com/products/illustrator.html 
 (Иллюстрации)
* Paint.NET 
 https://www.getpaint.net/ 
 (Иллюстрации)

]

.credits-header2[Основные шрифты]
.credits-block[

* Ubuntu Mono 
 https://fonts.google.com/specimen/Ubuntu+Mono 
 (Шрифт для текста)
* Iosevka 
 https://github.com/be5invis/Iosevka 
 (Шрифт для кода)
* Caveat
 https://fonts.google.com/specimen/Caveat 
 (Шрифт для комиксов)
* Tinos
 https://fonts.google.com/specimen/Tinos 
 (Шрифт для блок-схем)

]

.credits-header2[Оборудование для эксперимента с термальным троттлингом]
.credits-block[

* MacBook Pro (Retina, 15-inch, Mid 2015, 2,5 GHz Quad-Core Intel Core i7, 16 GB 1600 MHz DDR3)
* Тепловентилятор керамический Vitek VT-2052
* Плед ИГАБРИТТА из IKEA
* Холодильник NEFF
* Тепловизор Seek Thermal Compact для Android

]

.credits-header2[Используемые и упомянутые интернет-ресурсы]
.credits-block[

* https://en.wikipedia.org/
* https://getemoji.com/
* https://sci-hub.se/
* https://twitter.com/Protohedgehog/status/1005837592275881984

]

.credits-header2[Музыка]
.credits-block[

* Музыка для титров  
  Royalty Free Music from Bensound  
  https://www.bensound.com/royalty-free-music/track/ukulele

]

.credits-header2[Источники фотографий]

.credits-block[

* Карл Поппер  
  http://www.popper-symposium.de/
* Томас Уотсон старший  
  https://ultimatepeter.com/thomas-john-watson-sr-ibm/
* Craig M. Bennett  
  https://www.brainmapping.org/directory/index.php?userid=382
* Abigail A. Baird  
  https://www.vassar.edu/faculty/abbaird/
* Michael B. Miller  
  https://www.psych.ucsb.edu/people/michael-miller
* George L. Wolford  
  https://pbs.dartmouth.edu/people/george-wolford
* Биткоины  
  Photo by Thought Catalog on Unsplash  
  https://unsplash.com/photos/bj8U389A9N8

]

.credits-header2[Благодарности людям, которые помогли подготовить материал]
.credits-block[

* Иван Пащенко
* Алина Смирнова
* Константин Хоматьяно
* Сергей Кукс

]

.credits-header2[Вдходновение для иллюстраций]

.credits-block[

* Слайд "Типичный день перфоманс-инженера"  
  Charmander the Firefighter  
  https://www.youtube.com/watch?v=nW-PA3lEqnA
* Слайд "Кросс-бенчмаркинг"  
  Инструкция по сборке IKEA  
  https://www.ikea.com/ru/ru/assembly_instructions/kallaks-stellaz__AA-1055145-6_pub.pdf

]

.credits-header2[Библиография]
.credits-block[

* Tversky, Amos, and Daniel Kahneman. "Belief in the law of small numbers." Psychological bulletin 76, no. 2 (1971): 105. 
 http://www.stats.org.uk/statistical-inference/TverskyKahneman1971.pdf
* Панчин, Александр. "Апофения." СПб.: Питер, 2019. — 256 с. ISBN 978-5-4461-1086-5 
 https://www.goodreads.com/book/show/22846024
* Taleb, Nassim. Fooled by randomness: The hidden role of chance in life and in the markets. Vol. 1. Random House Incorporated, 2005. 
 https://www.fooledbyrandomness.com/
* Cohen, Jacob. "The earth is round (p<. 05)." In What if there were no significance tests?, pp. 69-82. Routledge, 2016. 
 https://pdfs.semanticscholar.org/fa63/cbf9b514a9bc4991a0ef48542b689e2fa08d.pdf
* Amrhein, Valentin, Fränzi Korner-Nievergelt, and Tobias Roth. "The earth is flat (p> 0.05): significance thresholds and the crisis of unreplicable research." PeerJ 5 (2017): e3544. 
 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5502092/
* Wasserstein, Ronald L., Allen L. Schirm, and Nicole A. Lazar. "Moving to a world beyond “p< 0.05”." (2019): 1-19. 
 https://www.tandfonline.com/doi/full/10.1080/00031305.2019.1583913
* Winder, W. C. "What you always wanted to know about testing but were afraid to ask." American dairy review (1973). 
 https://www.researchgate.net/publication/241372934
* Grieve, Andrew P. "How to test hypotheses if you must." Pharmaceutical statistics 14, no. 2 (2015): 139-150. 
 http://doi.wiley.com/10.1002/pst.1667
* Krawczyk, Michał. "The search for significance: a few peculiarities in the distribution of P values in experimental psychology literature." PloS one 10, no. 6 (2015). 
 https://dx.plos.org/10.1371/journal.pone.0127872
* Lakens, Daniël. "Performing high‐powered studies efficiently with sequential analyses." European Journal of Social Psychology 44, no. 7 (2014): 701-710. 
 http://doi.wiley.com/10.1002/ejsp.2023
* Peng, Chao-Ying Joanne, and Li-Ting Chen. "Beyond Cohen's d: Alternative effect size measures for between-subject designs." The Journal of Experimental Education 82, no. 1 (2014): 22-50. 
 http://www.tandfonline.com/doi/abs/10.1080/00220973.2012.745471
* Wilcox, Rand. "A Robust Nonparametric Measure of Effect Size Based on an Analog of Cohen's d, Plus Inferences About the Median of the Typical Difference." Journal of Modern Applied Statistical Methods 17, no. 2 (2019): 1. 
 https://digitalcommons.wayne.edu/jmasm/vol17/iss2/1
* Tomczak, Maciej, and Ewa Tomczak. "The need to report effect size estimates revisited. An overview of some recommended measures of effect size." (2014). 
 https://www.researchgate.net/publication/303919832
* Doksum, Kjell. "Empirical probability plots and statistical inference for nonlinear models in the two-sample case." The annals of statistics (1974): 267-277. 
 https://doi.org/10.1214/aos/1176342662
* Doksum, Kjell A., and Gerald L. Sievers. "Plotting with confidence: Graphical comparisons of two populations." Biometrika 63, no. 3 (1976): 421-434. 
 https://doi.org/10.2307/2335720
* Rousselet, Guillaume. "the shift function: a powerful tool to compare two entire distributions." 2016 
 https://garstats.wordpress.com/2016/07/12/shift-function/
* Rousselet, Guillaume. "Bayesian shift function." (2018) 
 https://garstats.wordpress.com/2018/04/06/bsf/
* Akinshin, Andrey. "Distribution comparison via the shift and ratio functions." (2019) 
 https://aakinshin.net/posts/shift-and-ratio-functions/
* Haynes, Kaylea, Paul Fearnhead, and Idris A. Eckley. "A computationally efficient nonparametric approach for changepoint detection." Statistics and Computing 27, no. 5 (2017): 1293-1305. 
 https://link.springer.com/article/10.1007/s11222-016-9687-5
* Akinshin, Andrey. "Implementation of efficient algorithm for changepoint detection: ED-PELT." (2019) 
 https://aakinshin.net/posts/edpelt/
* Gagie, Travis, Simon J. Puglisi, and Andrew Turpin. "Range quantile queries: Another virtue of wavelet trees." In International Symposium on String Processing and Information Retrieval, pp. 1-6. Springer, Berlin, Heidelberg, 2009. 
 https://arxiv.org/pdf/0903.4726.pdf
* Alexandrescu, Andrei. "Fast deterministic selection." arXiv preprint arXiv:1606.00484 (2016). 
 http://erdani.com/research/sea2017.pdf
* Harrell, Frank E., and C. E. Davis. "A new distribution-free quantile estimator." Biometrika 69, no. 3 (1982): 635-640. 
 http://biomet.oxfordjournals.org/content/69/3/635.abstract
* Geary, Roy Charles. "Testing for normality." Biometrika 34, no. 3/4 (1947): 209-242.
* Anscombe, Francis J. "Graphs in statistical analysis." The american statistician 27, no. 1 (1973): 17-21. 
 https://doi.org/10.1080%2F00031305.1973.10478966
* Matejka, Justin, and George Fitzmaurice. "Same stats, different graphs: generating datasets with varied appearance and identical statistics through simulated annealing." In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pp. 1290-1294. 2017. 
 https://www.autodeskresearch.com/publications/samestats 
 http://dx.doi.org/10.1145/3025453.3025912
* Bennet, C., A. Baird, M. Miller, and G. Wolford. "Neural correlates of interspecies perspective taking in the post-mortem Atlantic salmon: An argument for proper multiple comparisons correction." Journal of Serendipitous and Unexpected Results 1, no. 1 (2010): 1-5. 
 http://dx.doi.org/10.1016/S1053-8119(09)71202-9 
 https://teenspecies.github.io/pdfs/NeuralCorrelates.pdf 
 http://prefrontal.org/files/posters/Bennett-Salmon-2009.pdf
* Perezgonzalez, Jose D. "Fisher, Neyman-Pearson or NHST? A tutorial for teaching data testing." Frontiers in Psychology 6 (2015): 223. 
 https://doi.org/10.3389/fpsyg.2015.00223

]

.credits-header2[Хорошие инструменты]

.center[![:scale 100%](img/brand/bdn.svg)]

.center[
<img src="img/brand/perfolizer.svg" style="width: 70%; padding-left: 153px; padding-top: 10px;" />
]
.center[
<img src="img/brand/rider.svg" style="width: 50%; padding-right: 106px; margin-top: -31px;" />
]
.credits-header2[Организаторы конференции]

.center[![:scale 50%](img/brand/jugru.png)]

<div style="display: block; text-align: center;margin-top: 200px;">
 © 2020 Андрей Акиньшин | <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA 4.0</a>
</div>

</div>