> ## Documentation Index
> Fetch the complete documentation index at: https://tserjay.club/llms.txt
> Use this file to discover all available pages before exploring further.

# Flash-Atten

# 1.Flash Atten 分块

<Frame>
  <img src="https://mintcdn.com/tserjay/4FPvL9aCgc2cdC_X/images/image-1.png?fit=max&auto=format&n=4FPvL9aCgc2cdC_X&q=85&s=efe571cb5edf72429a642f124d2e4357" alt="Image" width="1246" height="582" data-path="images/image-1.png" />
</Frame>

# [2.online](http://2.online) sofxmax的原理以及公式的推导：

<Frame>
  <img src="https://mintcdn.com/tserjay/4FPvL9aCgc2cdC_X/images/image-2.png?fit=max&auto=format&n=4FPvL9aCgc2cdC_X&q=85&s=5fe5d21f90a73567f3ecb5c7777c6e04" alt="Image" width="555" height="471" data-path="images/image-2.png" />
</Frame>

对比原始的softmax，safe softmax 改进的点：online softmax将safe softmax需要二次遍历寻找最大值和求和进行了优化，将其优化成使用一次遍历求和

# Online softmax \* value

<Frame>
  <img src="https://mintcdn.com/tserjay/4FPvL9aCgc2cdC_X/images/image-3.png?fit=max&auto=format&n=4FPvL9aCgc2cdC_X&q=85&s=7479972f84d0474110561e2dfdd40be3" alt="Image" width="502" height="521" data-path="images/image-3.png" />
</Frame>

<Frame>
  <img src="https://mintcdn.com/tserjay/4FPvL9aCgc2cdC_X/images/image-4.png?fit=max&auto=format&n=4FPvL9aCgc2cdC_X&q=85&s=8ae9070e34543b044931eb0266ae7444" alt="Image" width="2268" height="4032" data-path="images/image-4.png" />
</Frame>

# Flash Atten cuda算子

<Frame>
  <img src="https://mintcdn.com/tserjay/4FPvL9aCgc2cdC_X/images/image-5.png?fit=max&auto=format&n=4FPvL9aCgc2cdC_X&q=85&s=df939e87d026751558075712d732c0f3" alt="Image" width="1212" height="690" data-path="images/image-5.png" />
</Frame>