sehee

RNN, LSTM

Transformer model

Attention

같은 문장 내에서 단어들 간의 관계

QKV (Query, Key, Value)

예: I love her 영어를 한국어로 번역하는 것

Query: '나는'이라는 주체

Key: 연관성을 찾는 대상 (I, love, her)

Value: Q와 K의 유사성을 계산하여 유사한 만큼의 값을 가져오는 것

d_k의 루트 값으로 나누어주는 이유: d_k 값이 너무 커지게 되면 행렬의 연산 값도 커지면서 softmax 함수가 극도로 작은 기울기를 갖는 영역을 가지게될 수 있음 → 0에 가까운 값으로 정규화시킴

References

[논문리뷰] Attention is All you need의 이해 - https://lcyking.tistory.com/entry/%EB%85%BC%EB%AC%B8%EB%A6%AC%EB%B7%B0-Attention-is-All-you-need%EC%9D%98-%EC%9D%B4%ED%95%B4

(-1)^부호 x 가수 x 2^지수

가수(Mantissa, precision): 정규화된 값, 가수 비트가 많을수록 더 정밀한 값을 표현

지수(Exponent, range): 지수 비트가 많을수록 표현할 수 있는 숫자의 범위가 넓어짐

예: -12.5를 표현한다면:
- 부호: 음수 (-)
- 가수: 1.5625 (정규화된 값)

- 지수: 3 (2³ = 8을 곱함)
- 결과: -1 × 1.5625 × 8 = -12.5

Float32(FP32)

- 8 bits: 지수부(exponent)

- 23 bits: 가수부(mantissa)

- 1 bits: 부호(sign)

ML에선 FP32를 full precision(4 bytes), BF16와 FP16을 half-precision(2 bytes)라고 부름 - 기준

BF: Brain Float

C-Float8: FP8, Quarter Precision

- E4M3: 4bits range + 3bits precision

- E5M2: 5bits range + 2bits precision

UINT8: Unsigned Integer 8-bit

- 부동소수점이 아닌 정수형, 0~255 

Double Quantization

QLoRA 최적화 요소 4비트 NormalFloat (FP32 -> NF4)

정규화, 역양자화, ...

References 

TensorFloat-32 in the A100 GPU Accelerates AI Training, HPC up to 20x, https://blogs.nvidia.com/blog/tensorfloat-32-precision-format/

[Paper Review] QLoRA: Efficient Finetuning of Quantized LLMs - https://moomyung-lab.tistory.com/11

Hugo is a fast, open-source static site generator that simplifies creating websites.

brew install hugo

Creating a new hugo site

hugo new site <my-site-name>

- replace <my-site-name> with my desired project name: sarah-hugo

Installing a Theme for my hugo site

- Hugo doesn’t come with a built-in default theme
- choose a theme from hugo themes that suits my project (popular: Ananke theme (for beginners))
+ Initializing a Git Repository (in <my-site-name> folder)

cd sarah-hugo
git init
git submodule add https://github.com/theNewDynamic/gohugo-theme-ananke.git themes/ananke

Configure Hugo to use the theme by editing the hugo.toml configuration file

# vi sarah-hugo/hugo.toml
baseURL = 'https://example.org/'
languageCode = 'en-us'
title = 'My New Hugo Site'

theme = "ananke"

Creating a Simple Content File

hugo new posts/my-first-post.md

• generates a new Markdown file located in the content/posts directory

open the newly created file in my preferred text editor: 

nano content/posts/my-first-post.md

+++
date = '2025-11-05T17:45:09+09:00'
draft = true
title = 'My First Post'
+++

add some content below the front matter

# Welcome to My First Post! 

This is my very first post using Hugo and Git. Exciting times ahead!

• in nano, CTR + 0, then Enter, and then CTRL + X to exit

Staging my changes

stage all changes in my project directory by running: 

git add .

Making my first commit

git config --global user.email "my@gmail.com"
git config --global user.name "my name"

git commit -m "Add first post: My First Post"

Viewing my changes with Hugo

start the Hugo server to view the changes locally

hugo server -D

• -D flag stands for Draft

open my web browser and go to ‘http://localhost:1313’

Viewing my commit history

git log

commit axxxxxxxxxxxxxxxxx0 (HEAD → main)
Author: my name <my@gmail.com>
Date: Wed Nov 5 18:17:59 2025 +0900

Add first post: My First Post

Ignoring unnecessary files with .gitignore

it’s important to recognize that not all files need to be tracked by Git
- such as logs, build artifacts, and environment configurations

inside the root of your project directory, create a file named .gitignore: 

touch .gitignore

In this file, you can specify file patterns that Git should ignore. For example:

# Ignore log files
*.log

# Ignore Hugo build directory
public/

# Ignore temporary editor files
*.swp

References: 

From Theory to Practice: A Git Workshop for Beginners - https://hjortberg.substack.com/p/from-theory-to-practice-a-git-workshop