Glusterfs

Overview

Glusterfs에 대한 기술 문서입니다.

Summary

• 무엇인가? - Glusterfs
• 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
• 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시

Purpose

이 문서가 존재하는 이유

• Goal: Glusterfs에 대한 기술 정보 제공
• Scope: Glusterfs의 개념, 사용법, 설정
• Non-goals: 다른 주제로의 확장

Key Concepts

Concept	Description	Related
Glusterfs	HPC/서버 환경에서 중요한 기술 개념	Linux, Server

Detailed Explanation

.glusterfs creates a hardlink for each of the file present in the brick. The contents of .glusterfs must not be touched unless you know very well what you are doing. Its apparent size is not real. It mostly contains hard-links to other files, so the net space usage contribution for this directory is very small. If you use 'du' on the entire volume, you will get the actual used space because 'du' already takes into account hard links.

From the root of the brick, run this:

# du -sh * .glusterfs 
# find .glusterfs -type f -links 1      #check if there are any files that have just 1 link
# find .glusterfs -type f | wc -l

Igor Cicimov describes well how to get rid of GlusterFS orphaned GFID hard links

# find /path-to-bric/.glusterfs -type f -links -2 -exec rm -fv {} \;

performance tuning, there are no magic values for these which work on all systems. The defaults in GlusterFS are configured at install time to provide best performance over mixed workloads. To squeeze performance out of GlusterFS, use an understanding of the below parameters and how them may be used in your setup. After making a change, be sure to restart all GlusterFS processes and begin benchmarking the new values. # gluster peer status # gluster --remote-host=nod1 peer status # gluster pool list # gluster peer probe <IP> # gluster peer detach <node> # gluster peer detach <node> force # gluster volume create <Volume name> transport tcp <brick path> # gluster volume create ucmvolume nod1:/blok1 nod2:/blok1

Best Practices

• 최신 버전 사용 권장
• 공식 문서 참고
• 테스트 환경에서 먼저 검증

References

• Glusterfs

Related Pages

• Linux
• Server
• Hardware
• Network

Knowledge Graph

Related

→ Linux → Server → Hardware → Network

GlusterFS Performance Tuning

GlusterFS 성능 튜닝은 워크로드 특성에 따라 다름.

Read/Write Buffering

# /etc/glusterfs/glusterd.vol 수정
option rpc-auth.auth-unix.allow-missing-on # 성능 개선
option rpc-auth.auth-multipath on
option rpc-auth.allow-insecure on

# Gluster volume 옵션
gluster volume set <volname> performance.read-ahead on
gluster volume set <volname> performance.io-thread-count 32
gluster volume set <volname> network.ping-timeout 30

Replication Optimization

# 리플리케이션 스레드 수 증가
gluster volume set <volname> performance.replica-io-thread-count 16

# 디스트리뷰션 스레드 수 증가
gluster volume set <volname> performance.dist-io-thread-count 32

# RDMA 사용 (네트워크 대역폭↑)
gluster volume set <volname> server.rdma on

Extended Attributes (xattr)

# xattr 캐싱 활성화
gluster volume set <volname> network.inode-lru-limit 100000
gluster volume set <volname> server.allow-insecure on

확장성 가이드

구성 ! 최대 브릭 수 ! 최대 파일 수 ! 권장 용도
1,000 | 100M | 소규모 HPC
500 | 50M | 중규모 클러스터
2,000 | 500M | 대규모 HPC
10,000+ | 1B+ | 엔터프라이즈