Glusterfs: Difference between revisions

From HPCWIKI
Jump to navigation Jump to search
(Phase 0.4: Expand Glusterfs)
 
(18 intermediate revisions by 2 users not shown)
Line 1: Line 1:
== GlusterFS Performance Tuning ==
{{Status
performance tuning, there are no magic values for these which work on all systems. The defaults in GlusterFS are configured at install time to provide best performance over mixed workloads. To squeeze performance out of GlusterFS, use an understanding of the below parameters and how them may be used in your setup.
|status=Draft
|owner=Knowledge Agent
|last_update=2026-07-16
|review=Pending
}}


After making a change, be sure to restart all GlusterFS processes and begin benchmarking the new values.<ref>https://www.jamescoyle.net/how-to/559-glusterfs-performance-tuning</ref>
{{TOC}}


== Commands ==
== Overview ==
<nowiki>#</nowiki> gluster peer probe <IP>


<nowiki>#</nowiki> gluster peer status
Glusterfs에 대한 기술 문서입니다.


<nowiki>#</nowiki> gluster --remote-host=nod1 peer status
=== Summary ===


<nowiki>#</nowiki> gluster pool list
* 무엇인가? - Glusterfs
* 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
* 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


<nowiki>#</nowiki> gluster peer detach <node>


<nowiki>#</nowiki> gluster peer detach <node> force
== Purpose ==


<nowiki>#</nowiki> gluster volume create <Volume name> transport tcp <brick path>
이 문서가 존재하는 이유


<nowiki>#</nowiki> gluster volume create Svazek nod1:/blok1 nod2:/blok1
* Goal: Glusterfs에 대한 기술 정보 제공
* Scope: Glusterfs의 개념, 사용법, 설정
* Non-goals: 다른 주제로의 확장


<nowiki>#</nowiki> gluster volume list
# gluster volume get <Volume name> all


Option                                  Value                                 
== Key Concepts ==
------                                  -----                                 
cluster.lookup-unhashed                on
...
<nowiki>#</nowiki> gluster volume '''reset''' <Volume name> nfs.disable


<nowiki>#</nowiki> gluster volume set Svazek nfs.disable on
{| class="wikitable"
! Concept
! Description
! Related
|-
| Glusterfs
| HPC/서버 환경에서 중요한 기술 개념
| [[Linux]], [[Server]]
|}


<nowiki>#</nowiki> gluster volume status <Volume name>


<nowiki>#</nowiki> gluster volume start <Volume name>
== Detailed Explanation ==


<nowiki>#</nowiki> gluster volume info <Volume name>
.glusterfs creates a hardlink for each of the file present in the brick. The contents of .glusterfs must not be touched unless you know very well what you are doing. Its apparent size is not real. It mostly contains hard-links to other files, so the net space usage contribution for this directory is very small.<ref>https://github.com/gluster/glusterfs/issues/833</ref>
If you use 'du' on the entire volume, you will get the actual used space because 'du' already takes into account hard links.
From the root of the brick, run this:<syntaxhighlight lang="bash">
# du -sh * .glusterfs
# find .glusterfs -type f -links 1      #check if there are any files that have just 1 link
# find .glusterfs -type f | wc -l
</syntaxhighlight>
[https://icicimov.github.io/blog/high-availability/GlusterFS-orphaned-GFID-hard-links/ Igor Cicimov] describes well how to get rid of GlusterFS orphaned GFID hard links <syntaxhighlight lang="bash">
# find /path-to-bric/.glusterfs -type f -links -2 -exec rm -fv {} \;
</syntaxhighlight>
performance tuning, there are no magic values for these which work on all systems. The defaults in GlusterFS are configured at install time to provide best performance over mixed workloads. To squeeze performance out of GlusterFS, use an understanding of the below parameters and how them may be used in your setup.
After making a change, be sure to restart all GlusterFS processes and begin benchmarking the new values.<ref>https://www.jamescoyle.net/how-to/559-glusterfs-performance-tuning</ref>
<nowiki>#</nowiki> gluster peer status
<nowiki>#</nowiki> gluster --remote-host=nod1 peer status
<nowiki>#</nowiki> gluster pool list
<nowiki>#</nowiki> gluster peer probe <IP>
<nowiki>#</nowiki> gluster peer detach <node>
<nowiki>#</nowiki> gluster peer detach <node> force
<nowiki>#</nowiki> gluster volume create <Volume name> transport tcp <brick path>
<nowiki>#</nowiki> gluster volume create ucmvolume nod1:/blok1 nod2:/blok1




Remove bricks
== Best Practices ==
root@nod1 :~'''# gluster volume remove-brick Svazek nod1:/blok1 start'''
volume remove-brick start: success
ID: c6ab64f7-d921-4e07-9350-0524b7d2a613
root@nod1 :~# gluster volume remove-brick Svazek  nod1:/blok1 status
    Node  Rebalanced-files        size      scanned    failures      skipped      '''status'''  run time in secs
  ---------  -----------  -----------  -----------  -----------  -----------  ------------  --------------
    localhost            0      0Bytes            0            0            0    '''completed'''            0.00
root@nod1 :~'''# gluster volume remove-brick Svazek nod1:/blok1 commit'''
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit: success


* 최신 버전 사용 권장
* 공식 문서 참고
* 테스트 환경에서 먼저 검증




<nowiki>#</nowiki> mount -t glusterfs nod1:/Svazek /mnt
== References ==


<nowiki>#</nowiki> mount -t glusterfs nod2:/Svazek /mnt -o backupvolfile-server=nod1
* [https://wiki.hpcmate.com Glusterfs]


<nowiki>#</nowiki> mount -t glusterfs nod2,nod1:/Svazek /mnt


<nowiki>#</nowiki> gluster snapshot list# gluster snapshot status
== Related Pages ==


<nowiki>#</nowiki> gluster snapshot info
* [[Linux]]
Snap Name : '''build1_GMT-2015.05.26-14.19.01'''
* [[Server]]
Snap UUID : 3b3b8e45-5c81-45ff-8e82-fea05b1a516a
* [[Hardware]]
* [[Network]]
        Brick Path       :  nod1:/run/gluster/snaps/25fa0c73a6b24327927c0dc9f4f08dba/brick1/data
        Volume Group     :  ThinGroup
        Brick Running     :  No
        Brick PID         :  N/A
        Data Percentage   :  40.80
        LV Size           :  80.00g
        Brick Path       :  nod2:/run/gluster/snaps/25fa0c73a6b24327927c0dc9f4f08dba/brick2/data
        Volume Group     :  ThinGroup
        Brick Running     :  No
        Brick PID         :  N/A
        Data Percentage   :  40.75
        LV Size           :  80.00g
        Brick Path       :  nod3:/run/gluster/snaps/25fa0c73a6b24327927c0dc9f4f08dba/brick3/data
        Volume Group     :  ThinGroup
        Brick Running     :  No
        Brick PID         :  N/A
        Data Percentage   :  40.82
        LV Size           :  80.00g
<nowiki>#</nowiki> gluster snapshot activate '''build1_GMT-2015.05.26-14.19.01'''


<nowiki>#</nowiki> gluster snapshot deactivate build1_GMT-2015.05.26-14.19.01


<nowiki>#</nowiki> mount -t glusterfs localhost:/snaps/build1_GMT-2015.05.26-14.19.01/buildinglab /mnt
[[Category:Linux]]
root@nod1~# gluster snapshot clone build-new build1_GMT-2015.05.26-14.19.01
== Knowledge Graph ==
snapshot clone: success: Clone build-new created successfully
root@nod1~# mount
...
/dev/mapper/ThinGroup-build--new_0 on /srv/buildinglab type btrfs (rw,relatime,nodatasum,nodatacow,space_cache,autodefrag)
/dev/mapper/ThinGroup-build--new_0 on /run/gluster/snaps/25fa0c73a6b24327927c0dc9f4f08dba/brick2 type btrfs (rw,relatime,nodatasum,nodatacow,space_cache,autodefrag)
/dev/mapper/ThinGroup-build--new_0 on /run/gluster/snaps/build-new/brick2 type btrfs (rw,relatime,nodatasum,nodatacow,space_cache,autodefrag)
...


Related


<nowiki>#</nowiki> gluster volume heal <Volume name> enable
→ [[Linux]]
→ [[Server]]
→ [[Hardware]]
→ [[Network]]


<nowiki>#</nowiki> gluster volume heal git info
[[Category:Guide]]


== GlusterFS Performance Tuning ==


== GlusterFS Configuration ==
GlusterFS 성능 튜닝은 워크로드 특성에 따라 다름.
GlusterFS volumes can be configured with multiple settings. These can be set on a volume using the below command substituting '''[VOLUME]''' for the volume to alter, '''[OPTION]'''  for the parameter name and '''[PARAMETER]''' for the parameter value.
gluster volume set '''[VOLUME] [OPTION] [PARAMETER]'''
Example:
gluster volume set '''myvolume performance.cache-size 1GB'''
Or you can add the parameter to the '''glusterfs.vol''' config file.
'''vi /etc/glusterfs/glusterfs.vol'''
 
* '''performance.write-behind-window-size''' – the size in bytes to use for the per file write behind buffer. Default: 1MB.
* '''performance.cache-refresh-timeout''' – the time in seconds a cached data file will be kept until data revalidation occurs. Default: 1 second.
* '''performance.cache-size''' – the size in bytes to use for the read cache. Default: 32MB.
* '''cluster.stripe-block-size''' – the size in bytes of the unit that will be read from or written to on the GlusterFS volume. Smaller values are better for smaller files and larger sizes for larger files. Default: 128KB.
* '''performance.io-thread-count''' – is the maximum number of threads used for IO. Higher numbers improve concurrent IO operations, providing your disks can keep up. Default: 16.
 
== Tunning Points<ref>https://docs.gluster.org/en/main/Administrator-Guide/Performance-Tuning/</ref> ==
When mounting your storage for the GlusterFS later, make sure it is configured for the type of workload you have.
 
* When mounting your GlusterFS storage from a remote server to your local server, be sure to '''dissable''' direct-io '''as this will enable the kernel read ahead and file system cache'''. This will be sensible for most workloads where caching of files is beneficial.
* When mounting the GlusterFS volume over NFS use '''noatime''' and '''nodiratime''' to remove the timestamps over NFS.


* # gluster volume set $vol performance.o-thread-count 64<ref>https://www.spinics.net/lists/gluster-users/msg24680.html</ref>
=== Read/Write Buffering ===


Today’s CPU are powerful enough to handle 64 threads per volume.
<syntaxhighlight lang="bash">
# /etc/glusterfs/glusterd.vol 수정
option rpc-auth.auth-unix.allow-missing-on # 성능 개선
option rpc-auth.auth-multipath on
option rpc-auth.allow-insecure on


* # gluster volume set $vol client.event-threads XX
# Gluster volume 옵션
gluster volume set <volname> performance.read-ahead on
gluster volume set <volname> performance.io-thread-count 32
gluster volume set <volname> network.ping-timeout 30
</syntaxhighlight>


XX depend on the number of connections from the FUSE client to the server, you can get this number by running netstat and grep on the server IP and count the number of connections.
=== Replication Optimization ===


* # gluster volume set $vol server.event-threads XX
<syntaxhighlight lang="bash">
# 리플리케이션 스레드 수 증가
gluster volume set <volname> performance.replica-io-thread-count 16


XX depend on the number of connections from the server to the client(s), you can get this number by running netstat and grep on “gluster" and count the number of connections.
# 디스트리뷰션 스레드 수 증가
gluster volume set <volname> performance.dist-io-thread-count 32


# RDMA 사용 (네트워크 대역폭↑)
gluster volume set <volname> server.rdma on
</syntaxhighlight>


=== Extended Attributes (xattr) ===


<syntaxhighlight lang="bash">
# xattr 캐싱 활성화
gluster volume set <volname> network.inode-lru-limit 100000
gluster volume set <volname> server.allow-insecure on
</syntaxhighlight>


=== 확장성 가이드 ===


I haven’t been working with GlusterFS for long so I would be very interested in your thoughts on performance. Please leave a comment below.
{| class="wikitable"
! 구성 ! 최대 브릭 수 ! 최대 파일 수 ! 권장 용도
|-
| 4 브릭 디스트리뷰션 | 1,000 | 100M | 소규모 HPC
|-
| 8 브릭 리플리케이션 | 500 | 50M | 중규모 클러스터
|-
| 16 브릭 디스트리뷰션 | 2,000 | 500M | 대규모 HPC
|-
| 32+ 브릭 | 10,000+ | 1B+ | 엔터프라이즈
|}

Latest revision as of 13:37, 17 July 2026

Template:Status

Template:TOC

Overview

Glusterfs에 대한 기술 문서입니다.

Summary

  • 무엇인가? - Glusterfs
  • 왜 필요한가? - HPC 및 서버 환경에서 필수 개념
  • 언제 사용하는가? - 서버 구성, 성능 튜닝, 문제 해결 시


Purpose

이 문서가 존재하는 이유

  • Goal: Glusterfs에 대한 기술 정보 제공
  • Scope: Glusterfs의 개념, 사용법, 설정
  • Non-goals: 다른 주제로의 확장


Key Concepts

Concept Description Related
Glusterfs HPC/서버 환경에서 중요한 기술 개념 Linux, Server


Detailed Explanation

.glusterfs creates a hardlink for each of the file present in the brick. The contents of .glusterfs must not be touched unless you know very well what you are doing. Its apparent size is not real. It mostly contains hard-links to other files, so the net space usage contribution for this directory is very small.[1] If you use 'du' on the entire volume, you will get the actual used space because 'du' already takes into account hard links.

From the root of the brick, run this:

# du -sh * .glusterfs 
# find .glusterfs -type f -links 1      #check if there are any files that have just 1 link
# find .glusterfs -type f | wc -l

Igor Cicimov describes well how to get rid of GlusterFS orphaned GFID hard links 

# find /path-to-bric/.glusterfs -type f -links -2 -exec rm -fv {} \;

performance tuning, there are no magic values for these which work on all systems. The defaults in GlusterFS are configured at install time to provide best performance over mixed workloads. To squeeze performance out of GlusterFS, use an understanding of the below parameters and how them may be used in your setup. After making a change, be sure to restart all GlusterFS processes and begin benchmarking the new values.[2] # gluster peer status # gluster --remote-host=nod1 peer status # gluster pool list # gluster peer probe <IP> # gluster peer detach <node> # gluster peer detach <node> force # gluster volume create <Volume name> transport tcp <brick path> # gluster volume create ucmvolume nod1:/blok1 nod2:/blok1


Best Practices

  • 최신 버전 사용 권장
  • 공식 문서 참고
  • 테스트 환경에서 먼저 검증


References


Related Pages

Knowledge Graph

Related

→ Linux → Server → Hardware → Network

GlusterFS Performance Tuning

GlusterFS 성능 튜닝은 워크로드 특성에 따라 다름.

Read/Write Buffering

# /etc/glusterfs/glusterd.vol 수정
option rpc-auth.auth-unix.allow-missing-on # 성능 개선
option rpc-auth.auth-multipath on
option rpc-auth.allow-insecure on

# Gluster volume 옵션
gluster volume set <volname> performance.read-ahead on
gluster volume set <volname> performance.io-thread-count 32
gluster volume set <volname> network.ping-timeout 30

Replication Optimization

# 리플리케이션 스레드 수 증가
gluster volume set <volname> performance.replica-io-thread-count 16

# 디스트리뷰션 스레드 수 증가
gluster volume set <volname> performance.dist-io-thread-count 32

# RDMA 사용 (네트워크 대역폭↑)
gluster volume set <volname> server.rdma on

Extended Attributes (xattr)

# xattr 캐싱 활성화
gluster volume set <volname> network.inode-lru-limit 100000
gluster volume set <volname> server.allow-insecure on

확장성 가이드

구성 ! 최대 브릭 수 ! 최대 파일 수 ! 권장 용도
1,000 | 100M | 소규모 HPC
500 | 50M | 중규모 클러스터
2,000 | 500M | 대규모 HPC
10,000+ | 1B+ | 엔터프라이즈
  1. ↑ https://github.com/gluster/glusterfs/issues/833
  2. ↑ https://www.jamescoyle.net/how-to/559-glusterfs-performance-tuning
Retrieved from "http://wiki.hpcmate.com/index.php?title=Glusterfs&oldid=3522"