由于网站使用nfs共享方式保存用户上传的图片,附件等资料,然后通过apache下载的方式供用户访问,在网站架构初期,使用这种简单的方式实现了静态资源的读写分离,但随着网站数据量的增加,图片服务器渐渐成为整个网站的短板,缘次催生了使用fastfds的想法,故而先进行一番简单的测试!在开始之前还是先来看看fastfds的介绍信息:
FastDFS is an open source high performance distributed file system (DFS). It's major functions include: file storing, file syncing and file accessing, and design for high capacity and load balancing.
FastDFS is an open source high performance distributed file system. It's major functions include: file storing, file syncing and file accessing (file uploading and file downloading), and it can resolve the high capacity and load balancing problem. FastDFS should meet the requirement of the website whose service based on files such as photo sharing site and vidio sharing site.
FastDFS has two roles: tracker and storage. The tracker takes charge of scheduling and load balancing for file access. The storage store files and it's function is file management including: file storing, file syncing, providing file access interface. It also manage the meta data which are attributes representing as key value pair of the file. For example: width=1024, the key is "width" and the value is "1024".
The tracker and storage contain one or more servers. The servers in the tracker or storage cluster can be added to or removed from the cluster by any time without affecting the online services. The servers in the tracker cluster are peer to peer.
The storarge servers organizing by the file volume/group to obtain high capacity. The storage system contains one or more volumes whose files are independent among these volumes. The capacity of the whole storage system equals to the sum of all volumes' capacity. A file volume contains one or more storage servers whose files are same among these servers. The servers in a file volume backup each other, and all these servers are load balancing. When adding a storage server to a volume, files already existing in this volume are replicated to this new server automatically, and when this replication done, system will switch this server online to providing storage services. When the whole storage capacity is insufficiency, you can add one or more volumes to expand the storage capacity. To do this, you need to add one or more storage servers.
The identification of a file is composed of two parts: the volume name and the file name.
|
大意为:
fastdfs是一个开源的,高性能的的分布式文件系统,他主要的功能包括:文件存储,同步和访问,设计基于高可用和负载均衡,fastfd非常适用于基于文件服务的站点,例如图片分享和视频分享网站
fastfds有两个角色:跟踪服务和存储服务,跟踪服务控制,调度文件以负载均衡的方式访问;存储服务包括:文件存储,文件同步,提供文件访问接口,同时以key value的方式管理文件的元数据
跟踪和存储服务可以由1台或者多台服务器组成,同时可以动态的添加,删除跟踪和存储服务而不会对在线的服务产生影响,在集群中,tracker服务是对等的
存储系统由一个或多个卷组成,卷与卷之间的文件是相互独立的,所有卷的文件容量累加就是整个存储系统中的文件容量。一个卷可以由一台或多台存储服务器组成,一个卷下的存储服务器中的文件都是相同的,卷中的多台存储服务器起到了冗余备份和负载均衡的作用。在卷中增加服务器时,同步已有的文件由系统自动完成,同步完成后,系统自动将新增服务器切换到线上提供服务。当存储空间不足或即将耗尽时,可以动态添加卷。只需要增加一台或多台服务器,并将它们配置为一个新的卷,这样就扩大了存储系统的容量。
下面几张图可以清楚的说明fastfds的架构和文件上传和下载流程等:
下面将介绍下fastdfs在rhel上的部署过程
tracker服务器:192.168.123.110/24
storage服务器:192.168.123.20/24
一:编译安装
- [root@db1 ~]# wget http://cloud.github.com/downloads/libevent/libevent/libevent-2.0.21
- -stable.tar.gz
- [root@db1 ~]# tar -zxvpf libevent-2.0.21-stable.tar.gz
- [root@db1 ~]# cd libevent-2.0.21-stable
-
[root@db1 libevent-2.0.21-stable]# ./configure --prefix=/usr/local/libevent-2.0.21 &&
- make && make install
- [root@db1 ~]# wget http://fastdfs.googlecode.com/files/FastDFS_v3.02.tar.gz
- [root@db1 ~]# tar -zxvf FastDFS_v3.02.tar.gz
- [root@db1 ~]# cd FastDFS
- [root@db1 FastDFS]# grep -A 4 '/usr/local/FastDFS' make.sh
-
TARGET_PREFIX=/usr/local/FastDFS
-
TARGET_CONF_PATH=/etc/fdfs
-
WITH_HTTPD=1
-
WITH_LINUX_SERVICE=1
-
[root@db1 FastDFS]# ./make.sh C_INCLUDE_PATH=/usr/local/libevent-2.0.14/include
-
LIBRARY_PATH=/usr/local/libevent-2.0.14/lib
- [root@db1 FastDFS]# ./make.sh install
- [root@db1 FastDFS]# ls /etc/fdfs/
- client.conf http.conf mime.types storage.conf tracker.conf
二:tracker配置文件
- [root@db1 ~]# grep -v '^#' /etc/fdfs/tracker.conf |grep -v '^$'
-
disabled=false
-
bind_addr=192.168.123.110
-
port=22122
-
connect_timeout=30
-
network_timeout=60
-
base_path=/home/data/fastdfs
-
max_connections=256
-
work_threads=4
-
store_lookup=2
-
store_group=group2
-
store_server=0
-
store_path=0
-
download_server=0
-
reserved_storage_space = 4GB
-
log_level=info
-
run_by_group=
-
run_by_user=
-
allow_hosts=*
-
sync_log_buff_interval = 10
-
check_active_interval = 120
-
thread_stack_size = 64KB
-
storage_ip_changed_auto_adjust = true
-
storage_sync_file_max_delay = 86400
-
storage_sync_file_max_time = 300
-
use_trunk_file = false
-
slot_min_size = 256
-
slot_max_size = 16MB
-
trunk_file_size = 64MB
-
http.disabled=false
-
http.server_port=8080
-
http.check_alive_interval=30
-
http.check_alive_type=tcp
-
http.check_alive_uri=/status.html
-
http.need_find_content_type=true
-
- [root@db1 ~]# grep -v '^#' /etc/fdfs/http.conf |grep -v '^$'
-
http.default_content_type = application/octet-stream
-
http.mime_types_filename=/etc/fdfs/mime.types
-
http.anti_steal.check_token=false
-
http.anti_steal.token_ttl=900
-
http.anti_steal.secret_key=FastDFS1234567890
-
http.anti_steal.token_check_fail=/home/data/fastdfs/conf/anti-steal.jpg
三:启动tracker服务,需要注意tracker.conf文件最后一行为#include httpd.conf
- [root@db1 ~]# /usr/local/FastDFS/bin/fdfs_trackerd /etc/fdfs/tracker.conf
- /usr/local/FastDFS/bin/fdfs_trackerd: error while loading shared libraries: libevent-
- 2.0.so.5: cannot open shared object file: No such file or directory
-
[root@db1 ~]# echo '/usr/local/libevent-2.0.14/include/' >> /etc/ld.so.conf
-
[root@db1 ~]# echo '/usr/local/libevent-2.0.14/lib/' >> /etc/ld.so.conf
- [root@db1 ~]# ldconfig
- [root@db1 ~]# /usr/local/FastDFS/bin/fdfs_trackerd /etc/fdfs/tracker.conf
- [2012-07-04 17:52:25] ERROR - file: tracker_func.c, line: 160, "/home/data/fastdfs"
- can't be accessed, error info: No such file or directory
- [root@db1 ~]# mkdir -p /home/data/fastdfs
- [root@db1 ~]# /usr/local/FastDFS/bin/fdfs_trackerd /etc/fdfs/tracker.conf
- [root@db1 ~]# echo $?
- 22
- [root@db1 ~]# cat /home/data/fastdfs/logs/trackerd.log
- [2012-07-04 17:52:50] ERROR - file: ../common/fdfs_http_shared.c, line: 128, param
- "http.mime_types_filename" not exist or is empty
- [root@db1 ~]# tail -1 /etc/fdfs/tracker.conf
- #include http.conf
- [root@db1 ~]# /usr/local/FastDFS/bin/fdfs_trackerd /etc/fdfs/tracker.conf
- [root@db1 ~]# echo $?
- 0
- [root@db1 ~]# ps -ef |grep track
- root 3535 1 0 15:47 ? 00:00:00 /usr/local/FastDFS/bin/fdfs_trackerd
- /etc/fdfs/tracker.conf
-
- [root@db1 ~]# netstat -ntpl |grep fdfs
- tcp 0 0 192.168.123.110:22122 0.0.0.0:* LISTEN
- 3535/fdfs_trackerd
- tcp 0 0 192.168.123.110:8080 0.0.0.0:* LISTEN
- 3535/fdfs_trackerd
四:storage配置文件
- [root@db2 ~]# grep -v '^#' /etc/fdfs/storage.conf |grep -v '^$'
-
disabled=false
-
group_name=group1
-
bind_addr=192.168.123.20
-
client_bind=true
-
port=23000
-
connect_timeout=30
-
network_timeout=60
-
heart_beat_interval=30
-
stat_report_interval=60
-
base_path=/home/data/fastdfs
-
max_connections=256
-
buff_size = 256KB
-
work_threads=4
-
disk_rw_separated = true
-
disk_rw_direct = false
-
disk_reader_threads = 1
-
disk_writer_threads = 1
-
sync_wait_msec=50
-
sync_interval=0
-
sync_start_time=00:00
-
sync_end_time=23:59
-
write_mark_file_freq=500
-
store_path_count=1
-
store_path0=/home/data/fastdfs
-
subdir_count_per_path=256
-
tracker_server=192.168.123.110:22122
-
log_level=info
-
run_by_group=
-
run_by_user=
-
allow_hosts=*
-
file_distribute_path_mode=0
-
file_distribute_rotate_count=100
-
fsync_after_written_bytes=0
-
sync_log_buff_interval=10
-
sync_binlog_buff_interval=10
-
sync_stat_file_interval=300
-
thread_stack_size=512KB
-
upload_priority=10
-
if_alias_prefix=
-
check_file_duplicate=0
-
key_namespace=FastDFS
-
keep_alive=0
-
http.disabled=false
-
httphttp.domain_name=
-
http.server_port=8888
-
http.trunk_size=256KB
-
http.need_find_content_type=true
-
- [root@db2 ~]# grep -v '^#' /etc/fdfs/client.conf |grep -v '^$'
-
connect_timeout=30
-
network_timeout=60
-
base_path=/home/data/fastdfs
-
tracker_server=192.168.123.110:22122
-
log_level=info
-
http.tracker_server_port=8080
-
- [root@db2 ~]# grep -v '^#' /etc/fdfs/http.conf |grep -v '^$'
-
http.default_content_type = application/octet-stream
-
http.mime_types_filename=mime.types
-
http.anti_steal.check_token=false
-
http.anti_steal.token_ttl=900
-
http.anti_steal.secret_key=FastDFS1234567890
-
http.anti_steal.token_check_fail=/home/data/fastdfs/conf/anti-steal.jpg
五:启动storage,需要注意storage.conf文件最后一行为#include httpd.conf
- [root@db2 ~]# mkdir -p /home/data/fastdfs
-
[root@db2 ~]# echo '/usr/local/libevent-2.0.14/include/' >> /etc/ld.so.conf
-
[root@db2 ~]# echo '/usr/local/libevent-2.0.14/lib/' >> /etc/ld.so.conf
- [root@db2 ~]# ldconfig
-
- [root@db2 ~]# tail -2 /etc/fdfs/storage.conf
- #use "#include" directive to include HTTP other settings
- #include http.conf
-
- [root@db2 ~]# /usr/local/FastFDS/bin/fdfs_storaged /etc/fdfs/storage.conf
- mkdir data path: 00 ...
- mkdir data path: 01 ...
- mkdir data path: 02 ...
- ———输出省略————
- data path: /home/data/fastdfs/data, mkdir sub dir done.
-
- [root@db2 ~]# ps -ef |grep fdfs
- root 14451 1 0 16:15 ? 00:00:00 /usr/local/FastFDS/bin/fdfs_storaged
- /etc/fdfs/storage.conf
- root 14468 8238 0 16:16 pts/1 00:00:00 grep fdfs
-
- [root@db2 ~]# netstat -ntpl |grep fdfs
- tcp 0 0 192.168.123.20:8888 0.0.0.0:* LISTEN
- 14451/fdfs_storaged
- tcp 0 0 192.168.123.20:23000 0.0.0.0:* LISTEN
- 14451/fdfs_storaged
五:测试文件上传
[root@db2 ~]# /usr/local/FastFDS/bin/fdfs_test /etc/fdfs/client.conf upload /etc/passwd This is FastDFS client test program v3.02 Copyright (C) 2008, Happy Fish / YuQing FastDFS may be copied only under the terms of the GNU General Public License V3, which may be found in the FastDFS source kit. Please visit the FastDFS Home Page http://www.csource.org/ for more detail.
[2012-07-30 16:25:59] INFO - base_path=/home/data/fastdfs, connect_timeout=30,
network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0tracker_query_storage_store_list_without_group: server 1. group_name=group1, ip_addr=192.168.123.20, port=23000 group_name=group1, ip_addr=192.168.123.20, port=23000 storage_upload_by_filename group_name=group1, remote_filename=M00/00/00/wKh7FFAWRRfcMOGtAAAHKBGhZhE8065783 source ip address: 192.168.123.20 file timestamp=2012-07-30 16:25:59 file size=1832 file crc32=295790097 file url: http://192.168.123.110:8080/group1/M00/00/00/wKh7FFAWRRfcMOGtAAAHKBGhZhE8065783 storage_upload_slave_by_filename group_name=group1, remote_filename=M00/00/00/wKh7FFAWRRfcMOGtAAAHKBGhZhE8065783_big source ip address: 192.168.123.20 file timestamp=2012-07-30 16:25:59 file size=1832 file crc32=295790097 file url: http://192.168.123.110:8080/group1/M00/00/00/wKh7FFAWRRfcMOGtAAAHKBGhZhE8065783_big
[root@db2 ~]# /usr/local/FastFDS/bin/fdfs_test /etc/fdfs/client.conf upload 2.jpg This is FastDFS client test program v3.02 Copyright (C) 2008, Happy Fish / YuQing FastDFS may be copied only under the terms of the GNU General Public License V3, which may be found in the FastDFS source kit. Please visit the FastDFS Home Page http://www.csource.org/ for more detail.
[2012-07-30 16:29:22] INFO - base_path=/home/data/fastdfs, connect_timeout=30, network_timeout=60, tracker_server_count=1, anti_steal_token=0, anti_steal_secret_key length=0tracker_query_storage_store_list_without_group: server 1. group_name=group1, ip_addr=192.168.123.20, port=23000 group_name=group1, ip_addr=192.168.123.20, port=23000 storage_upload_by_filename group_name=group1, remote_filename=M00/00/00/wKh7FFAWReLzPs4sAAEYquSDGwc472.jpg source ip address: 192.168.123.20 file timestamp=2012-07-30 16:29:22 file size=71850 file crc32=3833797383 file url: http://192.168.123.110:8080/group1/M00/00/00/wKh7FFAWReLzPs4sAAEYquSDGwc472.jpg storage_upload_slave_by_filename group_name=group1, remote_filename=M00/00/00/wKh7FFAWReLzPs4sAAEYquSDGwc472_big.jpg source ip address: 192.168.123.20 file timestamp=2012-07-30 16:29:22 file size=71850 file crc32=3833797383 file url: http://192.168.123.110:8080/group1/M00/00/00/wKh7FFAWReLzPs4sAAEYquSDGwc472_big.jpg
|
分享到:
相关推荐
FastDFS安装和配置过程
整合配置fastdfs、nginx、nginx扩张插件,配置文件配置说明以及需要创建文件说明(zip是对应安装需要的zip)
FastDFS客户端配置文件,主要是为了配置FastDFS服务器的ip地址
FastDFS 安装和配置.docx。• 安装 libfastcommon-1.0.7.tar.gz o 解压:tar zxvf libfastcommon-1.0.7.tar.gz o 进入解压后目录:cd libfastcommon-1.0.7/ o 编译:./make.sh o 安装:./make.sh install o ...
fastDFS配置文件,主要修改其中的Service就可以了,其他可默认。
FastDFS_配置文件详解
FastDFS集群安装配置,详细介绍了FastDFS的安装与配置。
FastDFS集群的配置文件,详情请参见博文:http://blog.csdn.net/l1028386804/article/details/72803681
storage/group1--group2,mod_fastdfs.conf;master/backup--keepalived.conf,nginx.conf,nginx_check.sh
FastDFS+Nginx 安装及配置。FastDFS 是一个开源的轻量级分布式文件系统,纯 C 实现,支持 Linux、FreeBSD 等 UNIX 系统,FastDFS 可以看做是基于文件的 key value pair 存储系统,包含两个角色,分别为跟踪 服务器...
client.conf,storage.conf,tracker.conf,fastdfs-5.11.tar.gz等等一系列FastDFS所需的一切配置,需要的可以自取,或者私信本人,再给你
fastdfs分布式架构配置,详解:http://blog.csdn.net/clevercode/article/details/52267080。
FastDFS配置反向代理时nginx.conf配置文件.confFastDFS配置反向代理时nginx.conf配置文件.confFastDFS配置反向代理时nginx.conf配置文件.confFastDFS配置反向代理时nginx.conf配置文件.confFastDFS配置反向代理时...
NULL 博文链接:https://sunzhen930.iteye.com/blog/1885939
fastdfs的安装手册说明+配置文件+fastdfs安装包+nginx安装包,压缩包内是自己结合网上资料整理的fastdfs的全套配置及说明,已成功安装并运行
非常经典的一篇文档,包括FastDFS原理讲解、FastDFS模块组成、模块之间的流程关系,FastDFS集群安装配置,配置参数讲解。最最最重要的是第三方软件如何使用fastdfs分布式文件,网上搭建fastdfs的文档很多、优化、...
详见我的博客《FastDFS配置文件详解》
Nginx+FastDFS安装与配置图文教程,亲自把项目中所用到的安装配置过程记录下来,帮忙后面需要的同学更快地完成FastDfs搭建
fastdfs 安装配置过程,经过测试,java 客户端成功调用。
关于虚拟机的比较实用的小技巧;