最近配置一個GridFTP 用於測試其傳輸FTP性能, 在這裡簡單記錄，備忘；使用本教程可以簡單起一個GridFTP用於測試服務； 預配置環境： 測試系統：CentOS 7 1806 配置Yum倉庫：https://opensciencegrid.org/docs/common/yum/ 安裝Gri ...
最近配置一個GridFTP 用於測試其傳輸FTP性能, 在這裡簡單記錄，備忘；使用本教程可以簡單起一個GridFTP用於測試服務；
預配置環境：
測試系統：CentOS 7 1806
配置Yum倉庫：https://opensciencegrid.org/docs/common/yum/
安裝GridFTP:
參考文檔：
https://opensciencegrid.org/docs/common/yum/
https://opensciencegrid.org/docs/data/gridftp/
https://gridcf.org/gct-docs/latest/gridftp/admin/index.html#gridftp-admin-config-security-anonymous
https://gridcf.org/gct-docs/latest/gridftp/admin/index.html#_globus_gridftp_server_quickstart (管理員手冊)
https://opensciencegrid.org/docs/common/ca/
yum install osg-gridftp
也可以直接從epel中直接安裝：
yum install globus-gridftp-server
啟動GridFTP匿名伺服器：（帶有認證的沒有配置成功，也不想試了）
globus-gridftp-server -control-interface 0.0.0.0 -aa -p 5000 -anonymous-user 1000 # root 用戶使用 1000 表示本地任意一個非root用戶
globus-gridftp-server -control-interface 0.0.0.0 -aa -p 5000 # 非 root 用戶使用
globus-gridftp-server -control-interface 0.0.0.0 -aa -p 8080 -anonymous-user 1040 -home-dir /tmp/ -auth-level 0 -allow-root
更多選項可以使用 --help 查看；
安裝 globus-url-copy 命令：
yum install globus-gass-copy-progs
生成測試文件：
dd if=/dev/zero of=size80m bs=8k count=10240
輸出測試日誌：
tail -f /var/log/gridftp-auth.log
運行匿名用戶測試：
root@localhost ~# globus-url-copy -v file:///root/size80m ftp://127.0.0.1:5000/tmp/size80m Source: file:///root/ Dest: ftp://127.0.0.1:5000/tmp/ size80m
測試成功：
[21230] Thu Sep 19 12:29:09 2019 :: New connection from: localhost:40224 [21230] Thu Sep 19 12:29:09 2019 :: User anonymous successfully authorized. [21230] Thu Sep 19 12:29:09 2019 :: Starting to transfer "/tmp/size80m". [21230] Thu Sep 19 12:29:12 2019 :: Finished transferring "/tmp/size80m". [21230] Thu Sep 19 12:29:12 2019 :: Closed connection from localhost:40224 [20327] Thu Sep 19 12:29:12 2019 :: Child process 21230 ended with rc = 0
安裝gridftp 客戶端：
[root@xuyaowen ~]# yum search gridftp | grep -i client Last metadata expiration check: 2:28:09 ago on Thu 19 Sep 2019 11:44:29 AM CST. edg-gridftp-client.x86_64 : Command line clients to GridFTP libraries uberftp.x86_64 : GridFTP-enabled ftp client (聽說是最好用的工具, 但是我一運行就崩潰，最終還是使用 globus-url-copy 命令進行測試) globus-ftp-client.x86_64 : Grid Community Toolkit - GridFTP Client Library globus-ftp-client-devel.x86_64 : Grid Community Toolkit - GridFTP Client Library
測試吞吐：
# test network throughput globus-url-copy -vb -p 4 ftp://10.134.150.5:5000/dev/zero file:///dev/null
順序讀：
# tmp globus-url-copy -vb -p 4 ftp://10.134.150.5:5000/dev/zero file:///dev/null
順序寫：
# tmp globus-url-copy -vb -p 4 file:///dev/zero ftp://10.134.150.5:5000/dev/null
使用UDT:
globus-gridftp-server -control-interface 0.0.0.0 -aa -p 5000 -anonymous-user 1040 -home-dir /tmp/ -auth-level 0 -allow-root -allow-udt globus-url-copy -vb -p 4 -udt ftp://10.134.150.5:5000/dev/zero file:///dev/null
保持更新，如果對您有幫助或者對配置過程有疑問，請關註cnblogs.com/xuyaowen，並留言；
其他可用參考文檔：
https://fasterdata.es.net/data-transfer-tools/gridftp/
https://www.xsede.org/wwwteragrid/archive/web/user-support/gridftp.html
globus-url-copy 使用說明：
GLOBUS-URL-COPY(1) Grid Community Toolkit Manual GLOBUS-URL-COPY(1) NAME globus-url-copy - globus-url-copy SYNOPSIS globus-url-copy [options] SOURCE-URL DESTINATION-URL DESCRIPTION The globus-url-copy program is a command line tool for multi-protocol data movement. It supports gsiftp:// (GridFTP), ftp://, http://, https://, sshftp:// and file:/// protocol specifiers in the URL. OPTIONS -help, -usage Print help. -versions Print the versions of all modules that this program uses -c, -continue-on-error Do not die after any errors. By default, program will exit after most errors. -a, -ascii Convert the file to/from ASCII format to/from local file format -b, -binary Do not apply any conversion to the files. default -f FILENAME Read a list of URL pairs from filename. Each line should contain sourceURL destURL. Enclose URLs with spaces in double qoutes ("). Blank lines and lines beginning with # will be ignored. -cd, -create-dest Create destination directory if needed. -r Copy files in subdirectories -fast Recommended when using GridFTP servers. Use MODE E for all data transfers, including reusing data channels between list and transfer operations. -t SECONDS Run the transfer for this number of seconds and then end. Useful for performance testing or forced restart loops. -q, -quiet Suppress all output for successful operation. -v, -verbose Display URLs being transferred -vb, -verbose-perf During the transfer, display the number of bytes transferred and the transfer rate per second. Show URLs being transferred -dbg, -debugftp Debug ftp connections. Prints control channel communication to stderr -rst, -restart Restart failed ftp operations. -rst-retries RETRIES The maximum number of times to retry the operation before giving up on the transfer. Use 0 for infinite. Default is 5. -rst-interval SECONDS The interval in seconds to wait after a failure before retrying the transfer. Use 0 for an exponential backoff. Default is 0. -rst-timeout SECONDS Maximum time after a failure to keep retrying. Use 0 for no timeout. Default is 0. -stall-timeout SECONDS, -st SECONDS How long before cancelling/restarting a transfer with no data movement. Set to 0 to disable. Default is 600 seconds. -df FILENAME, -dumpfile FILENAME Path to a file where untransferred URLs will be saved for later restarting. Resulting file is the same format as the -f input file. If file exists, it will be read and all other URL input will be ignored. -do FILENAME, -dump-only FILENAME Perform no write operations on the destination. Instead, all files that would be transferred are enumerated and dumped to the specified file. Resulting file is the same format as the -f input file. Note: if you intend to use this file as input for a future transfer, the -create-dest option will be required if any destination directories do not already exist. -rp, -relative-paths The path portion of ftp URLs will be interpreted as relative to the user’s starting directory on the server. By default, all paths are root-relative. When this flag is set, the path portion of the ftp URL must start with %2F if it designates a root-relative path. -s SUBJECT, -subject SUBJECT Use this subject to match with both the source and dest servers. -ss SUBJECT, -source-subject SUBJECT Use this subject to match with the source server -ds SUBJECT, -dest-subject SUBJECT Use this subject to match with the destination server. -tcp-bs SIZE, -tcp-buffer-size SIZE Specify the size (in bytes) of the buffer to be used by the underlying ftp data channels. -bs block SIZE, -block-size block SIZE Specify the size (in bytes) of the buffer to be used by the underlying transfer methods. -p PARALLELISM, -parallel PARALLELISM Specify the number of parallel data connections should be used. -notpt, -no-third-party-transfers Turn third-party transfers off (on by default). -nodcau, -no-data-channel-authentication Turn off data channel authentication for ftp transfers. -dcsafe, -data-channel-safe Set data channel protection mode to SAFE -dcpriv, -data-channel-private Set data channel protection mode to PRIVATE -off, -partial-offset Offset for partial ftp file transfers, defaults to 0. -len, -partial-length Length for partial ftp file transfers, used only for the source url, defaults the full file. -list URL List the files located at URL. -stripe Enable striped transfers on supported servers. -striped-block-size, -sbs Set layout mode and block size for striped transfers. If not set, server defaults will be used. If set to 0, Partitioned mode will be used. If set to > 0, Blocked mode will be used, with this as the block size. -ipv6 Use ipv6 when available (EXPERIMENTAL) -udt Use UDT, a reliable udp based transport protocol, for data transfers -g2, -gridftp2 Use GridFTP v2 protocol enhancements when possible. -dp, -delayed-pasv Enable delayed passive. -mn NAME, -module-name NAME Set the back-end storage module to use for both the source and destination in a GridFTP transfer. -mp PARAMETERS, -module-parameters PARAMETERS Set the back-end storage module arguments to use for both the source and destination in a GridFTP transfer. -smn NAME, -src-module-name NAME Set the back-end storage module to use for the source in a GridFTP transfer. -smp PARAMETERS, -src-module-parameters PARAMETERS Set the back-end storage module arguments to use for the source in a GridFTP transfer. -dmn NAME, -dst-module-name NAME Set the back-end storage module to use for the destination in a GridFTP transfer. -dmp PARAMETERS, -dst-module-parameters PARAMETERS Set the back-end storage module arguments to use for the destination in a GridFTP transfer. -aa FILE, -authz-assert FILE Use the assertions in FILE to authorize the access with both source and destination servers. -saa FILE, -src-authz-assert FILE Use the assertions in this file to authorize the access with source server. -daa FILE, -dst-authz-assert FILE Use the assertions in this file to authorize the access with dest server. -cache-aa, -cache-authz-assert Cache the authz assertion for subsequent transfers. -cache-saa, -cache-src-authz-assert Cache the src authz assertion for subsequent transfers. -cache-daa, -cache-dst-authz-assert Cache the dst authz assertion for subsequent transfers. -pipeline, -pp Enable pipelining support for multi-file ftp transfers. Currently third-party transfers benefit from this. EXPERIMENTAL -concurrency, -cc Number of concurrent ftp connections to use for multiple transfers. -nl-bottleneck, -nlb Use NetLogger to estimate speeds of disk and network read/write system calls, and attempt to determine the bottleneck component. -sp COMMANDS, -src-pipe COMMANDS Set the source end of a remote transfer to use piped in input with the given command line. Do not use with -fsstack. -DP COMMANDS, -dst-pipe COMMANDS Set the destination end of a remote transfer to write data to then standard input of the program run via the given command line. Do not use with -fsstack. -pipe COMMANDS Sets both -src-pipe and -dst-pipe to the same thing. -dcstack STACK, -data-channel-stack STACK Set the XIO driver stack for the network on both the source and the destination. Both must be GridFTP servers. The stack should contain all network drivers to use, in the order specified from bottom to top (e.g. -dcstack tcp,gsi). If the gsi driver is not included in the stack and data channel authentication is enabled, it will be inserted above the transport driver in the stack. -fsstack STACK, -file-system-stack STACK Set the XIO driver stack for the disk on both the source and the destination. Both must be GridFTP servers. The stack should contain all file system drivers to use, in the order specified from bottom to top. -src-dcstack STACK, -source-data-channel-stack STACK Set the XIO driver stack for the network on the source GridFTP server. See -dcstack above for description of the STACK string. -src-fsstack STACK, -source-file-system-stack STACK Set the XIO driver stack for the disk on the source GridFTP server. See -fsstack above for description of the STACK string. -dst-dcstack STACK, -dest-data-channel-stack STACK Set the XIO driver stack for the network on the destination GridFTP server. See -dcstack above for description of the STACK string. -dst-fsstack STACK, -dest-file-system-stack STACK Set the XIO driver stack for the disk on the destination GridFTP server. See -fsstack above for description of the STACK string. -cred PATH Set the credentials to use for both ftp connections. -src-cred CRED-FILE, -sc CRED-FILE Set the credentials to use for source ftp connections. -dst-cred CRED-FILE, -dc CRED-FILE Set the credentials to use for destination ftp connections. -af FILENAME, -alias-file FILENAME File with mapping of logical host aliases to lists of physical hosts. When used with multiple concurrent connections, each connection uses the next host in the list. Each line should either be an alias, noted with the @ symbol, or a hostname[:port]. Currently, only the aliases @source and @destination are valid, and they are used for every source or destination URL. -sync Only transfer files where the destination does not exist or differs from the source. -sync-level controls how to determine if files differ. -sync-level number Criteria for determining if files differ when performing a sync transfer. The default sync level is 2. The available levels are: · Level 0 will only transfer if the destination does not exist. · Level 1 will transfer if the size of the destination does not match the size of the source. · Level 2 will transfer if the time stamp of the destination is older than the time stamp of the source. · Level 3 will perform a checksum of the source and destination and transfer if the checksums do not match. The default algorithm used for this checksum is MD5, but other algorithms can be specified with the -algo parameter. -checksum-alg CHECKSUM-ALGORITHM Set the algorithm type to use for all checksum operations during the transfer. -verify-checksum Perform a checksum on the source and destination after each file transfer and compare the two. If they do not match, fail the transfer. The default algorithm used for this checksum is MD5, but other algorithms can be specified with the -checksum-alg parameter. AUTHOR Copyright © 1999-2016 University of Chicago Grid Community Toolkit 6 03/31/2018 GLOBUS-URL-COPY(1)globus-url-copy使用說明
globus-gridftp-server 使用說明:
GLOBUS-GRIDFTP-SER(8) Grid Community Toolkit Manual GLOBUS-GRIDFTP-SER(8) NAME globus-gridftp-server - The Globus GridFTP server daemon SYNOPSIS globus-gridftp-server OPTIONS DESCRIPTION The globus-gridftp-server program is a ftp server with support for GridFTP protocol extensions, including strong authentication, parallel data transfers, and parallel data layouts. OPTIONS The list below contains the command-line options for the server, and also the name of the configuration file entry that implements that option. Note that any boolean option can be negated on the command line by preceding the specified option with -no- or -n. example: -no-cas or -nf. Informational Options -h,-help Show usage information and exit. This option can also be set in the configuration file as help. The default value of this option is FALSE. -hh,-longhelp Show more usage information and exit. This option can also be set in the configuration file as longhelp. The default value of this option is FALSE. -v,-version Show version information for the server and exit. This option can also be set in the configuration file as version. The default value of this option is FALSE. -V,-versions Show version information for all loaded globus libraries and exit. This option can also be set in the configuration file as versions. The default value of this option is FALSE. Modes of Operation -i,-inetd Run under an inetd service. This option can also be set in the configuration file as inetd. The default value of this option is FALSE. -s,-daemon Run as a daemon. All connections will fork off a new process and setuid if allowed. This option can also be set in the configuration file as daemon. The default value of this option is TRUE. -S,-detach Run as a background daemon detached from any controlling terminals. This option can also be set in the configuration file as detach. The default value of this option is FALSE. -ssh Run over a connected ssh session. This option can also be set in the configuration file as ssh. The default value of this option is FALSE. -exec string For statically compiled or non-GLOBUS_LOCATION standard binary locations, specify the full path of the server binary here. Only needed when run in daemon mode. This option can also be set in the configuration file as exec. -chdir Change directory when the server starts. This will change directory to the dir specified by the chdir_to option. This option can also be set in the configuration file as chdir. The default value of this option is TRUE. -chdir-to string Directory to chdir to after starting. Will use / if not set. Note that this is the directory of the process, not the client’s home directory. This option can also be set in the configuration file as chdir_to. -threads number Enable threaded operation and set the number of threads. The default is 0, which is non-threaded. When threading is required, a thread count of 1 or 2 should be sufficient. This option can also be set in the configuration file as threads. -f,-fork Server will fork for each new connection. Disabling this option is only recommended when debugging. Note that non-forked servers running as root will only accept a single connection, and then exit. This option can also be set in the configuration file as fork. The default value of this option is TRUE. -1,-single Exit after a single connection. This option can also be set in the configuration file as single. The default value of this option is FALSE. -chroot-path string Path to become the new root after authentication. This path must contain a valid certificate structure, /etc/passwd, and /etc/group. The command globus-gridftp-server-setup-chroot can help create a suitable directory structure. This option can also be set in the configuration file as chroot_path. Authentication, Authorization, and Security Options -auth-level number Add levels together to use more than one. If not set uses level 2 for front ends and level 1 for data nodes. Note that levels 2 and 4 imply level 1 as well. 0 = Disables all authorization checks. 1 = Authorize identity. 2 = Authorize all file/resource accesses. 4 = Disable changing process uid to authenticated user (no setuid) -- DO NOT use this when process is started as root. This option can also be set in the configuration file as auth_level. -process-user string User to setuid to upon login for all connections. Only applies when running as root. This option can also be set in the configuration file as process_user. -process-group string Group to setgid to upon login for all connections. If unset, the default group of process_user will be used. This option can also be set in the configuration file as process_group. -ipc-allow-from string Only allow connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.1. will match and allow a connection from 192.168.1.45. Note that if this option is used any address not specifically allowed will be denied. This option can also be set in the configuration file as ipc_allow_from. -ipc-deny-from string Deny connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.2. will match and deny a connection from 192.168.2.45. This option can also be set in the configuration file as ipc_deny_from. -allow-from string Only allow connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.1. will match and allow a connection from 192.168.1.45. Note that if this option is used any address not specifically allowed will be denied. This option can also be set in the configuration file as allow_from. -deny-from string Deny connections from these source ip addresses. Specify a comma separated list of ip address fragments. A match is any ip address that starts with the specified fragment. Example: 192.168.2. will match and deny a connection from 192.168.2.45. This option can also be set in the configuration file as deny_from. -encrypt-data Require encrypted data channels. This will cause an error and prevent all transfers in which the client does not request an authenticated and encrypted data channel. This option can also be set in the configuration file as encrypt_data. The default value of this option is FALSE. -si,-secure-ipc Use GSI security on ipc channel. This option can also be set in the configuration file as secure_ipc. The default value of this option is TRUE. -ia string,-ipc-auth-mode string Set GSI authorization mode for the ipc connection. Options are: none, host, self or subject:[subject]. This option can also be set in the configuration file as ipc_auth_mode. The default value of this option is host. -aa,-allow-anonymous Allow clear text anonymous access. If server is running as root anonymous_user must also be set. Disables ipc security. This option can also be set in the configuration file as allow_anonymous. The default value of this option is FALSE. -anonymous-names-allowed string Comma separated list of names to treat as anonymous users when allowing anonymous access. If not set, the default names of anonymous and ftp will be allowed. Use * to allow any username. This option can also be set in the configuration file as anonymous_names_allowed. -anonymous-user string User to setuid to for an anonymous connection. Only applies when running as root. This option can also be set in the configuration file as anonymous_user. -anonymous-group string Group to setgid to for an anonymous connection. If unset, the default group of anonymous_user will be used. This option can also be set in the configuration file as anonymous_group. -sharing-dn string Allow sharing when using the supplied DN. A client connected with these credentials will be able to access any user for which sharing is enabled. This option can also be set in the configuration file as sharing_dn. -sharing-state-dir string Full path to a directory that will contain files used by GridFTP to control sharing access for individual local accounts. The special variables $HOME and $USER can be used to create a dynamic path that is unique to each local account. This pathmust be writable by the associated account. The default path is $HOME/.globus/sharing/. This must refer to a path on the filesystem, not a path that is only accessible via a DSI plugin. This option can also be set in the configuration file as sharing_state_dir. -sharing-control Allow a local user account to control its own sharing access via special GridFTP client commands. The user account must have filesystem write access to the sharing state dir. This option can also be set in the configuration file as sharing_control. The default value of this option is TRUE. -sharing-rp string Sharing specific path restrictions. This completely replaces the normal path restrictions (-rp) when an account is being shared by a sharing-dn login.Follows normal path restriction semantics. This option can also be set in the configuration file as sharing_rp. -sharing-users-allow string Comma separated list of usernames that are allowed to share unless matched in the user deny lists. If this list is set, users that are not included will be denied unless matched in the group allow list. This option can also be set in the configuration file as sharing_users_allow. -sharing-users-deny string Comma separated list of usernames that are denied sharing even if matched in the user or group allow lists. This option can also be set in the configuration file as sharing_users_deny. -sharing-groups-allow string Comma separated list of groups whose members are allowed to share unless matched in the user or group deny lists. If this list is set, groups that are not included will be denied unless matched in the user allow list. This option can also be set in the configuration file as sharing_groups_allow. -sharing-groups-deny string Comma separated list of groups whose members will be denied sharing unless matched in the user allow list. This option can also be set in the configuration file as sharing_groups_deny. -allow-root Allow clients to be mapped to the root account. This option can also be set in the configuration file as allow_root. The default value of this option is FALSE. -allow-disabled-login Do not check if a user’s system account is disabled before allowing login. This option can also be set in the configuration file as allow_disabled_login. The default value of this option is FALSE. -password-file string Enable clear text access and authenticate users against this /etc/passwd formatted file. This option can also be set in the configuration file as pw_file. -connections-max number Maximum concurrent connections allowed. Only applies when running in daemon mode. Unlimited if not set. This option can also be set in the configuration file as connections_max. -connections-disabled Disable all new connections. For daemon mode, issue a SIGHUP to the server process after changing the config file in order to not affect ongoing connections. This option can also be set in the configuration file as connections_disabled. The default value of this option is FALSE. -offline-msg string Custom message to be displayed to clients when the server is offline via the connections_disabled or connections_max = 0 options. This option can also be set in the configuration file as offline_msg. -disable-command-list string A comma separated list of client commands that will be disabled. This option can also be set in the configuration file as disable_command_list. -authz-callouts,-cas Enable the GSI authorization callout framework, for callouts such as CAS. This option can also be set in the configuration file as cas. The default value of this option is TRUE. -use-home-dirs Set the starting directory to the authenticated users home dir. Disabling this is the same as setting -home-dir /. This option can also be set in the configuration file as use_home_dirs. The default value of this option is TRUE. -home-dir string
