对象存储OSS功能测试

Posted on | In | |

概述

本文针对对象存储OSS的基本功能，提供能测试的方法和脚本
一些基于boto3API的简单操作示例：boto3API.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#!/usr/bin/env python
#-*- coding:utf-8 -*-

import boto3

# creating a client
s3client = boto3.client('s3',
                        aws_secret_access_key = 'ZSU2I***8aNgGyMHBFhqwWnRzKz1fO',
                        aws_access_key_id = 'C6Y1E0C3EB***W8YIKEW',
                        endpoint_url = 'http://10.1.0.29')

# creating a bucket
bucket_name = 'ictfox'
response = s3client.create_bucket(Bucket = bucket_name)
print "Creating bucket {0} returns => {1}\n".format(bucket_name, response)

# listing owned buckets
response = s3client.list_buckets()
for bucket in response['Buckets']:
    print "Listing owned buckets returns => {0} was created on {1}\n".format(bucket['Name'], bucket['CreationDate'])

# creating an object
object_key = 'hello.txt'
response = s3client.put_object(Bucket = bucket_name, Key = object_key, Body = 'Hello World!')
print "Creating object {0} returns => {1}\n".format(object_key, response)

# Listing a bucket's content
response = s3client.list_objects(Bucket = bucket_name)
for obj in response['Contents']:
    print "Listing a bucket's content returns => {0}\t{1}\t{2}\n".format(obj['Key'], obj['Size'], obj['LastModified'])

# Changing an object's metadata(head object)
metadata = {'x-amz-meta-datastore': 'qr', 'x-amz-meta-datastore-version': '1.0.1'}
copySrc = '{0}/{1}'.format(bucket_name, object_key)
response = s3client.copy_object(Bucket = bucket_name, CopySource = copySrc, Key = object_key, Metadata = metadata, MetadataDirective = 'REPLACE')
print "Changing metadata of object {0} returns => {1}\n".format(object_key, response)

# Deleting an object
response = s3client.delete_object(Bucket = bucket_name, Key = object_key)
print "Deleting object {0} returns => {1}\n".format(object_key, response)

# deleting a bucket
response = s3client.delete_bucket(Bucket = bucket_name)
print "Deleting bucket {0} returns => {1}\n".format(bucket_name, response)

用户管理

用户的管理是通过Admin Ops API提供；这里我们通过之前创建好的管理账户来执行Admin Ops API，具体可以参考：http://docs.ceph.com/docs/master/radosgw/adminops/

创建用户

1
2
3
4
5
6
7
8
9
10
11
12
13
$ cat usercreate.sh
#!/bin/bash
token=5L65QDE4df8JJ8RM7MN5 ## USER_TOKEN
secret=Y9HPiBCwLDeSMSaiQhmPT2h7NgNUndvLERhktnIZ ## USER_SECRET
query=$1
name=$2
echo $query, $name
query3="&uid="
query2=admin/user
date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-8 )):%M:%S +0000" ; done)
header="PUT\n\n\n${date}\n/${query2}"
sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64)
curl -v -H "Date: ${date}" -H "Authorization: AWS ${token}:${sig}" -L -X PUT "http://<your-host-ip>/${query2}?format=json${query3}${query}&display-name=${name}" -H "Host: <your-host-ip>"

删除用户

1
2
3
4
5
6
7
8
9
10
11
$ cat userdelete.sh
#!/bin/bash
token=5L65QDE4238JJ8RM7MN5 ## USER_TOKEN
secret=Y9HPiBCwLDeSMSaiQhmPT2h7NgNUnqVLERhktnIZ ## USER_SECRET
query=$1
query3="&uid="
query2=admin/user
date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-8 )):%M:%S +0000" ; done)
header="DELETE\n\n\n${date}\n/${query2}"
sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64)
curl -v -H "Date: ${date}" -H "Authorization: AWS ${token}:${sig}" -L -X DELETE "http://<your-host-ip>/${query2}?format=json${query3}${query}" -H "Host: <your-host-ip>"

User keys

管理User的keys也是通过admin Ops API操作的，同样需要通过管理账户来执行；

创建user key

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$cat createKey.sh
#!/bin/bash
token=5L65QDE4238JJ8RM7MN5 ## USER_TOKEN
secret=Y9HPiBCwLDeSMSaiQhmPT2h7NgNUnqVLERhktnIZ ## USER_SECRET
operate="PUT"
user=$1
query="admin/user"
date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-8 )):%M:%S +0000" ; done)
header="${operate}\n\n\n${date}\n/${query}"
sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64)
curl -v -L -X ${operate} "http://<your-host-ip>/${query}?key&format=json&uid=${user}&generate-key=True" \
 -H "Date: ${date}" \
 -H "Authorization: AWS ${token}:${sig}" \
 -H "Host: <your-host-ip>"

删除user key

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat removeKey.sh
#!/bin/bash
token=5L65QDE4238JJ8RM7MN5 ## USER_TOKEN
secret=Y9HPiBCwLDeSMSaiQhmPT2h7NgNUnqVLERhktnIZ ## USER_SECRET
operate="DELETE"
key=$1
query="admin/user"
date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-8 )):%M:%S +0000" ; done)
header="${operate}\n\n\n${date}\n/${query}"
sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64)
curl -v -L -X ${operate} "http://<your-host-ip>/${query}?key&format=json&access-key=${key}" \
           -H "Date: ${date}" \
           -H "Authorization: AWS ${token}:${sig}" \
           -H "Host: <your-host-ip>"

列出user keys

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat getUserInfo.sh
#!/bin/bash
token=5L65QDE4238JJ8RM7MN5 ## USER_TOKEN
secret=Y9HPiBCwLDeSMSaiQhmPT2h7NgNUnqVLERhktnIZ ## USER_SECRET
user=$1
query=admin/user
date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-8 )):%M:%S +0000" ; done)
header="GET\n\n\n${date}\n/${query}"
sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64)
echo $sig
curl -v -L -X GET "http://<your-host-ip>/${query}?format=json&uid=${user}" \
           -H "Date: ${date}" \
           -H "Authorization: AWS ${token}:${sig}" \
           -H "Host: <your-host-ip>"

权限管理

开源的比较常用的s3的SDK有boto3，我们在RDS的备份中已经使用过
参考：http://boto3.readthedocs.io/en/latest/reference/services/s3.html#s3

下面的操作都简要列出了调用的api和说明；

获取Bucket权限

get_bucket_acl(**kwargs)

  • Gets the access control policy for the bucket.

设置Bucket权限

put_bucket_acl(**kwargs)

  • Sets the permissions on a bucket using access control lists (ACL).

Bucket操作

同权限管理，下面的操作基于boto3，都简要列出了调用的api和说明；

列出Buckets

list_buckets()

  • Returns a list of all buckets owned by the authenticated sender of the request.

创建Bucket

create_bucket(**kwargs)

  • Creates a new bucket.

删除Bucket

delete_bucket(**kwargs)

  • Deletes the bucket. All objects (including all object versions and Delete Markers) in the bucket must be deleted before the bucket itself can be deleted.

列出Bucket Objects

  1. list_objects(**kwargs)
    Returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket.

  2. list_objects_v2(**kwargs)
    Returns some or all (up to 1000) of the objects in a bucket. You can use the request parameters as selection criteria to return a subset of the objects in a bucket. Note: ListObjectsV2 is the revised List Objects API and we recommend you use this revised API for new application development.

获取Bucket ACL

get_bucket_acl(**kwargs)

  • Gets the access control policy for the bucket.

设置Bucket ACL

put_bucket_acl(**kwargs)

  • Sets the permissions on a bucket using access control lists (ACL).

获取Bucket Info

head_bucket(**kwargs)

  • This operation is useful to determine if a bucket exists and you have permission to access it.

枚举Bucket分块上传

list_multipart_uploads(**kwargs)

  • This operation lists in-progress multipart uploads.

Object操作

上传Object

  1. put_object(**kwargs)
    Adds an object to a bucket.

  2. upload_file(Filename, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)
    Upload a file to an S3 object.

  3. upload_fileobj(Fileobj, Bucket, Key, ExtraArgs=None, Callback=None, Config=None)
    Upload a file-like object to S3.
    The file-like object must be in binary mode.
    This is a managed transfer which will perform a multipart upload in multiple threads if necessary.

复制Object

copy_object(**kwargs)

  • Creates a copy of an object that is already stored in Amazon S3.

删除Object

  1. delete_object(**kwargs)
    Removes the null version (if there is one) of an object and inserts a delete marker, which becomes the latest version of the object. If there isn’t a null version, Amazon S3 does not remove any objects.

  2. delete_objects(**kwargs)
    This operation enables you to delete multiple objects from a bucket using a single HTTP request. You may specify up to 1000 keys.

下载Object

  1. get_object(**kwargs)
    Retrieves objects from Amazon S3.

  2. download_file(Bucket, Key, Filename, ExtraArgs=None, Callback=None, Config=None)
    Download an S3 object to a file.

  3. download_fileobj(Bucket, Key, Fileobj, ExtraArgs=None, Callback=None, Config=None)
    Download an object from S3 to a file-like object.
    The file-like object must be in binary mode.
    This is a managed transfer which will perform a multipart download in multiple threads if necessary.

获取Object ACL

get_object_acl(**kwargs)

  • Returns the access control list (ACL) of an object.

设置Object ACL

put_object_acl(**kwargs)

  • uses the acl subresource to set the access control list (ACL) permissions for an object that already exists in a bucket

获取Object Info

head_object(**kwargs)

  • The HEAD operation retrieves metadata from an object without returning the object itself. This operation is useful if you’re only interested in an object’s metadata. To use HEAD, you must have READ access to the object.

支持Object Multipart

  1. create_multipart_upload(**kwargs)
    Initiates a multipart upload and returns an upload ID.

  2. complete_multipart_upload(**kwargs)
    Completes a multipart upload by assembling previously uploaded parts.

  3. abort_multipart_upload(**kwargs)
    Aborts a multipart upload.
    To verify that all parts have been removed, so you don’t get charged for the part storage, you should call the List Parts operation and ensure the parts list is empty.

  4. upload_part(**kwargs)
    Uploads a part in a multipart upload.

其他操作

获取RGW User统计信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
$ cat getUsage.sh
#!/bin/bash
token=5L65QDE4238JJ8RM7MN5 ## USER_TOKEN
secret=Y9HPiBCwLDeSMSaiQhmPT2h7NgNUnqVLERhktnIZ ## USER_SECRET
operate="GET"
user=$1
query="admin/usage"
date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-8 )):%M:%S +0000" ; done)
header="${operate}\n\n\n${date}\n/${query}"
sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64)
curl -v -L -X ${operate} "http://<your-host-ip>/${query}?format=json&uid=${user}" \
           -H "Date: ${date}" \
           -H "Authorization: AWS ${token}:${sig}" \
           -H "Host: <your-host-ip>"

删除RGW User统计信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
$ cat trimeUsage.sh
#!/bin/bash
token=5L65QDE4238JJ8RM7MN5 ## USER_TOKEN
secret=Y9HPiBCwLDeSMSaiQhmPT2h7NgNUnqVLERhktnIZ ## USER_SECRET
operate="DELETE"
user=$1
stime=$2 #start time
etime=$3 #end time
query="admin/usage"
date=$(for i in $(date "+%H") ; do date "+%a, %d %b %Y $(( 10#$i-8 )):%M:%S +0000" ; done)
header="${operate}\n\n\n${date}\n/${query}"
sig=$(echo -en ${header} | openssl sha1 -hmac ${secret} -binary | base64)
curl -v -L -X ${operate} "http://<your-host-ip>/${query}?format=json&uid=${user}&start=${stime}&end=${etime}" \
           -H "Date: ${date}" \
           -H "Authorization: AWS ${token}:${sig}" \
           -H "Host: <your-host-ip>"
支持原创