Radosgw如何统计用户使用信息?

需求

在我们的需求中，要统计radosgw中每个用户的使用信息，然后计算用户的花费

包括三部分：

存储空间费用
流量费用
请求费用

存储空间统计：可以通过radosgw-admin命令获取
流量统计/请求次数统计：可以通过radosgw支持usage info的统计和获取，对应的也是radosgw-admin命令

Bucket stats

radosgw-admin bucket stats

命令：radosgw-admin bucket stats --uid=[userid]

$ radosgw-admin bucket stats --uid=S3User
[
    {
        "bucket": "database_backups",
        "pool": ".rgw.buckets",
        "index_pool": ".rgw.buckets.index",
        "id": "default.553597.94",
        "marker": "default.553597.94",
        "owner": "S3User",
        "ver": "0#959",
        "master_ver": "0#0",
        "mtime": "2016-06-14 16:22:20.000000",
        "max_marker": "0#",
        "usage": {
            "rgw.main": {
                "size_kb": 4792,
                "size_kb_actual": 4828,
                "num_objects": 18
            }
        },
        "bucket_quota": {
            "enabled": false,
            "max_size_kb": -1,
            "max_objects": -1
        }
    }
]

上面输出信息中，有bucket的objects个数和占用空间。

bucket stats对应的admin ops

1 2	GET /{admin}/bucket?format=json HTTP/1.1 Host {fqdn}

http://docs.ceph.com/docs/jewel/radosgw/adminops/#get-bucket-info

支持RGW usage

enable rgw usage

默认radosgw的usage是关闭的，需要修改ceph.conf文件打开，配置如下：

[client.rgw.BJ-001-001]
    host = BJ-001-001
    keyring = /etc/ceph/ceph.client.radosgw.keyring
    log file = /var/log/ceph/client.radosgw.gateway.log
    rgw socket path = /var/run/ceph/ceph.client.rgw.BJ-001-001.sock
    rgw enable usage log = true
    rgw_frontends = civetweb port=80

创建对应pool .usage

创建usage需要的pools

1	ceph osd pool create .usage 64 ## usage_log_pool

重启rgw服务

重启radosgw服务：/etc/init.d/radosgw restart

检查rgw的usage配置

$ ceph daemon /var/run/ceph/ceph-client/ceph-client.rgw.BJ-001-001.76550.asok config show | grep usage
    "rgw_usage_max_shards": "32",
    "rgw_usage_max_user_shards": "1",
    "rgw_enable_usage_log": "true",
    "rgw_usage_log_flush_threshold": "1024",
    "rgw_usage_log_tick_interval": "30",

上面的配置信息中有关于usage log更新的配置，所以默认刚开启配置后查看usage信息还是空的，等上一段时间再查看就有输出了。

查看RGW usage信息

命令格式

radosgw-admin -h
...
  usage show                 show usage (by user, date range)
  usage trim                 trim usage (by user, date range)
...

radosgw-admin usage show命令

可以查看指定user在某一时间段的统计信息。

1
2

用法：radosgw-admin usage show [--uid={uid}] [--start-date={date}] [--end-date={date}] [--categories=<list>] [--show-log-entries=<flag>] [--show-log-sum=<flag>]
示例：radosgw-admin usage show --uid=demo --start-date="2016-09-01 08:00:00" --end-date="2016-09-02 08:00:00" --categories="put_obj,put_acls"

线上环境实际输出：

$ radosgw-admin usage show --uid=S3User
{
    "entries": [
        {
            "owner": "S3User",
            "buckets": [
                {
                    "bucket": "",
                    "time": "2016-10-09 05:00:00.000000Z",
                    "epoch": 1475989200,
                    "categories": [
                        {
                            "category": "list_buckets",
                            "bytes_sent": 698,
                            "bytes_received": 0,
                            "ops": 2,
                            "successful_ops": 2
                        }
                    ]
                }
,
                {
                    "bucket": "database_backups",
                    "time": "2016-10-09 05:00:00.000000Z",
                    "epoch": 1475989200,
                    "categories": [
                        {
                            "category": "get_acls",
                            "bytes_sent": 474,
                            "bytes_received": 0,
                            "ops": 1,
                            "successful_ops": 1
                        },
                        {
                            "category": "get_cors",
                            "bytes_sent": 75,
                            "bytes_received": 0,
                            "ops": 1,
                            "successful_ops": 0
                        },
                        {
                            "category": "get_obj",
                            "bytes_sent": 555936,
                            "bytes_received": 0,
                            "ops": 3,
                            "successful_ops": 3
                        },
                        {
                            "category": "list_bucket",
                            "bytes_sent": 37633,
                            "bytes_received": 0,
                            "ops": 6,
                            "successful_ops": 6
                        },
                        {
                            "category": "put_obj",
                            "bytes_sent": 0,
                            "bytes_received": 558144,
                            "ops": 2,
                            "successful_ops": 2
                        }
                    ]
                }
            ]
        }
    ],
    "summary": [
        {
            "user": "S3User",
            "categories": [
                {
                    "category": "get_acls",
                    "bytes_sent": 474,
                    "bytes_received": 0,
                    "ops": 1,
                    "successful_ops": 1
                },
                {
                    "category": "get_cors",
                    "bytes_sent": 75,
                    "bytes_received": 0,
                    "ops": 1,
                    "successful_ops": 0
                },
                {
                    "category": "get_obj",
                    "bytes_sent": 555936,
                    "bytes_received": 0,
                    "ops": 3,
                    "successful_ops": 3
                },
                {
                    "category": "list_bucket",
                    "bytes_sent": 37633,
                    "bytes_received": 0,
                    "ops": 6,
                    "successful_ops": 6
                },
                {
                    "category": "list_buckets",
                    "bytes_sent": 698,
                    "bytes_received": 0,
                    "ops": 2,
                    "successful_ops": 2
                },
                {
                    "category": "put_obj",
                    "bytes_sent": 0,
                    "bytes_received": 558144,
                    "ops": 2,
                    "successful_ops": 2
                }
            ],
            "total": {
                "bytes_sent": 594816,
                "bytes_received": 558144,
                "ops": 15,
                "successful_ops": 14
            }
        }
    ]
}

上面输出中，summery里的total信息就足够我们来计费使用了。

usage show对应的admin ops

1 2	GET /{admin}/usage?format=json HTTP/1.1 Host: {fqdn}

http://docs.ceph.com/docs/jewel/radosgw/adminops/#get-usage

radosgw-admin usage trim命令

修剪rgw的usage log，这是因为在长时间使用下，以往的usage log会占用ceph存储空间。在确定不需要的时间段后，通过该命令删除这部分记录。

用法示例：

1
2
3

radosgw-admin usage trim --start-date=2010-01-01 --end-date=2010-12-31
radosgw-admin usage trim --uid=johndoe
radosgw-admin usage trim --uid=johndoe --end-date=2013-12-31

usage trim对应的admin ops

1 2	DELETE /{admin}/usage?format=json HTTP/1.1 Host: {fqdn}

http://docs.ceph.com/docs/jewel/radosgw/adminops/#trim-usage

RGW log信息

radosgw log功能可以记录每个ops的操作记录，方面查看历史操作和统计信息，使用方法如下。

enable rgw log

1
2
3

[client.rgw.BJ-001-001]
...
    rgw enable ops log = true

或者命令：

ceph daemon /var/run/ceph/ceph-client/ceph-client.rgw.BJ-001-001.7455.asok config set rgw_enable_ops_log true
``` 

### 创建log对应的pools

创建rgw log需要的pools

```sh
ceph osd pool create .log 64            ## log_pool
ceph osd pool create .intent-log 64     ## intent_log_pool

rgw log命令

可用通过radosgw-admin log命令操作

$ radosgw-admin log list
[
    "2016-10-11-14-default.553597.94-database_backups"
]

$ radosgw-admin log show --bucket=database_backups --date=2016-10-11-14 --bucket-id=default.553597.94
{
    "bucket_id": "default.553597.94",
    "bucket_owner": "S3User",
    "bucket": "database_backups",
    "log_entries": [
        {
            "bucket": "database_backups",
            "time": "2016-10-11 06:02:20.595010Z",
            "time_local": "2016-10-11 14:02:20.595010",
            "remote_addr": "",
            "object_owner": "S3User",
            "user": "S3User",
            "operation": "GET",
            "uri": "\/database_backups\/a2edb41b-3515-4075-9b37-43dd7daf79dc.xbstream.gz.enc",
            "http_status": "200",
            "error_code": "",
            "bytes_sent": 279072,
            "bytes_received": 0,
            "object_size": 279072,
            "total_time": 0,
            "user_agent": "",
            "referrer": ""
        }
    ],
    "log_sum": {
        "bytes_sent": 279072,
        "bytes_received": 0,
        "total_time": 0,
        "total_entries": 1
    }
}
 
$ radosgw-admin log rm --object=2016-10-11-14-default.553597.94-database_backups

问题

测试发现，无论是部署了几个rgw，因为usage信息是存储在rados系统的.usage pool里的，不区分不同rgw过来的请求，所以通过radosgw-admin usage命令看到的是所有rgw操作的统计信息，这样我们没法区分内网和外网的rgw流量和ops个数。

参考青云，青云支持内网/外网流量和ops统计分开计费。
https://docs.qingcloud.com/guide/qingstor.html#id15

若内网的rgw流量不计费，则可以通过配置rgw的参数实现，可配置的rgw参数为：

rgw_enable_usage_log：true/false           ## 是否enable usage统计
rgw_enable_ops_log：true/false             ## 是否enable ops log统计
rgw_ops_log_rados：true/false              ## 是否记录ops log到rados里，对用pool为：.log
rgw_ops_log_socket_path：<string>          ## 配置ops log的socket path，ops log保存在内存，可以通过socket接口访问