Google perftools usage with ceph

Install

apt-get install libtool

git clone https://github.com/gperftools/gperftools.git
cd gperftools
./autogen.sh
./configure
make
make check
make install

Run

ceph tell osd.1 heap start_profiler
ceph tell osd.1 heap stats
ceph tell osd.1 heap dump
ceph tell osd.1 heap stop_profiler
ceph tell osd.1 heap release

Then you can find files in dir ／var/log/ceph such as osd.1.profile.0001.heap

Text display

# pprof --text /usr/bin/ceph-osd /var/log/ceph/osd.1.profile.0001.heap
Using local file /usr/bin/ceph-osd.
Using local file /var/log/ceph/osd.1.profile.0001.heap.
Total: 1.7 MB
     0.8  45.6%  45.6%      0.8  45.6% ceph::log::Log::create_entry
     0.6  38.0%  83.6%      0.6  38.0% std::string::_Rep::_S_create
     0.2  10.3%  93.9%      0.2  10.3% leveldb::Arena::AllocateNewBlock
     0.0   1.8%  95.7%      0.0   1.8% intrusive_ptr_release@7c61d0
     0.0   1.0%  96.7%      0.0   1.2% SimpleMessenger::add_accept_pipe
     0.0   0.8%  97.5%      0.7  43.6% TrackedOp::mark_event
...

* The first column contains the direct memory use in MB.
* The fourth column contains memory use by the procedure and all of its callees.
* The second and fifth columns are just percentage representations of the numbers in the first and fourth columns.
* The third column is a cumulative sum of the second column (i.e., the kth entry in the third column is the sum of the first k entries in the second column.)

Comparing Profiles

# pprof --text /usr/bin/ceph-osd --base=osd.1.profile.0002.heap osd.1.profile.0001.heap
Using local file /usr/bin/ceph-osd.
Using local file osd.1.profile.0001.heap.
Total: 1.5 MB
     0.7  47.2%  47.2%      0.7  47.2% ceph::log::Log::create_entry
     0.6  38.6%  85.8%      0.6  38.6% std::string::_Rep::_S_create
     0.2  10.6%  96.4%      0.2  10.6% leveldb::Arena::AllocateNewBlock
     0.0   1.8%  98.2%      0.0   1.8% intrusive_ptr_release@7c61d0
     0.0   0.6%  98.8%      0.7  44.2% TrackedOp::mark_event
     0.0   0.3%  99.1%      0.0   0.6% decode_message