Graphite

what methods are people using to make sure data is getting in to graphite?

Asked by Matt Snow

i'm looking for some assistance with monitoring metrics going in to graphite.

Is there a simple way to obtain a list of all metrics through the webapp with the last let's say 5-15 values in each metric?
This method seems like a heavy load on the webapp/apache and i don't think is a good approach. just throwing it out there though.

Another method I am considering is to simply pole the raw data from the carbon.agents.*.metricsReceived metric ( http://<server>/render/?_salt=1318353549.484&target=carbon.agents.*.metricsReceived&from=-15minutes&rawData=1) then create low/high thresholds for use with nagios. one downside to this is that as we add more machines and tools to report in to graphite these thresholds will change.

How are others monitoring that data is going in to graphite for specific applications or systems?

Thanks.

Question information

Language:
English Edit question
Status:
Answered
For:
Graphite Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
chrismd (chrismd) said : 		#1

You could capture this by instrumenting whatever piece of software is sending the metrics to graphite to record them efficiently in some other way (ie. expiring memcache keys).

Revision history for this message
Nicholas Leskiw (nleskiw) said : 		#2

It may not as heavy a load as you think, since many of those values may be in carbon's cache or (if you're checking every minute) in a memcached instance. Remember the 2 rules of Optimization:

1. If you don't know what you're doing, don't.

2. Even if you know what you're doing, don't do it *yet*.

Revision history for this message
Launchpad Janitor (janitor) said : 		#3

This question was expired because it remained in the 'Open' state without activity for the last 15 days.

Revision history for this message
chrismd (chrismd) said : 		#4

What I did at a previous company I worked at was to send all monitoring data to a processing application first (custom developed and unforunately not open source) which forwarded metrics to graphite but it also applied thresholds, etc and did alerting. I also made it track heartbeats from all the clients reporting data and alert whenever it stopped getting heartbeats from a particular client.

Revision history for this message
chrismd (chrismd) said : 		#5

answered

Can you help with this problem?

Provide an answer of your own, or ask Matt Snow for more information if necessary.

To post a message you must log in.