Simplest procblock Demo

This is the simplest demo I could think of, it features one script that is run by procblock, specified by a YAML file: simplest.yaml:

run:
 - script: simplest.py

Which runs simplest.py:

import random

def ProcessBlock(pipe_data, block, request_state, input_data, tag=None, cwd=None, env=None, block_parent=None):
 """Simplest demo possible."""
 data = {'random':random.randint(0, 100)}
 return data

To invoke procblock, run:

cd demo_simplest
./procblock simplest.yaml

This will invoke the script “simplest.py”, which has a standard module function ProcessBlock().  This is the standardized method for all code process blocks, and allows them to chain together, passing relevant information, and also shared data between them.  In this case, there is only one script, so it is simply returning it’s result.

Example:

monkey:demo_simplest ghowland$ ./procblock simplest.yaml 2> /dev/null
{'run': {'__duration': 0.010447025299072266,
 '__start_time': 1282114491.079386,
 'args': [],
 'random': 99}}
monkey:demo_simplest ghowland$

I redirected STDERR to /dev/null because I am leaving the STDERR logging on until procblock is ready to leave Beta Testing.

For a second example there is a slightly more complicated procblock called: simplest_monitor.yaml

As the name implies, this is the simplest monitor I could think of.  It monitors random numbers simplest.py generates every 5 seconds, stored them in an RRD file, and then graphs it.

Because it runs the script every 5 seconds, it is labled a “long running” process, and so it will continue to run until CTRL-C is pressed, which will send a notification for all threads to exit gracefully, which they will do if they are properly written.

Here is the contents of simplest_monitor.yaml:

run:
 - script: simplest.py
   cache: 5
   thread_id: simplest
   timeseries collect:
     path: simplest.rrd
     interval: 5

     fields:
       random:
         type: GAUGE

     graph:
       - path: simplest.png
         title: Simplest Monitoring Demo
         fields: [random]
         method: STACK
         interval: 10
         vertical label: "Random #s"

__usage:
  name: simplest
  author: Geoff Howland

  # Let this run, so we can monitor it
  longrunning: true

The big additions here are the “timeseries collect” statement, which defines what fields to collect from the run script, and how to graph it, and the addition of a __usage section to the YAML file, which defines the name of the block (simplest), the author, and sets longrunning=True, so that the script wont quit as soon as the thread is created to monitor simplest.py’s results.

Example 2:

Run:

monkey:demo_simplest ghowland$ ./procblock simplest_monitor.yaml 2> /dev/null
Running Thread: Starting: simplest
Waiting for interval thread output: simplest
{'run': {'__duration': 0.41039705276489258,
 '__start_time': 1282114743.028677,
 'random': 6,
 'run_thread.simplest': simplest: Is Running: True  Scripts: ['simplest.py']}}
^CRunning Thread: Quitting: simplest
monkey:demo_simplest ghowland$

The result, after a bit of waiting:


What happens this time is a bit different.  Right away we get a returned object, that shows the duration being quite short, and a single random number, and a field called “run_thread.simplest”.  “simplest” is the thread_id name I gave to this monitor thread.  The value of this is a __repr__() string representation from a RunThread object, and you can see it is running (Is Running: True) and running scripts [simplest.py].  It is also formatted in HTML, because that is where I am using it in testing web server internals right now.  Another Beta artifact.

Nothing else happens in this script, until I press CTRL-C, and it quits.  That is because I have the logging turned off.  With the logging on, it shows what is going on:

monkey:demo_simplest ghowland$ ./procblock simplest_monitor.yaml
DEBUG:20100818000408:procyaml.py:138:ImportYaml: Importing YAML: simplest_monitor.yaml
DEBUG:20100818000408:mainfunctions.py:434:ProcessAndLoop: Long Running Process: Starting...  (CWD: /Users/ghowland/blocks/demo_simplest)
DEBUG:20100818000408:procyaml.py:138:ImportYaml: Importing YAML: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/procblock-201008.1-py2.6.egg/procblock/data/default_tag_functions.yaml
DEBUG:20100818000408:procyaml.py:138:ImportYaml: Importing YAML: /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/procblock-201008.1-py2.6.egg/procblock/data/default_condition_functions.yaml
DEBUG:20100818000408:rrd.py:48:StoreInRrd: Storing RRD Occurrance: simplest.rrd: 1282115048.28: {'random': 67}
DEBUG:20100818000408:rrd.py:183:GraphRrd: Graphing RRD: simplest.rrd
{'run': {'__duration': 0.4735870361328125,
 '__start_time': 1282115048.26404,
 'random': 67,
 'run_thread.simplest': simplest: Is Running: True  Scripts: ['simplest.py']}}
DEBUG:20100818000413:rrd.py:48:StoreInRrd: Storing RRD Occurrance: simplest.rrd: 1282115053.58: {'random': 93}
DEBUG:20100818000413:rrd.py:183:GraphRrd: Graphing RRD: simplest.rrd
DEBUG:20100818000418:rrd.py:48:StoreInRrd: Storing RRD Occurrance: simplest.rrd: 1282115058.84: {'random': 85}
DEBUG:20100818000418:rrd.py:183:GraphRrd: Graphing RRD: simplest.rrd
DEBUG:20100818000424:rrd.py:48:StoreInRrd: Storing RRD Occurrance: simplest.rrd: 1282115064.14: {'random': 57}
DEBUG:20100818000424:rrd.py:183:GraphRrd: Graphing RRD: simplest.rrd
DEBUG:20100818000429:rrd.py:48:StoreInRrd: Storing RRD Occurrance: simplest.rrd: 1282115069.48: {'random': 79}
DEBUG:20100818000429:rrd.py:183:GraphRrd: Graphing RRD: simplest.rrd
DEBUG:20100818000434:rrd.py:48:StoreInRrd: Storing RRD Occurrance: simplest.rrd: 1282115074.81: {'random': 32}
DEBUG:20100818000434:rrd.py:183:GraphRrd: Graphing RRD: simplest.rrd
^CDEBUG:20100818000435:mainfunctions.py:451:ProcessAndLoop: ProcessAndLoop: Keyboard Interrupt: Releasing lock: __running
DEBUG:20100818000435:mainfunctions.py:470:ProcessAndLoop: Quitting...
Running Thread: Quitting: simplest
monkey:demo_simplest ghowland$

The first 4 lines show procblock starting up, all YAML files that are loaded are logged, to be able to trace what it is doing.  Then the first run of simplest.py is made, and returned in the “run” tag result, along with the RunThread object which is still running in a thread.

Then every 5 seconds, the simplest.py:ProcessBlock() is invoked, and the result ({‘random’:99}) is stored in simplest.rrd, and then simplest.png is graphed.

Then I hit the CTRL-C key and this caused a shared lock called __running to be released, and the thread that was running simplest.py every 5 seconds quit the next time it was invoked, releasing the process and procblock is finished.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: