remlite on the path to R.E.M.

Messing around with names for my projects a bit.  I originally created Red Eye Monitor to manage cloud systems, then I expanded the design to manage multiple cloud vendors and private data centers in an integrated fashion.  Then I made a newer-lighter version called remlite, so that I could have something that takes much less time to update, but still has understanding of different services and deployments (prod, staging, QA), so it would work in a larger environment than the original software, but didn’t take as much forethought as Red Eye Monitor’s integrated multi-cloud and datacenter took.

Now remlite has so many improvements over REM, specifically it’s very cool dynamic script running system which integrates any scripts being run with monitoring, graphing and alerting, and it’s equally cool simplification of RRD graphing (which I have always found a pain to set up, but is now fast and easy).

Because remlite is running so smoothly, and has so many good non-organizational features, I’ve decided to make it the core of the Red Eye Monitor project.  Since remlite runs on YAML files and REM runs on a fairly massive MySQL table structure I am going to split them up, so you will always be able to run REM off of YAML files, for a simple case.

The cloud infrastructure is going to be broken out so that wrapping Amazon EC2 API calls and creating your own in-house cloud, or connecting to other cloud vendors will sit in it’s own project.  Temporarily I’m thinking about this as the Red Eye Monitor Cloud, which will abstract all things cloudy.  The requesting of instances, storage, load balanced or external IP addresses, etc, will all be wrapped into the Red Eye Cloud package, which will be stand-alone and contain a cache for read commands so failure to connect to APIs will still result in expected usage.

remlite, which is all the management of interfacing with the cloud, and defining deployments and services to do so, will become Red Eye Monitor.  This is the core package, and will depend on Red Eye Monitor Cloud for interfacing with EC2 or any other cloud vendors (including an internal  Home Cloud).

The old REM specification, with all it’s massive data modeling of your physical and virtual environment, will be known as Red Eye Control.  This will be able to be used as a stand-alone system that could act as the organizational brain for any operations center, and not interface with Red Eye Monitor.

Finally, all the HTTP and XMLRPC stuff I built into REM is being turned into a standalone or embedded web server called dropSTAR (Scripts Templates and RPC).  This is a Python based HTTP/S and XMLRPC server that can run multiple listening thread pools on different ports, and maps page paths to a series of scripts, much like the format remlite scripts, to render data to a webpage or an RPC response.

So the final package list will look like this:

  • Red Eye Monitor (REM).  Currently “remlite”, this will be a YAML configured system for managing services and deployment in a cloud.
  • Red Eye Cloud.  Required by REM, wraps commands for any number of cloud environments, including your Home Cloud hardware wrappers.  Amazon EC2 is included.  This can be run as a library or stand-alone as well.
  • Red Eye Control.  This is a massive brain for your operational environments.  It will track every piece of hardware, down to their components, and all media connections between components to give you a complete understanding of your current infrastructure, and provides a comprehensive dependency graph for alert suppression.  In a standard REM configuration, Red Eye Control will be the source that is used to create all the YAML files that run Red Eye Monitor.  This will separate the brain data from the control scripts, but still allow one to drive the other.
  • dropSTAR.  HTTP and XMLRPC server.  Integrates into any Python program to provide threaded web server with easy Web 2.0 functionality.  Much work has been done to work out a system to easily create dynamic pages and interact and update them from any long-running Python program.  Also works as a drop-in web server, which is much easier to throw onto a non-web system to provide insight into what is going on.

I may also break out the RRD automatic creation and updating, as this is such a hard thing to get right, and being able to dynamically throw any data into RRDs with minimal specification is very useful.  I have to figure out how to do this still, and I’ll probably start looking at it after these other projects are completed and launched.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: