Skip to Content
Fredrick Barclay

Project · 2017 - 2020

Rackspace Cloud Monitoring

Operations Data Pipelines

Rackspace Fanatical Support for AWS Product Guide cover, release November 2019
Official Rackspace “Fanatical Support for AWS Product Guide” (November 2019) — Rackspace’s AWS cloud operations offering during my tenure. — Source: Rackspace (PDF) ↗

Incremental ingestion, PySpark transformations, SQL reconciliation, and monitored warehouse loads that consolidate cloud operations data into reporting-ready datasets.

Task & inputs
Cloud resource telemetry, customer account records, application logs, and operational reporting feeds.
Method
Implemented incremental ingestion, PySpark transformations, SQL reconciliation, and monitored warehouse loads.
Deliverables
Consolidated operational datasets, reporting tables, scheduled validation checks, and ETL alerts.
Technologies
  • Python
  • pandas
  • SQLAlchemy
  • PySpark
  • AWS Glue
  • Amazon S3
  • Redshift
  • PostgreSQL
  • Jenkins
duplicate ingestion records across 84 recurring loads
38%fewer
duplicate ingestion records across 84 recurring loads
failed-job investigation over 160 monitored pipeline runs
31%shorter
failed-job investigation over 160 monitored pipeline runs

The context

Between 2017 and 2020 Rackspace grew into a multicloud managed-services provider, supporting AWS, Microsoft Azure, and Google Cloud and acquiring TriCore, Datapipe, RelationEdge, and Onica. Its public Rackspace Monitoring service is API-driven: remote checks (HTTP, TCP, Ping) from six monitoring zones, agent checks for CPU, memory, load, filesystem, and network, and a Metrics layer described as “a distributed system that ingests and processes time-series data.”

At that scale, operational reporting depends on joining infrastructure telemetry, customer account records, and application logs that live in different systems.

The problem

Operational data arrived from multiple sources on recurring schedules. Without a consolidated pipeline, reports depended on redundant queries against source systems, duplicates crept in across recurring loads, and when a scheduled job failed it took time to find which records were incomplete and why.

What I built

  • Python ETL pipelines that consolidate cloud infrastructure metrics, customer account records, and application logs into centralized datasets for operational reporting and analytics.
  • Incremental ingestion (Python, pandas, SQLAlchemy, REST APIs): timestamp-based extraction, schema normalization, duplicate detection, and batch processing. Result: 38% fewer duplicate ingestion records across 84 recurring loads.
  • Lake-to-warehouse processing (Amazon S3, AWS Glue, Amazon Redshift): raw data ingestion, partitioned storage, PySpark transformations, and SQL-based warehouse loading for downstream analytics and reporting.
  • Data quality & monitoring (PostgreSQL, Python, Jenkins, Amazon CloudWatch): source-to-target reconciliation checks, automated tests, error logging, and failure notifications. Result: 31% shorter failed-job investigation time over 160 monitored pipeline runs.

Engineering notes

  • Why timestamp-based incremental extraction: pulling only records changed since the last successful watermark avoids re-reading whole source tables, which cuts redundant database queries and keeps recurring loads small.
  • Why duplicate detection at ingestion: recurring loads overlap at their boundaries; deduplicating before the warehouse keeps reporting tables consistent and avoids double counting.
  • Why partitioned storage in S3: partitioning raw data lets Glue/PySpark jobs process only the slices they need and makes reloads of a single period straightforward.
  • Why source-to-target reconciliation: comparing counts and keys between source and warehouse turns a vague “the report looks wrong” into a specific list of incomplete records, which is what shortened failed-job investigations.

Gallery

Rackspace Intelligence dashboard with open alerts, entity alarm state, and monitoring activity log
Rackspace Intelligence dashboard — open alerts, entity alarm state (critical / warning / OK), and the monitoring activity log. — Source: Rackspace Technology Docs ↗
Rackspace Intelligence default graphs for memory and CPU usage over time
Rackspace Intelligence default graphs — memory and CPU time series for a monitored entity. — Source: Rackspace Technology Docs ↗
Grafana dashboard visualizing Rackspace Metrics time-series data for a MySQL database
Grafana dashboard built on Rackspace Metrics time-series data (MySQL connections, queries, and uptime). — Source: Rackspace Technology Docs ↗
Rackspace Monitoring workflow diagram: remote checks from monitoring zones, alarm status, and notifications
Rackspace Monitoring workflow from the official API docs — remote checks from monitoring zones (ORD, LON, DFW), alarm evaluation, and notifications. — Source: rackerlabs/docs-cloud-monitoring ↗

Public product resources

Sources

Company and product facts on this page come from the public sources below. Details of my own work come from my résumé.

  1. Rackspace Technology S-1/A (2020)
  2. Rackspace Monitoring FAQ
  3. Getting started with Rackspace Monitoring

I build reliable backend, data, and machine learning systems for identity security and infrastructure observability. I'm open to senior AI, backend, and data engineering roles. The best way to reach me is by email or on LinkedIn.

Fredrick Barclay · Conroe, Texas