Project · 2017 - 2020
Rackspace Cloud Monitoring
Operations Data Pipelines

Incremental ingestion, PySpark transformations, SQL reconciliation, and monitored warehouse loads that consolidate cloud operations data into reporting-ready datasets.
- Task & inputs
- Cloud resource telemetry, customer account records, application logs, and operational reporting feeds.
- Method
- Implemented incremental ingestion, PySpark transformations, SQL reconciliation, and monitored warehouse loads.
- Deliverables
- Consolidated operational datasets, reporting tables, scheduled validation checks, and ETL alerts.
- Technologies
- Python
- pandas
- SQLAlchemy
- PySpark
- AWS Glue
- Amazon S3
- Redshift
- PostgreSQL
- Jenkins
- duplicate ingestion records across 84 recurring loads
- 38%fewerduplicate ingestion records across 84 recurring loads
- failed-job investigation over 160 monitored pipeline runs
- 31%shorterfailed-job investigation over 160 monitored pipeline runs
The context
Between 2017 and 2020 Rackspace grew into a multicloud managed-services provider, supporting AWS, Microsoft Azure, and Google Cloud and acquiring TriCore, Datapipe, RelationEdge, and Onica. Its public Rackspace Monitoring service is API-driven: remote checks (HTTP, TCP, Ping) from six monitoring zones, agent checks for CPU, memory, load, filesystem, and network, and a Metrics layer described as “a distributed system that ingests and processes time-series data.”
At that scale, operational reporting depends on joining infrastructure telemetry, customer account records, and application logs that live in different systems.
The problem
Operational data arrived from multiple sources on recurring schedules. Without a consolidated pipeline, reports depended on redundant queries against source systems, duplicates crept in across recurring loads, and when a scheduled job failed it took time to find which records were incomplete and why.
What I built
- Python ETL pipelines that consolidate cloud infrastructure metrics, customer account records, and application logs into centralized datasets for operational reporting and analytics.
- Incremental ingestion (Python, pandas, SQLAlchemy, REST APIs): timestamp-based extraction, schema normalization, duplicate detection, and batch processing. Result: 38% fewer duplicate ingestion records across 84 recurring loads.
- Lake-to-warehouse processing (Amazon S3, AWS Glue, Amazon Redshift): raw data ingestion, partitioned storage, PySpark transformations, and SQL-based warehouse loading for downstream analytics and reporting.
- Data quality & monitoring (PostgreSQL, Python, Jenkins, Amazon CloudWatch): source-to-target reconciliation checks, automated tests, error logging, and failure notifications. Result: 31% shorter failed-job investigation time over 160 monitored pipeline runs.
Engineering notes
- Why timestamp-based incremental extraction: pulling only records changed since the last successful watermark avoids re-reading whole source tables, which cuts redundant database queries and keeps recurring loads small.
- Why duplicate detection at ingestion: recurring loads overlap at their boundaries; deduplicating before the warehouse keeps reporting tables consistent and avoids double counting.
- Why partitioned storage in S3: partitioning raw data lets Glue/PySpark jobs process only the slices they need and makes reloads of a single period straightforward.
- Why source-to-target reconciliation: comparing counts and keys between source and warehouse turns a vague “the report looks wrong” into a specific list of incomplete records, which is what shortened failed-job investigations.
Gallery




Public product resources
Sources
Company and product facts on this page come from the public sources below. Details of my own work come from my résumé.