Client: https://www.deepchannel.com/

Role: Senior Data Pipeline Engineer

Objective: Developed and optimized a custom data pipeline using Python and Prefect to extract, transform, and load data from various sources into a Redshift data warehouse.

Technologies and Tools:

• Data Warehouses: Redshift, Snowflake
• Data Sources: SharePoint, AWS S3
• Programming Language: Python
• Scheduling and Orchestration: Prefect
• Containerization: Docker
• CI/CD: AWS CodeBuild
• Cloud Infrastructure: AWS EC2
• Databases: AWS DynamoDB, MySQL
• Data Visualization: ThoughtSpot

Key Accomplishments:

1. Enhanced and maintained a custom data pipeline to extract files from a file system, perform data manipulations and standardization, and load the processed data into a Redshift warehouse.
   • Implemented efficient data extraction techniques to handle various file formats and sources.
   • Developed data transformation logic to ensure data quality and consistency.
   • Optimized data loading processes to improve pipeline performance and scalability.
2. Created a modular data-processor component that could be built into a Docker image and deployed to AWS EC2 instances using a CI/CD pipeline.
   • Designed and implemented a containerized architecture for the data-processor module.
   • Integrated the module with AWS CodeBuild for automated building and deployment.
   • Ensured seamless integration with the existing data pipeline infrastructure.
3. Ingested data from diverse data warehouses, including SharePoint, Snowflake, and Redshift, to feed the data pipeline.
   • Established secure data connections and configured data extraction processes.
   • Implemented data validation and error handling mechanisms to ensure data integrity.
   • Optimized data ingestion workflows to handle large volumes of data efficiently.
4. Built transformation functions to map and convert data from source systems to staging and aggregation layers.
   • Designed and implemented data transformation logic using Python.
   • Developed reusable and modular functions to handle various data transformations.
   • Ensured data consistency and accuracy throughout the transformation process.