Client: https://www.deepchannel.com/
Role: Senior Data Pipeline Engineer
Objective:
Developed and optimized a custom data pipeline using Python and Prefect to extract, transform, and load data from various sources into a Redshift data warehouse.
Technologies and Tools:
- Data Warehouses: Redshift, Snowflake
- Data Sources: SharePoint, AWS S3
- Programming Language: Python
- Scheduling and Orchestration: Prefect
- Containerization: Docker
- CI/CD: AWS CodeBuild
- Cloud Infrastructure: AWS EC2
- Databases: AWS DynamoDB, MySQL
- Data Visualization: ThoughtSpot
Key Accomplishments:
- Enhanced and maintained a custom data pipeline to extract files from a file system, perform data manipulations and standardization, and load the processed data into a Redshift warehouse.
- Implemented efficient data extraction techniques to handle various file formats and sources.
- Developed data transformation logic to ensure data quality and consistency.
- Optimized data loading processes to improve pipeline performance and scalability.
- Created a modular data-processor component that could be built into a Docker image and deployed to AWS EC2 instances using a CI/CD pipeline.
- Designed and implemented a containerized architecture for the data-processor module.
- Integrated the module with AWS CodeBuild for automated building and deployment.
- Ensured seamless integration with the existing data pipeline infrastructure.
- Ingested data from diverse data warehouses, including SharePoint, Snowflake, and Redshift, to feed the data pipeline.
- Established secure data connections and configured data extraction processes.
- Implemented data validation and error handling mechanisms to ensure data integrity.
- Optimized data ingestion workflows to handle large volumes of data efficiently.
- Built transformation functions to map and convert data from source systems to staging and aggregation layers.
- Designed and implemented data transformation logic using Python.
- Developed reusable and modular functions to handle various data transformations.
- Ensured data consistency and accuracy throughout the transformation process.