Azure Data Engineer- Hybrid in Pittsburgh, PA (candidates must be local to Pittsburgh, PA)
US-PA-Pittsburgh
careers
Req #: 21711
Type: Full-Time
|
|||||||||
|
|||||||||
Overview: Location: Hybrid- Pittsburgh, PA (candidates must be local to Pittsburgh, PA) Job Type: Full Time/Permanent Work Authorization: No Sponsorship The A.C.Coy has an immediate need for an Azure Data Engineer for a full time opportunity. Responsibilities: * Automate data pipeline monitoring, error handling, and performance tuning to ensure data integrity and availability * Collaborating with the Digital Data Solutions Team or external consultants as needed * Design, develop, and implement scalable and high-performance data solutions on the Azure platform, specifically architecting data pipelines and ETL/ELT processes utilizing Azure Data Factory and Azure Databricks to extract, transform, and load data from various sources into Azure data storage solutions * Optimize data storage and retrieval mechanisms to ensure efficient processing of large datasets, particularly focusing on the capabilities of Databricks * Develop and maintain data models and schemas to support data analysis and reporting requirements * Implement data security and privacy measures to protect sensitive data * Develop and maintain data ingestion processes to acquire data from various sources, including APIs, databases, and streaming platforms * Implement data quality checks and validation procedures to ensure the accuracy and integrity of data * Design and implement data transformation and enrichment processes using Azure Databricks, Azure Data Factory, and other relevant tools * Document technical design, deployment processes, and best practices for data engineering solutions * Implement data partitioning, indexing, and optimization techniques to improve query performance * Collaborate with data scientists and analysts to operationalize and deploy machine learning models on Azure, utilizing Databricks for model training and deployment * Configure and manage data storage options, including Azure Blob Storage, Azure Data Lake Storage, and Databricks LakeHouse * Collaborate with DevOps teams to deploy and manage data engineering solutions in Azure Qualifications: Education: Bachelor's degree in Computer Science, Computer Engineering, or other technical field * 3-5 years' eperience with Azure data services, including Azure Data Lake Storage, Azure SQL Database, Azure Data Factory, and Azure Databricks. Understanding of Azure Synapse Data Warehouse preferred * Strong experience in data engineering and architecture, with a focus on Azure cloud technologies * Proficiency in data-centric programming languages, including SQL, Python, and PowerShell * Knowledge of data integration techniques and tools, such as ETL/ELT processes, data pipelines, and data warehousing concepts * Familiarity with data modeling and schema design principles * Understanding of data security and privacy regulations, including GDPR and CCPA * Azure certifications, such as Microsoft Certified: Enterprise Data Engineer Associate, are a plus
Share this job:
