Principal Data Engineer – 4265

Our client is a leading player in the content-creation sphere, enabling creators all over the world to reach new heights and expand their brand.  

What You Will Be Doing

• Architect and manage scalable data pipelines, encompassing both single and multi-node ETL solutions
• Establish data quality assurance measures for new and existing pipelines
• Generate derived datasets with enhanced attributes
• Enhance analytics-ready datasets to empower internal and creator-facing tools
• Address issues promptly, collaborating directly with internal data consumers
• Automate pipeline executions using scheduling and orchestration tools
• Handle extensive datasets and utilize various external APIs to enrich data
• Configure database tables for analytics users to access the Data Engineering team’s collected data
• Employ big data technologies to enhance data availability and quality in the AWS cloud
• Lead projects, collaborating with team members and serving as a mentor
• Actively engage in team discussions on technology, architecture, and solutions for both new and existing projects  

What You Will Need

• Bachelor’s degree, preferably in Computer Science or Computer Information Systems
• 6+ years of software engineering experience
• 5+ years of data engineering know-how with Apache Spark or Apache Flink
• 4+ years of hands-on experience running software and services in the cloud
• Skilled in working with DataFrame APIs (Pandas and Spark) for both parallel and single-node processing
• Proficiency in using languages like Python, Scala, etc., with modern data file formats like Parquet and Avro
• Proficiency in SQL for RDBMS and data warehouse solutions, such as Redshift
• Hands-on experience with Data Lake technologies like Delta Lake and Iceberg
• Experience in data acquisition from external APIs at a large scale and parallel processing
• Practical experience supporting ML/AI projects, including deploying pipelines for computing features and using models for inference on large datasets

Why Us

Benefits and Perks:

• Competitive Salary : $180,000 – $210,000 per Year
• Full Health, Vision, and Dental Coverage   

Applicants must be currently authorized to work in the United States on a full-time basis now and in the future.

This position does not offer sponsorship.

#LI-BA1

REF: JOB-4265