About the Role
We are looking for a highly skilled and experienced Senior Data Engineer to join our team in Hyderabad. The ideal candidate will bring strong technical expertise in building scalable data platforms and pipelines using modern technologies such as Python, Scala, AWS, Redshift, Terraform, Jenkins, and Docker. This role demands a hands-on professional who thrives in a fast-paced, collaborative environment and is eager to solve complex data problems.
Key Responsibilities
- Design, build, and optimize robust, scalable, and secure data pipelines and platform components.
- Collaborate with data scientists, analysts, and engineering teams to ensure seamless data flow, integration, and availability across systems.
- Develop infrastructure as code using Terraform to automate provisioning and environment management.
- Manage containerized services and workflows using Docker.
- Set up, manage, and optimize CI/CD pipelines using Jenkins for continuous integration and deployment.
- Optimize performance, scalability, and reliability of large-scale data systems on AWS.
- Write clean, modular, and efficient code in Python and Scala to support ETL, data transformation, and processing tasks.
- Support data architecture planning and participate in technical reviews and design sessions.
Must-Have Skills
- Strong hands-on experience with Python, Scala, SQL, and Amazon Redshift.
- Proven expertise in AWS cloud services and ecosystem (EC2, S3, Redshift, Glue, Lambda, etc.).
- Experience implementing Infrastructure as Code (IaC) with Terraform.
- Proficient in managing and deploying Docker containers in development and production environments.
- Hands-on experience with CI/CD pipelines using Jenkins.
- Strong understanding of data architecture, ETL pipelines, and distributed data processing systems.
- Excellent problem-solving skills and ability to mentor junior engineers.
Nice-to-Have
- Experience working in regulated domains like healthcare or finance.
- Exposure to Apache Airflow, Spark, or Databricks.
- Familiarity with data quality frameworks and observability tool