Staff Cloud Platform Engineer

We, the Cloud Platform Engineering team at Calix are responsible for the Platforms, Tools, and CI/CD pipelines at Calix. Our mission is to enable Calix engineers to accelerate the delivery of world-class products while ensuring the high availability, reliability, security, and performance expected by our customers.

We are seeking Platform Engineers with proven expertise in the design and building of large-scale Cloud Infrastructure Platforms for Datalake to join our team and help us deliver the foundation on which Calix products can thrive. You will help us achieve our mission through innovative solutions.

Responsibilities:

• Design, build, and maintain a scalable and reliable cloud-native data lake infrastructure using Presto, Alluxio, S3, Hive and Spark technologies.
• Implement integration with Zookeeper for coordination and leader election across Datalake components.
• Configure Presto to access and query data stored in Alluxio and S3, optimizing performance and resource utilization.
• Deploy and manage Spark clusters for data processing and analytics tasks, leveraging Alluxio and S3 as storage layers.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement solutions that meet business objectives.
• Develop and maintain automated workflows for data ingestion, processing, and transformation using Spark and other relevant tools.
• Implement monitoring and alerting solutions to ensure the health and performance of the data lake infrastructure.
• Provide technical guidance and support to junior team members, fostering a culture of continuous learning and improvement.

Requirements:

• Bachelor’s degree in Computer Science, Engineering, or related field; or equivalent work experience.
• 10+ years of experience as a Data Engineer or similar role, with expertise in building and managing cloud-native data lake infrastructure.
• Strong proficiency in Presto, Alluxio, S3, Spark, Hive and Zookeeper, with hands-on experience in designing and implementing solutions using these technologies.
• Experience with Spark Administration, Hive administration, Presto administration and AWS resource management.
• Experience with monitoring of distributed systems.
• Experience with infrastructure as code tools such as Terraform/Ansible for provisioning and managing cloud resources.
• Experience with container orchestration platforms such as Kubernetes for deploying and managing Datalake components.
• Solid understanding of distributed computing principles, data modelling, and database technologies.
• Experience with serverless architecture would be plus like Lambda and EMR
• Experience in Observability and Monitoring in AWS.
• Fast learner and able to troubleshoot complex scenarios while process large volumes of data (Terabytes and Petabytes)
• Excellent problem-solving skills, with the ability to troubleshoot complex issues and implement effective solutions.
• Strong communication and collaboration skills, with the ability to work effectively in a cross-functional team environment.
• Having Databricks experience will be an added advantage.

Optional Skills:

• JVM tuning.
• Alluxio cluster management.