Distributed Systems Engineer – Apache Spark Internals (Remote)

About the role

We are looking for a Distributed Systems Engineer who will work on the core of Apache Spark, focusing on the engine itself rather than user‑level pipelines. The position is fully remote and involves contributing to Spark SQL, Structured Streaming and other low‑level components at petabyte scale.

Key responsibilities

• Write production‑grade code for Apache Spark, especially in Spark SQL and Structured Streaming.
• Debug, profile and optimise Spark internals such as Catalyst, Tungsten, the DAG Scheduler, Shuffle and memory management.
• Shape architectural decisions to improve Spark performance and scalability.
• Tune JVM behaviour (GC, memory layout, concurrency) at the engine level.
• Collaborate with cross‑functional teams and open‑source committers on integrations and ecosystem projects.
• Mentor senior engineers through code reviews and design critiques.

Required profile

• 6+ years of professional software development experience.
• Strong proficiency in Java and/or Scala.
• Solid background in distributed systems and concurrent/parallel programming.
• Working knowledge of Spark internals (Catalyst, Tungsten, DAG Scheduler, Shuffle, memory management).
• Understanding of JVM performance characteristics (GC, memory, threading).
• Advanced English communication skills.

Required skills

• Java
• Scala
• Apache Spark
• Spark SQL
• Structured Streaming
• Catalyst Optimizer
• Tungsten execution engine
• DAG Scheduler
• Shuffle subsystem
• JVM performance tuning (GC, memory, concurrency)

What we offer

• Fully remote work with flexible scheduling.
• Competitive compensation in USD or local currency.
• Company‑provided hardware and software for home office.
• Paid parental leave, vacation and national holidays.
• Multicultural, innovative environment with opportunities for open‑source contributions.