Distributed Systems Engineer – Apache Spark Internals (Remote)

About the role

BairesDev is seeking a senior Distributed Systems Engineer to work on the core of Apache Spark. You will focus on Spark internals—Catalyst Optimizer, Tungsten execution, DAG Scheduler, Shuffle subsystem, and memory management—at petabyte scale, contributing directly to the open‑source project and its ecosystem.

Key responsibilities

• Write production‑grade code for Apache Spark, especially Spark SQL and Structured Streaming.
• Debug, profile, and optimize Spark internals (Catalyst, Tungsten, DAG Scheduler, Shuffle, memory management) on massive data workloads.
• Shape architectural direction for Spark performance and scalability.
• Tune JVM behavior (GC, memory layout, concurrency) at the engine level.
• Collaborate with cross‑functional teams and open‑source committers on integrations and ecosystem projects.
• Mentor senior engineers, conduct code reviews, and drive design excellence.

Required profile

• 6+ years of software development experience.
• Strong Java and/or Scala programming skills.
• Solid background in distributed systems and concurrent/parallel programming.
• Working knowledge of Spark internals (Catalyst, Tungsten, DAG Scheduler, Shuffle, memory management).
• Familiarity with JVM performance characteristics (GC, memory, threading).
• Advanced proficiency in English.

Required skills

• Java
• Scala
• Apache Spark
• Spark SQL
• Structured Streaming
• Catalyst Optimizer
• Tungsten execution engine
• DAG Scheduler
• Shuffle subsystem
• JVM performance tuning (GC, memory, threading)
• Concurrent and parallel programming

What we offer

• Fully remote work arrangement.
• Competitive compensation in USD or local currency.
• Provided hardware and software setup for home office.
• Flexible schedule – create your own working hours.
• Paid parental leave, vacation days, and national holidays.
• Innovative, multicultural environment with opportunities for open‑source contributions.