Software Engineer - Python | Remote

Position: Senior Software Engineer – Python (LLM Evaluation & Repository Validation)

Type: Contractor Assignment (3 months)

Compensation: $19-$20/hr

Location: Remote

Commitment: 40 hrs/week with some overlap with PST

Role Responsibilities

* Evaluate and analyze GitHub issues across widely used open-source repositories.

* Set up and configure development environments for repositories, including Docker setup.

* Modify and run codebases locally to evaluate LLM performance in debugging and bug-fixing scenarios.

* Evaluate unit test coverage and assess the quality of existing tests.

* Work with real-world Python repositories to identify engineering tasks suitable for LLM evaluation.

* Collaborate with researchers to expand datasets used for training and evaluating AI models.

* Identify repositories and issues that present challenging real-world scenarios for LLMs.

* Support the creation of realistic software engineering tasks based on repository histories.

Requirements

* Strong experience in software development.

* Strong experience with Python programming.

* Proficiency in Git, Docker, and development environment setup.

* Experience working with well-maintained public GitHub repositories.

* Ability to understand, modify, and test complex codebases.

* Strong analytical thinking and problem-solving skills.

* Excellent written English and ability to document findings clearly.

* Ability to work independently in a remote environment.

* Desktop or laptop with a stable internet connection.

Preferred Qualifications

* Experience contributing to open-source projects.

* Exposure to LLM evaluation, AI research, or machine learning projects.

* Experience with developer tools, automation agents, or CI/CD pipelines.

Application Process

* Application Form

* ICF + Profile Review

* Assessment

* Submit