Language Specialist, LLM Programs

We are seeking multilingual raters to assist in creating evaluation sets for LLM model by using the product based on a given scenario. Raters will simulate real-world research tasks, collect source documents, generate prompts, and evaluate responses to ensure high-quality model outputs.

Responsibilities:

• Review the provided topic/domain and refine it into a specific sub-topic (e.g., from "Law" to "Arbitration").
• Conduct online research to collect relevant source documents, ensuring compliance with the given source type and word count guidelines.
• Identify and collect sources from Google search using:
• PDFs
• Google Docs
• Markdown files
• Pasted text from publicly available web pages
• Formulate prompts according to the specified category and input them into LLM model.
• Analyze the responses, and validate if the prompts can be answered by the LLM Model.
• Generate additional prompts within the same category to deepen the evaluation process.

Provide feedback on usability, accuracy, and comprehensiveness of LLM outputs.