LLM Data Quality Analyst

1 week ago


Sunnyvale, United States US Tech Solutions Full time

Duration: 5 Months Contract



Job Description:

  • We are seeking a highly skilled data quality expert with a strong understanding of Large Language Models (LLMs) to join our team as a Quality Analyst for Human-created Data.
  • In this role, you will liaise closely with the highly technical modeling teams as well as vendor leads, understand the data needs of LLM modeling teams, and ensure that the data produced by expert content creators meets the highest quality standards.
  • You’ll need a strong ability to understand LLM use cases, have a nuanced understanding of good versus great content, and be able to use quantitative and qualitative techniques to analyze LLM datasets.



Responsibilities:

  • Ensure high quality during task development:
  • Define data requirements based on a deep understanding of the modeling team goals and study of model loss patterns.
  • Design detailed instructions for content creator teams, illustrating nuanced differences with clear examples.
  • Rapidly iterate with content-production vendors on small batches of data to arrive at desired data quality.
  • Measure and demonstrate data quality during data collection:
  • Drive processes to deliver benchmark datasets for assessing data quality.
  • Benchmark vendor data quality against competitor models.
  • Use quantitative techniques to analyze vendor-produced data to assure high quality and drive rater pool optimization.
  • Audit datasets for quality issues and develop tools for accelerating qualitative analysis.
  • Use modeling and experimentation techniques to demonstrate data impact and identify blind spots.
  • Communicate on data quality:
  • Send out regular reports to project stakeholders on data quality.
  • Track key metrics such as iteration time and vendor quality audit results.
  • Author best practices and guideline documents (e.g. guidelines for teams wanting to work with vendors for human data, best practices for instruction design, etc).



Experience:

  • 3+ years of experience in Technical Writing, Quality Assurance or a related field.
  • Excellent writing and editing skills, with the ability to adapt to different styles and tones.
  • Strong analytical and problem-solving skills.
  • Ability to work independently and as part of a team.



Skills:

  • Data Quality
  • LLM processes like Pre-training, RLHF, SFT, Evals


Education:

  • Bachelor's degree in English, Creative Writing, Linguistics, Science, Computer Science, or a related field.



About US Tech Solutions:

US Tech Solutions is a global staff augmentation firm providing a wide range of talent on-demand and total workforce solutions. To know more about US Tech Solutions, please visit www.ustechsolutions.com.


US Tech Solutions is an Equal Opportunity Employer.All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, disability, or status as a protected veteran.


Recruiter Details:

Name: Ashik

Email: ashik.a@ustechsolutionsinc.com

Internal Id: 24-26342



  • sunnyvale, United States US Tech Solutions Full time

    Duration: 5 Months ContractJob Description:We are seeking a highly skilled data quality expert with a strong understanding of Large Language Models (LLMs) to join our team as a Quality Analyst for Human-created Data.In this role, you will liaise closely with the highly technical modeling teams as well as vendor leads, understand the data needs of LLM...

  • Business Analyst 2

    2 weeks ago


    Sunnyvale, United States The Judge Group Full time

    Our client is currently seeking a Business Analyst 2 We are seeking a highly skilled data quality expert with a strong understanding of Large Language Models (LLMs) to join our team as a Quality Analyst for Human-created Data. In this role, you will liaise closely with the highly technical modeling teams as well as vendor leads, understand the data needs of...

  • Business Analyst 2

    2 weeks ago


    Sunnyvale, United States The Judge Group Full time

    Our client is currently seeking a Business Analyst 2We are seeking a highly skilled data quality expert with a strong understanding of Large Language Models (LLMs) to join our team as a Quality Analyst for Human-created Data. In this role, you will liaise closely with the highly technical modeling teams as well as vendor leads, understand the data needs of...


  • Sunnyvale, United States META Full time

    Summary: Meta’s Reality Labs Quality and Compliance team is seeking an experienced Data Analyst to provide technical Analytics solutions for the millions of Quest and Ray Ban Meta devices we ship.You’ll be responsible for developing data products such as statistical models and forecasts, complex queries, and Tableau dashboards. You will also play a...

  • Quality Data Analyst

    3 weeks ago


    Sunnyvale, United States META Full time

    Summary: Meta’s Reality Labs Quality and Compliance team is seeking an experienced Data Analyst to provide technical Analytics solutions for the millions of Quest and Ray Ban Meta devices we ship.You’ll be responsible for developing data products such as statistical models and forecasts, complex queries, and Tableau dashboards. You will also play a...

  • Data Analyst

    6 days ago


    Sunnyvale, United States GForce Life Sciences Full time

    Business Intelligence Data Analyst Must be able to work on a W2 Primary Function of Position:The Data Analyst will be a member of the RA/QA Quality Data Analytics team and use technical expertise with the understanding of business needs to turn data into useful and actionable information. In this role, the analyst will support the RA/QA business needs for...

  • Data Analyst

    4 weeks ago


    Sunnyvale, California, United States Apple Full time

    At Apple, we are seeking a skilled Data Analyst to contribute to the development of innovative computer vision and machine learning solutions. The ideal candidate will have a strong background in data processing and analysis, as well as experience with machine learning algorithms and model optimization techniques.Key Responsibilities:Evaluate machine...

  • Data Analyst

    4 weeks ago


    Sunnyvale, California, United States Diverse Lynx Full time

    Role Overview:As a skilled Data Analyst at Diverse Lynx LLC, you will be responsible for leveraging your expertise in SQL, Snowflake, and Oracle to drive data analysis and solution design. With a strong background in data analysis and a passion for problem-solving, you will play a key role in driving business growth and innovation.Key...


  • Sunnyvale, California, United States Grid Dynamics Full time

    Job Title: ML Frameworks Engineer - Big Data ExpertAbout the Role:Grid Dynamics is seeking a highly skilled ML Frameworks Engineer to join our team. As a key member of our data analytics team, you will be responsible for building and maintaining machine learning models that drive business growth and improve customer experience.Key Responsibilities: Develop...


  • Sunnyvale, California, United States Apple Full time

    Job DescriptionApple's Manufacturing Systems & Infrastructure (MSI) team is seeking a Senior Software Engineer to develop extraordinary products with a deep appreciation for user experience. As a key member of our team, you will be responsible for defining and developing ML Platform and frameworks for generative AI powered applications at Apple Product...

  • Data Analyst

    4 weeks ago


    Sunnyvale, CA, United States Genesis10 Full time

    We are seeking a highly skilled Data & Business Intelligence Analyst to join our team at Genesis10. This 12-month contract position offers a unique opportunity to work with cross-functional partners and develop data foundations, data pipelines, data visualization assets, and analysis to enable data-driven decision-making for the Customer Success...


  • Sunnyvale, CA, United States Veear Full time

    Job Title: Data Analyst/ScientistJob Description:We are seeking a highly skilled Data Analyst/Scientist to join our team at Veear. As a key member of our data analytics team, you will be responsible for analyzing and interpreting complex data sets to inform business decisions.Key Responsibilities:* Develop and maintain advanced data models and algorithms to...

  • Senior Data Analyst

    4 weeks ago


    Sunnyvale, California, United States Hire Talent Full time

    Job Title:Business Analyst-III/Senior Data ScientistJob Location:RemoteJob Duration:3 Months on W2Note:SQL assessment required with candidate submittalWe are looking for a Senior Data Scientist who will work as a "floater" for the analytics team, moving from team to team, and focusing on the highest-priority projects.Some example projects might...


  • Sunnyvale, California, United States Genesis10 Full time

    We are seeking a highly skilled Data and Business Intelligence Analyst to join our team at Genesis10. This is a 12-month contract position with the opportunity for extension.Key Responsibilities:Design, develop, and maintain data tools and analytics dashboards to automate processes and provide real-time insights into customer metrics and product usage.Write...


  • sunnyvale, United States GForce Life Sciences Full time

    Business Intelligence Data Analyst Must be able to work on a W2 Primary Function of Position:The Data Analyst will be a member of the RA/QA Quality Data Analytics team and use technical expertise with the understanding of business needs to turn data into useful and actionable information. In this role, the analyst will support the RA/QA business needs for...


  • Sunnyvale, United States GForce Life Sciences Full time

    Job Title: SAP Business Analyst – Quality Management12 month contractMust be able to work on a W2 Job Summary:As an SAP Business Analyst specializing in Quality Management (QM), you will play a crucial role in the design, development, and maintenance of SAP S/4 Hana Quality Management processes. Key Responsibilities:• Conduct Business process blueprint...

  • Data Analyst

    4 weeks ago


    Sunnyvale, California, United States Saxon Global Full time

    Job Summary:Data Scientist plays a crucial role in driving business growth through data-driven insights. As a Data Scientist at Saxon Global, you will be responsible for designing and building new data set processes for modeling, data mining, and production purposes. Your expertise in data analysis, predictive modeling, and real-time implementation will help...


  • Sunnyvale, United States Talentburst Full time

    Title of the role: SAP Business Analyst – Quality ManagementFind out more about this role by reading the information below, then apply to be considered. Duration: 12+ months Contract/possibility for conversions: NA Onsite/Hybrid/Remote: Hybrid (3 days) in Sunnyvale, CA. Only Locals/Non locals can be submitted: Locals Mode of interview: Zoom No of rounds...


  • sunnyvale, United States TalentBurst Full time

    Title of the role: SAP Business Analyst – Quality Management Duration: 12+ months Contract/possibility for conversions: NA Onsite/Hybrid/Remote: Hybrid (3 days) in Sunnyvale, CA. Only Locals/Non locals can be submitted: Locals Mode of interview: Zoom No of rounds of interview: 2 rounds W2 Preferred (No Sponsorship Required) Job Summary: As an SAP...

  • Data Analyst

    4 weeks ago


    Sunnyvale, California, United States Diverse Lynx Full time

    Data Science RoleWe are seeking a highly skilled Data Scientist to join our team at Diverse Lynx LLC. As a Data Scientist, you will be responsible for developing and implementing machine learning models to drive business growth and improve customer experiences.Key Responsibilities:Design and develop machine learning models to solve complex business...