ChatGPT Survey 2.0: Optimized Screening & Survey Management for Continuous Listening
After the success of the survey 1.0 readouts, We introduced several key improvements to our survey program. By incorporating new behavioral and demographic data, we aimed to enhance our sampling. To increase response rates, we streamlined our survey design. We also re-focused on essential metrics and eliminated unnecessary data to better meet decision-makers' needs. Additionally, we incorporated new questions based on leadership feedback to gain nuanced insights into user experiences, focusing on task types, time-saving, attempted tasks, and both actual and perceived barriers.
ChatGPT Enterprise Adoption & ROI Survey
I led the quantitive research operations for a multi-million dollar product purchasing decision for +20,000 BCGers. This involved leading sampling, data quality, data cleaning, incentive management, survey design, survey programming, reporting, and partnering with other ROI research programs for evidence-gathering. We delivered presentations to high-level, and executive leadership teams.
B.Y.O. Agent Interface Conceptual Research & Prototype Testing
I led the conceptual and evaluative research for an interface that enables BCGers to build their own conversational agents. This product enables users to connect to internal data sources and handle complex research tasks uniquely tailored to their needs. The goal of this research was to measure how intuitive the experience of building an agent felt for users. I worked with AI engineers and designers to create the feasible and intuitive interface design and led the design iteration that resulted from this prototype testing.
*deck is sanitized of names and images
ChatGPT Super User Research
To complement our quantitative (behavioral + subjective) ChatGPT adoption research and to support cross-team investigation into use cases, I led the design and execution of a mixed-methods study that leveraged usage frequency data and qualitative interviews to find what constituted a ChatGPT “super user.” We discovered a small number of users who were in the top 10% of usage frequency at BCG were engaging in use cases that pushed the limits of ChatGPT and crossed the capabilities overhang that most users cannot cross. The reason they were able to unlock these advanced capabilities, was their due to their understanding of prompting best practices. Their prompts unlocked extreme time savings and quality improvements. which caused their use cases and prompts went viral among their teams and practice areas.
R&D: Planning an Open-Source LLM Development Initiative at BCG
This strategic R&D initiative at BCG outlines a comprehensive plan to develop open-source LLMs using leading libraries like Llama 2 and Vicuna13-B and aimed to outperform API-based LLM agents for BCG-specific use cases. The project emphasized advanced techniques such as prompt engineering, Retrieval-Augmented Generation (RAG), fine-tuning, Direct Preference Optimization (DPO), and Reinforcement Learning from Human Feedback (RLHF) to ensure alignment of LLM capabilities with the nuanced demands of BCG's consulting environment.
Introducing MT-Bench LLM Evaluation Framework to Drive LLM Vendor Selection for BCG
This project evaluated LLM vendors based on their alignment with high-value use cases and prompt designs for BCG's specific consulting practice areas, using custom MT-Bench analysis question. This effort ensures that user-centric prompts guide the LLM selection process. The initiative integrates LLM based prompt evaluation and user satisfaction evaluations to censure robust monitoring and evaluation of both well-vetted and emergent use cases. This evaluation dashboard and the prompts are available in a dashboard to BCGers.
R&D: Planning BCG's Future Hybrid Recommender System & GenAI Agent Workspace
This project informed high-level senior leadership roadmapping through 2025 and million dollar investment decisions. Our research proposed and began the transformation of BCG’s search-based knowledge discovery experience for BCG's client-facing staff into an advanced recommender system that also employs advanced GenAI agents that automate research queries and tasks. Our vision was based on extensive market research on recommender system algorithms and user resonance testing Our concepts and designs were delivered as a successful part of a visioning workshop to key stakeholders.
Prioritizing Consulting Research Tasks for Automation via Data & Textual Analysis of a Research Request Database
This project used various methods of textual exploratory and seeded keyword analysis (TF-IDF, N-Grams, SpaCy, NLTK, etc) as well as descriptive statistics and frequency analyses on a database containing 3 years (~400k rows) of research requests by consulting practice area. This analysis prioritized feature development for a new LLM agent. The goal was to automate high-frequency and high-duration research tasks, enhance user productivity by reducing the time spent executing and producing research reports across consulting practice areas.
Correlation & Regression Analyses for Focused Feature Development for Navi: A Content-Recommender LLM Agent
This survey-based research enabled users to evaluate factors (i.e. LLM features) critical to the LLM experience. Using correlation and regression analyses. we connected LLM features scores with user satisfaction, adoption likelihood, and recommendation likelihood scores. By analyzing how LLM features/attributes like speed, relevancy, and output structure impact user experience, we were able to make recommendations that drove targeted feature engineering improvements to enhance the post-MVP product prior to organization-wide launch to 20,000 users.
Creating a Replicable UTAUT2-Based Survey Framework for User Insights Across LLM & GenAI Product Ecosystem
I implemented a survey framework utilizes the UTAUT2 model causal chain to gather user perceptions across products in an internal LLM and GenAI ecosystem (5+ products). The survey framework enables multiple types of data analysis, including factors affecting adoption and satisfaction. The collected insights are used strategic product development and user enablement initiatives and enables product leadership to monitor important metrics across GenAI products that serve 40,000+ users.
Maximizing the Value of Explore Product Microenvironment Reports in PathAI’s TxR Platform for BioPharma Leadership
This project addresses challenges in the Translational Research (TxR) platform's interactive reports and proposes an enhanced reporting solution. Issues include variations in quality, operational inefficiencies, and a lack of interactive visualizations for quality control (QC) data exploration.
Embracing Benchmarking: Advanced Analyses for Usage Insights & Data-Driven Decision-Making
This platform analytics and data analysis project focuses on centralizing diverse benchmarking metrics to gain comprehensive insights into user segments, user behaviors, and feature usage to prioritize & optimize platform user experiences and drive innovation for AiSight | Clinical Trials & AiSight.
Productizing (AKA Monetizing) PathAI’s TxR Contributor Network
This was an interesting project with a unique product—the contributor pathologist network. This network is seen as a strategic differentiator for our company. The recommendations were equally as interesting as we found a compelling TxR opportunity to solve the high-impact problem of increasing TxR annotation quality and diversity, while at the same time finding an unanticipated revenue opportunity. This project was complicated as it began with a variety of internal assumptions or hypotheses, which were all invalidated during this study.
Optimizing PathAI’s Clinical Trial Platform for Scale & Discovering a Product-Market-Fit
This was an incredibly successful project that aimed to optimize the CTS platform and improve internal clinical trial operational efficiency from whole slide image (WSI) ingestion to participant report release. Through use of the lead user method and the jobs to be done framework, we found a common thread and a product-market-fit opportunity that solvs a significantly underserved user segment tasked with clinical trial data monitoring and clinical trial management.
Predicting and Improving the Probability of AiSight RUO Adoption in Clinical Diagnostic Settings
As PathAI began commercializing AiSight RUO in diagnostic settings, our organization needed to gather a “complete picture” of what it will take for our company to ensure adoption and improve product adoption likelihood. I led an industry-leading mixed-methods project that uses proven technology adoption & usage models and a survey to study the complex factors that contribute to successful implementation and usage of our products.
Advancing Digital Pathology System (DPS) & Laboratory Information System (LIS) Integrations at PathAI
A lack of interoperability is one of the greatest challenges facing healthcare informatics today, and is one of the primary barriers to digital pathology system adoption in diagnostic and research settings. Structured Data Capture (SDC) is an open-source technical framework that enables the capture and exchange of standardized and structured data in interoperable data entry forms (DEFs) at the point of care.
Design Research Operations
Organization-wide education about the purpose, strategy, and goals of the design research team to maximize our efficacy
As a design research leader, I have helped my organization centralize research endeavors to align disparate, multi-disciplinary research endeavors across various teams under a centralized research team in order to improve these research endeavors methodologically to provide strategic insights to organizational leadership.
Setting Design Research Standards
Creating Organizational Understanding About Design Research Processes & Deliverables
As a design research leader, it is also my responsibility to ensure that my team delivers the best-in-class approaches and deliverables for every design research project, and also to communicate what design research is to non-researchers and what our project stakeholders can expect from our team.
Design Research Mentorship
I’m also a design research mentor outside of PathAI through ADPList. I have mentored over 25 design researchers, and have received very positive reviews, such as, “Riley is super knowledgeable about the design research space, and is an encouraging mentor. In my two sessions with him, he's given me specific, actionable, and compelling advice - just as a design researcher would in a team. Coming from similar worlds in international development, he's helped me feel excited about this field while giving me perspective on what employers are really looking for.”
An overview of past companies and projects
Amazon Robotics—Boston, MA: As a Design Researcher at Amazon Robotics, I planned and analyzed qualitative benchmark / diary studies that used GoPro cameras to record an hour in the life of fulfillment center users prior to and post feature releases and for discovery efforts. Later, I was the research and design lead of a highly prioritized internal operational analytics dashboard (screen blurred for NDA) project. We pilot tested and released an MVP that increased dashboard usage metrics across fulfillment centers in the USA and Canada (know this due to benchmarking study). This was a long-term embedded research project with a product manager and engineering team working through the discovery and the development process. I led the design through iterative design team and product team critique and QA. During development, I sat in on daily sprints and did handoffs to front-end engineers and did design QA with engineering, feature prioritization with product, user stories in JIRA, etc. Lot’s of fun projects and white paper writing at Amazon. Lot’s of MVP prototype testing, collaborating with human factors, research ops, jobs-to-be done framework, future press releases, and heuristic analyses.
LEC & HUNTER—Lima, Peru: As a design researcher & strategist at LEC & HUNTER I conducted a variety of internal design thinking workshops & workshops with customers, facilitated interview training, worked with the Ministry of Public Health (MINSA), and did creative strategy & re-design work. The MINSA work was to align public health insights with conceptual and evaluative research to inform the design and development of technology products that that aim to modify behaviors and reduce public health crises. These products’ user experiences are built based on Education Entertainment (EE) and Social and Behavior Chance (SBCC) workflows and frameworks. Workshop tools are examples and no real results are shown.
WAVELENGTHS LLC—Boston, MA: LLC I started. Designed and developed 2 smartphone applications that function as ethnographic research and photo-journaling guides. They were intended for creative strategy and design research teams who embrace ethnographic research in physical locations (e.g. academic campuses, urban development contexts, etc). I also consulted with design research agencies to improve their own proprietary research applications.
MassArt Design Research Course Instructor—Boston, MA: During my MFA at MassArt, I co-taught the Introductory Design Research course at MassArt. It was fun. We did a bunch of activities, but a particularly fun one was “the hunch workshop,” which leads students through the research, brainstorming, & iterative design process with the goal of designing the “ideal desk” for one of three student typologies. I also wrote a thesis book on the intersection of social research and human-centered design research, which hypothesized: if we create design research apps that enable design research teams to (a) choose from a toolkit of social social and design research methods, and (b) the apps guide these users through the data collection and sharing process, then we can (1) increase the frequency and ease of use of new methods, (2) increase the frequency and ease of use during documentation and results sharing, and (3) improve the overall efficiency of arriving design research teams as measured by their arrival at insights.
Social Research in Public Health—Latin America: In a past prior to my MFA, I worked as a Social (sometimes field-based) Researcher in public health settings for a variety of NGOs, and in the private sector. My teams were focused on facilitating workshops for project innovation (think ideo.org) and participatory monitoring & evaluation (M&E). We often utilized participatory research methods and design thinking methods to ensure inclusive brainstorming, planning, and innovation. We set up long-term longitudinal and monitoring & evaluation studies (i.e. quantitative and qualitative benchmarking studies), and we trained local facilitators to use these methods and develop these programs themselves. We also planned communication & media interventions to promote the theory of change and the ideas for innovation. We worked with production teams to design content and strategies. We frequently planned and implemented a variety of participatory, ethnographic, and social and behavioral change toolkits (SBCC) and methodologies and engaged in common social research approaches (e.g. interviews, focus groups, surveys, etc). I worked with many governments and their research partners for this work (including indigenous tribunal councils). I have a MA in Social Research for Public Health from Ohio University.
Travel Researcher & Journalist—Central America & Mexico: In an even more distant past, I was a travel writer in Mexico. I like to say this is where my professional research career began. The process starts like a lot of projects do today. Thinking of interesting interventions (stories), ranking interest, contacting stakeholders to interview, recording, transcribing, and synthesizing the interviews, and storytelling detailed stories with “thick descriptions” of people, their daily lives, and painting a picture of daily like in unique cultural contexts!
