Original: https://www.hbs.edu/ris/PublicationFiles/24-013_d9b45b68-9e74-42d6-a1c6-c72fb70c7282.pdf
The purpose of this paper is to explore the impact of artificial intelligence on the productivity and quality of knowledge workers, drawing conclusions from field experiments. The research team includes experts from Harvard Business School, Wharton School of the University of Pennsylvania, MIT Sloan School of Management and other institutions. The results of the study will help to understand the application of AI in work scenarios.
The release of the Large Language Model (LLM) has drawn attention to the impact of artificial intelligence on the productivity and quality of knowledge workers.The LLM has a significant impact on the performance of knowledge workers, especially in innovation, analysis and writing tasks. A pre-registered randomized controlled experiment allows assessing the impact of LLM on professionals with high human capital.LLM significantly improves productivity and quality within the competence range.
Outside of the LLM's capabilities, the LLM reduces the rate of correctness. Users adopted two different modes of LLM use, "Centaur" and "Cyborg", and LLM created a "jagged technical boundary" with different impacts on the work depending on the task. LLM creates an "uneven technical boundary" with different impacts on the work depending on the task.
Users will need to determine whether tasks are within the capabilities of LLMs and how to collaborate effectively with LLMs. Organizations need to rethink human-computer collaboration, new roles, stakeholders, etc., in order to realize the full potential of LLMs, which will have a profound impact on knowledge work, and users and organizations will need to adapt to this impact. Adopting the "centaur" model, i.e., dividing the work between LLMs and humans based on their respective strengths, is the most effective use of AI that has been demonstrated in current experiments within and beyond the capabilities of LLMs.
Performance Impact of AI in Realistic, Complex and Knowledge Intensive Tasks
Artificial Intelligence (AI) capabilities have created a "rugged technology frontier" where some tasks can be easily accomplished by AI, while others are beyond the capabilities of current AI. In tasks within the frontier of AI capabilities, advisors using AI are significantly more productive, while in tasks outside the frontier, AI output is inaccurate, less useful, and reduces human performance. It is difficult for professionals to know exactly what the boundaries of this frontier might be at a given moment. Professionals who are adept at navigating this frontier gain significant productivity benefits when working with AI. The utility of AI can fluctuate across a professional's workflow, with some tasks located within the frontier and others outside of it. For tasks within the frontier, AI significantly improves the performance and quality of each model specification. AI appeared to both equalize performance differences across ability levels and improve the quality of tasks within the frontier. The challenge that AI use may lead to a reduction in diversity of ideas may pose a challenge to organizations. Artificial intelligence appears to hold promise for significantly impacting human cognitive and problem-solving abilities. The transformational potential of AI and provides insights into utilizing its capabilities to achieve optimal results. Optimism about AI's capabilities in high-end knowledge work tasks such as rapid idea generation, writing, persuasion, strategic analysis, and creative product innovation. The frontiers of AI remain challenging and the understanding of the frontiers needs to be recalibrated. AI may play a similar role in reducing the costs associated with human thinking and reasoning, potentially with broad and transformative impacts.
The impact of AI on advisor performance
AI had a significant impact on counselor performance. gpt + overview treatment had a more pronounced positive impact than gpt only. Overview increased "retention" and was associated with better performance. Factors such as gender, native language fluency, tenure, location, and openness to technology had an impact on the results.AI tools significantly improved task completion and quality. Subjects with lower skill levels are the biggest beneficiaries of AI use.GPT-4 helps generate better quality content, but can also lead to more homogenized output.AI can provide performance benefits in tasks that require intensive human interaction. The AI-treated group showed a significant decrease in performance in tasks located outside the frontier. the AI treatment had a significant negative impact on correctness in experimental tasks located outside the frontier. the AI treatment reduced the time required for subjects to complete the task in experimental tasks located outside the frontier. the AI treatment also reduced the amount of time required to complete the task in experimental tasks located outside the frontier. Subjects using AI provide higher quality advice in experimental tasks located outside the frontier. In the workflow of highly skilled professionals, AI can impact performance in a number of ways. For tasks located within the frontier, AI can improve human performance. For tasks located outside the frontier, over-reliance on AI can lead to errors. the use of AI can improve task completion rates. the use of AI can lead to less diversity in content generation. the use of AI can lead to time savings, but it can also affect the quality of the work.
The Impact of AI on Knowledge Workers
Studies have found that AI has both positive and negative impacts on knowledge workers.
Positive impacts include:
Boost productivity: knowledge workers using AI are able to complete tasks faster, with an average speed increase of 25.11 TP3T.
Improved task quality: the use of AI not only speeds up work, but also leads to a significant improvement in the quality of work, with ratings improving by an average of more than 401 TP3T.
Especially for less capable knowledge workers, the assistance of AI can greatly enhance their performance, with performance gains of up to 43%.
The use of AI can reduce repetitive and computationally intensive tasks in the workflow, thus giving knowledge workers more time and energy to focus on tasks that require more of the unique capabilities of humans.
Negative impacts include:
On some tasks, the use of AI instead reduces performance, such as a 19 percentage point reduction in the correctness rate of advisors using AI on tasks outside of the AI's capabilities.
Over-reliance on AI could lead knowledge workers to abandon their own judgment, which is especially dangerous in tasks that require unique human judgment and creativity.
The use of AI may reduce creative diversity, which is a potential problem for fields that require diverse creativity.
There can be ethical and legal issues regarding the misuse or abuse of AI that require special attention from organizations and employees.
Overall, the impact of AI on knowledge workers is two-fold; it can be used as a powerful tool to improve productivity and task quality, but it can also lead to performance degradation in some cases. Therefore, organizations and employees need to learn how to use AI effectively, while also being aware of its limitations and avoiding over-reliance on it for tasks it is not good at.
Assessing the impact of AI on knowledge worker productivity and quality
The impact of AI on the productivity and quality of knowledge workers was assessed through two randomized controlled experiments. The subjects of the experiments were 758 individual contributor consultants from the Boston Consulting Group (BCG). These consultants were randomly assigned to either a control or experimental group and were asked to complete 18 authentic consulting tasks within five hours. The tasks covered the range of AI capabilities and included analytical, creative, repetitive, and computationally intensive tasks.
In the experiment, advisors in the control group were asked to use traditional advising tools and search engines to complete their tasks, while advisors in the experimental group were allowed to use the GPT-4 to assist them in their work. The researchers assessed the impact of AI by comparing task completion between the two groups. They measured the number of tasks completed by each participant, task completion time, and task quality. Task quality was rated by outside experts who did not know whether each task was completed manually or with AI assistance.
The results of the study showed that knowledge workers who used AI completed an average of 12.21 TP3T more tasks than the control group who did not use AI, and the speed of task completion increased by 25.11 TP3T.In addition, the consultants who used AI produced tasks with a quality score that improved by an average of more than 401 TP3T.Particularly, for the consultants who were less competent, the performance of the consultants who used AI increased by 431 TP3T.These results suggest that AI can significantly improve the productivity and task quality of knowledge workers.
However, the study also found that advisors who used AI instead performed worse than controls who did not use AI in certain tasks that were outside the scope of AI's capabilities. This suggests that the impact of AI is two-fold: it can be used as a powerful tool to improve productivity and task quality, but it can also lead to a decline in performance in some cases. Therefore, organizations and employees need to learn how to use AI effectively, while also being aware of its limitations.
Effectively integrating AI and human work to improve productivity and quality of work
Strategies for effectively integrating AI and human work to improve productivity and quality include:
Task allocation strategy: rationally allocate tasks according to the areas of expertise of AI and humans. For example, let AI handle tasks that require extensive data processing and analysis, while humans focus on tasks that require creativity, emotional intelligence, and complex interpersonal interactions.
Centaurs: This strategy involves human workers using AI in conjunction with tasks that take full advantage of AI's strengths in areas such as information processing and language generation, while retaining human dominance in strategic decision-making and innovative thinking. Human workers need the ability to judge when and how best to use AI.
Cyborg Strategy (Cyborgs): this strategy emphasizes the close integration of humans and AI, where human workers continuously interact with the AI during the task process and optimize the AI's outputs through continuous experimentation and feedback, thus improving efficiency and quality of work.
Training and education: In order to effectively integrate AI, both organizations and employees need to be properly trained and educated to better understand the capabilities and limitations of AI and how to make the most effective use of it.
Monitoring and evaluation: Organizations should monitor the use of AI and regularly evaluate its impact on productivity and quality. This helps identify potential problems with AI and take action accordingly.
Ethical and legal considerations: In integrating AI, organizations need to consider ethical and legal concerns to ensure that the use of AI does not violate customer privacy, intellectual property, or other laws and regulations.
Innovative workflows: redesigning workflows to accommodate the capabilities of AI. This may involve overhauling existing workflows to maximize the potential of AI.
Maintaining Creative Diversity: While AI excels at certain tasks, human workers need to maintain creative diversity to avoid over-reliance on AI leading to creative exhaustion.
With these strategies, organizations can better leverage the benefits of AI while avoiding its potential negative impacts, resulting in both productivity and task quality.
Important points and conclusions made in the document
Important points and conclusions made in the document include:
Impact of AI on Knowledge Work: AI can significantly improve knowledge workers' productivity and task quality. For example, in the experiment, consultants using AI completed an average of 12.21 TP3T more consulting tasks than the control group without AI, and the speed of task completion increased by 25.11 TP3T, with significantly higher quality scores.
Strategies for using AI: The "Centaurs" (centaur strategies) and "Cyborgs" (cyborg strategies) mentioned in the study are two different strategies for using AI. The former refers to advisors who are able to effectively distribute AI and human tasks, while the latter refers to advisors who integrate AI into their workflow and interact frequently with AI.
AI on Creative Tasks: AI is particularly strong on creative tasks, which could have implications for the way innovation is organized in the future. At the same time, however, AI may lead to a decrease in creative diversity, which requires organizations to be cautious in their use of AI and to consider how to maintain creative diversity.
Limitations of AI: The study also found that advisors who used AI performed worse than the control group who did not use AI on certain tasks that were outside the scope of AI's capabilities. This demonstrates the limitations of AI and the problems that can arise from over-reliance on AI.
Impact of AI on marketing: AI can assist in generating marketing strategies, including market segmentation, selecting target markets, and developing marketing slogans. However, the potential shortcomings of AI in terms of creative diversity also need attention.
Role of AI in Product Development:AI can assist in product development, from the initial idea to the final market launch. For example, AI can provide product ideas, help select the best product, design product prototypes, perform market segmentation and marketing strategy development.
Impact of AI on decision-making: the use of AI may affect the decision-making process of human workers, including increased reliance on AI and the possibility that AI may cause decision-makers to abdicate their judgment in some cases.
Potential Competitive Advantages of AI in the Market: AI can help companies predict market trends, optimize product design, and improve production efficiency to gain an edge in market competition.
Potential impact of AI on workflows: while AI can transform workflows and increase efficiencies, it also requires proper training and education of employees so that they can effectively utilize AI.
Potential of AI in industries: the study highlights the potential of AI in various industries, especially in those tasks that require a lot of data processing and analysis.
These insights and conclusions are not only based on the researcher's analysis, but are also supported by specific experimental data and case studies. For example, the randomized controlled experiments mentioned in the study, the analysis of the performance of consultants from the Boston Consulting Group, and specific descriptions of the role of AI in product development and marketing provide empirical support for these ideas.
Limitations of AI
Limitations of AI include:
AI's propensity for error: AI models have a tendency to produce erroneous results, including hallucinations or confabulations, as well as problems with math and providing citations. These include hallucinations or confabulations, as well as problems with math and providing citations. For example, in the experiment, advisors who used AI were less likely to be correct on some tasks than the control group who did not use AI, a drop of 19 percentage points.
AI's range of capabilities: AI's capabilities are not evenly distributed; it has a "jagged technological frontier". This means that the AI is very good at some tasks, but poor at other seemingly similar tasks. This can make it difficult to find areas where AI performs significantly worse than human workers when designing tasks.
Impact of AI on human judgment: the study found that human workers may over-rely on AI outputs and even abandon their own judgment when using AI. For example, in the experiment, human workers rated higher on AI-generated incorrect answers, suggesting that AI output may influence human workers' judgment.
Potential shortcomings of AI in creative diversity: while AI excels at creative tasks, it can lead to a reduction in creative diversity. This is a potential problem for fields that require diverse creativity.
The Importance of High Quality Tips
The idea that "high levels of retention can come from engaging with high-quality prompts" suggests that knowledge workers who utilize AI assistants (e.g., ChatGPT) may be able to improve productivity and task quality by providing high-quality prompts to guide the AI to generate more useful content. This high-quality prompting behavior may include helping the AI iteratively improve its responses until it reaches a state of perfection before incorporating much of its content into its own answers.
In the provided documentation, the researchers experimentally observed that some participants tended to rely heavily on the AI's output, i.e., to "retain" large amounts of AI-generated content. These participants may have guided the AI to produce high-quality content through well-designed prompts based on an understanding of the AI's capabilities and limitations. This cueing behavior may be an important factor leading to high retention, as it enables effective use of the AI's capabilities while ensuring that the final output is accurate and useful.
For example, the document mentions that in the creative problem solving experiment, participants were asked to conceptualize new product ideas through a series of questions. The researchers found that those participants who were able to utilize the AI effectively not only included more AI-generated content in their answers, but the quality of this content was also rated highly by external experts. This suggests that guiding the AI with high-quality prompts can significantly improve the productivity of creative tasks and the quality of results.
However, the researchers also point out that the current analysis of the study has not been able to distinguish whether high retention is due to high-quality cueing behavior or due to an overreliance on AI. Therefore, this observation, while providing a possible explanation, needs to be confirmed by further research. At the same time, it suggests that in practice, we should emphasize proper training and education of employees so that they can better understand and utilize the capabilities of AI while avoiding over-reliance on it.
All points mentioned in the document
Here is a list of all the ideas mentioned in the document:
1. Impact of AI on knowledge workers: AI can significantly improve the productivity and task quality of knowledge workers.
2. Strategies for using AI: The "Centaurs" and "Cyborgs" mentioned in the study are two different strategies for using AI.
3. AI on creative tasks: AI is particularly strong on creative tasks, which could have implications for the way innovation is organized in the future.
4. Limitations of AI: The study also found that advisors using AI performed worse than a control group not using AI on certain tasks that were outside the scope of AI's capabilities.
5. Impact of AI on marketing: AI can assist in generating marketing strategies, including market segmentation, selection of target markets, and development of marketing slogans.
6. Role of AI in product development: AI can assist in product development from initial idea to final market launch.
7. Impact of AI on decision-making: The use of AI may affect the decision-making processes of human workers, including increased reliance on AI.
8. Potential competitive advantages of AI in the marketplace: AI can help companies predict market trends, optimize product design, and improve productivity.
9. Potential impact of AI on workflows: AI can change workflows and increase efficiency, but at the same time it requires proper training and education of employees.
10. Potential for AI applications in industries: The study highlighted the potential for AI applications in various industries, especially in those tasks that require large amounts of data processing and analysis.
11. AI's propensity for error: AI models have a tendency to produce erroneous results, including "hallucinations" or "confabulations".
12. Range of AI capabilities: AI capabilities are not evenly distributed; there is a "jagged technological frontier".
13. Impact of AI on human judgment: Studies have found that human workers may become overly reliant on the output of AI and even abandon their own judgment when using AI.
14. Potential shortcomings of AI in creative diversity: Although AI excels in creative tasks, it may lead to less creative diversity.
15. Importance of high-quality prompts: Knowledge workers improve productivity and task quality by providing high-quality prompts to guide AI to generate more useful content.
16. Factors influencing retention: The researchers observed that some participants tended to rely heavily on AI output, i.e., to "retain" large amounts of AI-generated content.
17. Role of AI in creative problem solving: AI can assist in generating marketing strategies, including market segmentation, selection of target markets, development of marketing slogans, etc.
18. Role of AI in product development: AI can assist in product development from initial idea to final market launch.
19. Impact of AI on marketing: AI can assist in generating marketing strategies, including market segmentation, selection of target markets, and development of marketing slogans.
20. Role of AI in product development: AI can assist in product development from initial idea to final market launch.
21. Impact of AI on decision-making: The use of AI may affect the decision-making processes of human workers, including increased reliance on AI.
22. Potential competitive advantages of AI in the marketplace: AI can help companies predict market trends, optimize product design, and improve productivity.
23. Potential impact of AI on workflows: AI can change workflows and increase efficiency, but it also requires proper training and education of employees.
24. Potential for AI applications in industries: The study highlighted the potential for AI applications in various industries, especially in those tasks that require large amounts of data processing and analysis.
25. AI's propensity for error: AI models have a tendency to produce erroneous results, including "hallucinations" or "confabulations".
26. Range of AI capabilities: AI capabilities are not evenly distributed; there is a "jagged technological frontier".
27. Impact of AI on human judgment: Studies have found that human workers may become overly reliant on the output of AI and even abandon their own judgment when using AI.
28. Potential shortcomings of AI in creative diversity: Although AI excels in creative tasks, it may lead to less creative diversity.
29. Importance of high-quality prompts: Knowledge workers improve productivity and task quality by providing high-quality prompts to guide AI to generate more useful content.
30. Factors influencing retention: The researchers observed that some participants tended to rely heavily on AI output, i.e., to "retain" large amounts of AI-generated content.
These perspectives cover the potential impact of AI on knowledge workers, marketing, product development, and many other areas, as well as limitations and strategies to be aware of when using AI.
LLM's release and impact
The release of the LLM (Large Language Model) has drawn attention to the impact of artificial intelligence on the productivity and quality of knowledge workers.The LLM has a significant impact on the performance of knowledge workers, especially in innovation, analysis, and writing tasks. To assess the impact of LLM on high human capital professionals, researchers conducted a pre-registered randomized controlled experiment. The results of the experiment showed that LLM significantly increased productivity and quality within the range of competencies, but decreased correctness outside the range of competencies.
Users adopt two different modes of LLM use, "Centaur" and "Cyborg". LLM creates a "jagged technological boundary" with different impacts on the work depending on the task. LLM creates a "jagged technical boundary" that affects work differently depending on the task. Users need to determine if a task is within the capabilities of LLM and how to collaborate effectively with LLM. Organizations need to rethink human-computer collaboration, new roles, stakeholders, etc. to realize the full potential of LLM.
LLM will have a profound impact on knowledge work, and users and organizations will need to adapt. Adopting the "centaur" model, i.e., dividing the work between LLM and humans based on their respective strengths, is the most effective use of AI that has been demonstrated in current experiments within and beyond the capabilities of LLM.
Randomized controlled experimental methods
The study used a pre-registered randomized controlled trial approach to assess the impact of LLM on high human capital professionals. The experiment involved 758 consultants from the Boston Consulting Group who were asked to complete 18 authentic consulting tasks. The experimental group was allowed to use LLM to assist in completing the tasks, while the control group used traditional methods. The researchers assessed the impact of LLM by comparing task completion between the two groups.
The results showed that knowledge workers who used LLM completed an average of 12.21 TP3T more tasks than the control group who did not use LLM, and the speed of task completion increased by 25.11 TP3T.In addition, the consultants who used LLM produced tasks with an average improvement in quality ratings of more than 401 TP3T.Particularly, for the consultants with lower competencies, the performance of the consultants who used LLM increased by 431 TP3T.
However, the study also found that consultants using LLM instead performed less well than a control group that did not use LLM on certain tasks that were outside the scope of LLM's capabilities. This suggests that the impact of LLM is twofold: it can be used as a powerful tool to improve productivity and task quality, but it can also lead to performance degradation in certain situations. Therefore, organizations and employees need to learn how to use LLM effectively, while also being aware of its limitations.
How to effectively integrate AI and human work
In order to effectively integrate AI and human work to improve productivity and quality, the following strategies can be adopted:
1. Task allocation strategy: rationally allocate tasks according to the areas of expertise of AI and humans. For example, let AI handle those tasks that require large amounts of data processing and analysis, while humans focus on tasks that require creativity, emotional intelligence and complex interpersonal interactions.
2. Centaurs: This strategy involves human workers using AI in conjunction with tasks that take full advantage of AI's strengths in information processing and language generation, while retaining human dominance in strategic decision-making and innovative thinking. Human workers need the ability to judge when and how best to use AI.
3. Cyborg strategy (Cyborgs): This strategy emphasizes the close integration of humans and AI, with human workers continuously interacting with the AI during the task process and optimizing the AI's outputs through continuous experimentation and feedback, thus improving efficiency and quality of work.
4. Training and education: In order to effectively integrate AI, both organizations and employees need to be properly trained and educated to better understand the capabilities and limitations of AI and how to make the most effective use of it.
5. Monitoring and evaluation: Organizations should monitor the use of AI and regularly evaluate its impact on productivity and quality of work. This helps to identify potential problems with AI and take action accordingly.
6. Ethical and legal considerations: In integrating AI, organizations need to consider ethical and legal issues to ensure that the use of AI does not infringe on customer privacy, intellectual property rights or other laws and regulations.
7. Innovative workflows: Re-engineering workflows to accommodate the capabilities of AI. This may involve overhauling existing workflows to maximize the potential of AI.
8. Maintaining creative diversity: Although AI excels at certain tasks, human workers need to maintain creative diversity to avoid over-reliance on AI leading to creative exhaustion.
With these strategies, organizations can better leverage the benefits of AI while avoiding its potential negative impacts, resulting in both productivity and task quality.
reach a verdict
The release of LLM has brought productivity and quality gains to knowledge workers, but there are limitations and risks. To make better use of LLM, users and organizations need to adopt appropriate strategies, including rethinking human-computer collaboration, new roles, stakeholders, etc., as well as learning how to work effectively with LLM. There is also a need to pay attention to the ethical and legal issues of AI to avoid negative impacts of AI use. Only on the basis of fully understanding and coping with the potential risks of AI can the advantages of AI be better utilized to achieve both efficiency and quality.
