iTranslated by AI
Reading Notes on "How AI Impacts Skill Formation"
Introduction
"How AI Impacts Skill Formation" is a research paper by Anthropic that experiments with how AI coding assistance affects the acquisition of new skills.
Overview
Full Paper
This article is a set of notes taken while reading the paper. At the end, there are also some notes on Trio, which appears in the paper.
Paper Notes
Experimental Method
The study focuses on Trio, an asynchronous library for Python that is less widely adopted than asyncio, and conducts the following experiments:
Overview of learning tasks and comprehension checks, created based on Figure 4 and 4.3 Study Design in the paper.
The experiment involved 52 participants and consisted of the following steps:
- Perform warm-up tasks to get accustomed to the coding platform in the experimental environment.
- Perform two coding tasks using Trio:
- Task 1: Tasks related to nurseries, how to start tasks, and concurrent execution.
- Task 2: Tasks related to error handling and memory channels.
- Participants were divided into two groups: one group used an AI assistant to complete the tasks, and the other did not.
- For details on how the groups were assigned, see Table 1: Balance table of main study participants.
- Perform a knowledge assessment related to Trio.
- Complete a questionnaire.
Results
Please refer to Figure 6 in the paper for a comparison of task completion time and accuracy on the knowledge assessment, which are the experimental results.
All participants in the group that used the AI assistant completed both tasks within the 35-minute time limit. In the group that did not use the AI assistant, 4 out of 26 participants were unable to complete the tasks within the time limit.
Regarding task completion time, the group that used the AI assistant tended to be slightly faster. However, no statistically significant speedup was observed (p = 0.391).
Next, comparing the accuracy of the knowledge assessment, it was observed that the group using the AI assistant had significantly lower performance (p = 0.01, Cohen’s d ≈ 0.74).
Looking at the task completion time and knowledge assessment accuracy of the group that used AI, they were further divided into several patterns based on their AI usage habits.
The following figure analyzes the relationship between task completion time and knowledge assessment accuracy, as well as how they used the AI assistant.
Created from Figure 11 and 6 Qualitative Analysis.
- Low-performance patterns
- AI Delegation (n = 4)
- Participants in this group relied entirely on AI for code generation and task completion. This group completed tasks the fastest and encountered few to no errors along the way.
- Progressive AI Reliance (n = 4)
- Participants in this group started with one or two questions but eventually relied on AI for all code creation. This group had low quiz scores, especially due to their failure to master the concepts of the second task.
- Iterative AI Debugging (n = 4)
- Participants in this group relied on AI to debug and verify their own code. While they queried the AI frequently, they left problem-solving to the AI rather than deepening their own understanding. As a result, their quiz scores were low, and they were relatively slow in completing the two tasks.
- AI Delegation (n = 4)
- High-performance patterns
- Generation-Then-Comprehension (n = 2)
- Participants in this group first generated code, then manually copied or pasted it. After code generation, they asked follow-up questions to the AI to deepen their understanding. While visually similar to the AI Delegation group, the difference is that they used the AI to verify their own comprehension. This group was not particularly fast, but demonstrated high understanding in the quiz.
- Hybrid Code-Explanation (n = 3)
- Participants in this group used a hybrid query approach, asking for code generation while simultaneously requesting explanations. They spent time reading and understanding the requested explanations.
- Conceptual Inquiry (n = 7)
- Participants in this group only asked conceptual questions and completed the tasks based on the understanding they gained. This group encountered many errors but resolved them on their own. On average, this pattern was the fastest among the high-scoring group and the second-fastest completion speed overall after AI Delegation.
- Generation-Then-Comprehension (n = 2)
Additionally, each knowledge assessment question was classified into one of the two tasks (Task 1 or Task 2) and one of the question types (conceptual, debugging, or code reading). Figure 8 in the paper shows the breakdown of scores by question type. Looking at this, debugging questions show the largest difference in average quiz scores between the treatment and control groups. It is speculated that the lower incidence of Trio-related errors among the group using the AI assistant led to this difference in debugging-related understanding.
Based on these results, the paper provides the following discussion:
Overall, the findings of this study suggest that the proactive introduction of AI into the workplace without maintaining cognitive engagement may have a negative impact on the professional development of workers. Under time constraints or organizational pressure, junior developers and other professionals may rely on AI at the expense of substantive skill development.
Furthermore, the largest differences in test scores were seen in debugging questions. This suggests that as companies advance AI-based code generation and transition to a system where humans act as supervisors, if skill formation is hindered by AI usage, humans may not possess sufficient skills to verify and debug AI-generated code.
Reflections
Some readers might interpret the abstract as suggesting that using AI assistants hinders the acquisition of new knowledge. However, the critical takeaway here is likely that if you rely on an AI assistant solely to process tasks, it becomes difficult for you, as the human, to gain actual knowledge.
I suspect this isn't an AI-specific problem. It was common practice in the past to search on Google, find some code you didn't quite understand, paste it in to finish a task, and then find yourself unable to debug it. The issue of skill acquisition being neglected due to the pressures of daily work is something that has always existed.
Also, personally, I felt that learning the concepts and usage of Trio within just 35 minutes would be quite difficult.
Trio Notes
Trio appears to be a library that aims for more intuitive asynchronous processing compared to asyncio.
Below is a simple example of asynchronous execution.
import trio
async def async_func(x):
print('start .... async_func', x)
await trio.sleep(x)
print('end .... async_func')
print('start .... main')
trio.run(async_func, 1)
print('end .... main')
Implementation for cancellation via timeout looks like the following. You can observe a task that would take 3 seconds being canceled after 2 seconds.
# https://trio.readthedocs.io/en/stable/reference-core.html#cancellation-and-timeouts
import trio
async def async_func(x):
print('start .... async_func', x)
await trio.sleep(x)
print('end .... async_func')
async def main():
print('start .... main')
with trio.move_on_after(2):
await async_func(3) # Returns before async_func completes
print('end .... main')
if __name__ == "__main__":
trio.run(main)
If you use fail_after instead of move_on_after, you can raise an exception upon cancellation.
To run multiple asynchronous processes, use open_nursery as shown below.
import trio
async def async_func(x):
print('start .... async_func', x)
await trio.sleep(x)
print('end .... async_func', x)
async def parent():
print("parent: started!")
async with trio.open_nursery() as nursery:
print("parent: spawning child1...")
nursery.start_soon(async_func, 1)
print("parent: spawning child2...")
nursery.start_soon(async_func, 2)
print("parent: waiting for children to finish...")
# -- we exit the nursery block here --
print("parent: all done!")
print('start .... main')
trio.run(parent)
print('end .... main')
To receive messages between asynchronous processes, use channels.
The following example demonstrates using channels to pass messages from producer to worker to consumer.
import trio
async def producer(work_send: trio.MemorySendChannel[int], n: int) -> None:
async with work_send:
for i in range(n):
print(f"[producer] sending work {i}")
await work_send.send(i)
print(f"[producer] sent work {i}")
async def worker(
work_recv: trio.MemoryReceiveChannel[int],
result_send: trio.MemorySendChannel[str],
) -> None:
async with work_recv, result_send:
async for x in work_recv:
# Pretend we are doing some processing
print(f" [worker] start {x}")
await trio.sleep(0.1)
msg = f"work {x} -> result {x * x}"
print(f" [worker] sending {x}")
await result_send.send(msg)
print(f" [worker] processed {x}")
async def consumer(result_recv: trio.MemoryReceiveChannel[str]) -> None:
async with result_recv:
async for msg in result_recv:
print(f" [consumer] got start: {msg}")
await trio.sleep(0.1)
print(f" [consumer] got end: {msg}")
async def main() -> None:
# Work channel (producer -> worker)
work_send, work_recv = trio.open_memory_channel(0) # 0 is a synchronous channel (no buffer)
# Result channel (worker -> consumer)
result_send, result_recv = trio.open_memory_channel(2) # This one has a small buffer
async with trio.open_nursery() as nursery:
nursery.start_soon(producer, work_send, 5)
nursery.start_soon(worker, work_recv, result_send)
nursery.start_soon(consumer, result_recv)
# producer closes work_send -> worker's async for ends -> worker closes result_send
# -> consumer's async for also ends and all tasks complete naturally
if __name__ == "__main__":
trio.run(main)
Discussion