Labs

This individual assignment is due March 3rd, 2026

Objective

This assignment examines how Generative AI is used in real pull request workflows — including when suggestions are accepted, adapted, or rejected.

You are not required to use GenAI in your project. However, you are expected to understand how it is used and evaluated in professional software development.

Lab Structure & Grading

This lab consists of a single graded component:

• Quiz (30 points) – A quiz designed to assess your understanding of how Generative AI is used in real pull request workflows, including when AI suggestions are adopted, adapted, rejected, or used for reasoning support.

The quiz questions are based on the lecture slides. The case studies below provide additional detail and context to support your understanding.

Introduction: Generative AI in Pull Request Workflows

This lab examines how Generative AI (e.g., ChatGPT) participates in real-world software engineering workflows — particularly within GitHub pull requests (PRs).

Rather than focusing on prompt techniques, this lab focuses on how AI suggestions are evaluated in practice:

• When they are adopted
• When they are modified
• When they are rejected
• When they influence reasoning without direct code integration

The material builds on empirical research analyzing AI-assisted pull requests in open-source repositories (EMSE 2025; PatchTrack study).

Our findings show that AI participation is not binary. Instead, integration outcomes are shaped by engineering judgment, architectural alignment, maintainability constraints, and project scope.

GenAI Integration Outcomes

We identify four primary outcomes when AI is used in pull requests:

• PA (Patch Applied) – AI-generated code is integrated (often partially).
• PN (Patch Not Applied) – AI suggestions are modified or rejected after review.
• NE (No Existing Patch) – AI provides conceptual or methodological guidance without direct code integration.
• CL (Closed) – The pull request is closed due to architectural, scope, or quality concerns, even if AI suggestions were technically valid.

Case Studies

The following case studies illustrate how AI is used across these outcomes.

As you read, consider:

• Who initiated AI use (author or reviewer)?
• Was the AI output integrated verbatim, adapted, rejected, or used only for reasoning?
• What engineering constraints influenced the final decision (architecture, maintainability, scope)?

• Did AI improve the quality of reasoning, even when its code was not merged?

Patch Applied (PA)

In Patch Applied (PA) cases, AI-generated code is merged into the repository.
However, structural preservation is often limited.

Empirical Observations (PA Cases)

• The median structural integration of AI-generated patches is 25%.
• AI-generated code is frequently adapted rather than merged verbatim.
• Even when merged, AI suggestions are typically refined to align with project standards and context.
• Integration reflects engineering judgment, not automatic acceptance.

The boxplot below shows the percentage of AI-generated lines preserved in merged pull requests.

Integration Levels:

1. Low (0–25%) – Minor structural preservation.
2. Moderate (25–75%) – Partial adaptation to project context.
3. High (75–100%) – Mostly integrated with minimal changes.

Example 1: Code Duplication and Refactoring

• PR link
• ChatGPT Link

A developer used ChatGPT during the Software maintenance task of code duplication and refactoring in the nostr-tools project. The developer aimed to resolve code repetition and refactor the code.

The pull request shows code changes to address duplication issues, improving code quality. Code duplication often leads to maintenance challenges, bugs, and reduced readability. By using ChatGPT, the developer sought guidance to efficiently refactor the code.

Developer Prompt:

I have some duplication in my TypeScript code. I want to resolve it by creating a discriminated union based on the keys and values of the interface. Is this possible?

GitHub file diff: View here

Example 2: Deployment Documentation

Links to PR and ChatGPT conversation:

• PR Link
• Note: The original ChatGPT conversation link has been deleted by the developer. However, all the necessary details and context from the conversation are provided here for your reference. You can still follow along with the example using the information already included.

Documentation plays a key role in ensuring effective communication and collaboration in software development. It can take various forms such as:

• Requirements Documentation: Specifications (functional & non-functional).
• Design Documentation: System architecture, data flow.
• Technical Documentation: Code, APIs, algorithms, and data.
• Testing & QA Documentation: Test plans and quality assurance.
• Deployment Documentation: Guidelines for software deployment.

In this case, the developer used ChatGPT to generate user-friendly, concise deployment documentation for the release.sh script.

Developer’s prompt:

Act as a developer advocate with 5 years of experience. Write quick documentation for this `release.sh` script. Use bullet points and short sentences. Add emojis where needed.

As a result, the CONTRIBUTION.md file was updated with documentation from ChatGPT to help new contributors get started.

Example 3: Configuring GitHub Actions Workflow

• PR link - Git Diff File
• ChatGPT Conversation

In Examples 1 and 2 above, the developer used the code generated by ChatGPT “as-is.” However, this example demonstrates another use case where the developer integrates only a portion of the code snippet suggested by ChatGPT. The prompt provided by the developer aligns with the Configuring GitHub Actions Workflow task, which falls under the Software Configuration Management (SCM) activity. SCM encompasses version control, configuration management, and process management. Configuring GitHub Actions workflow specifically relates to automating the building, testing, and deploying of code changes, which aligns with CI practices.

In this case, after reviewing the GitHub link and ChatGPT conversation, we can see that the developer is seeking guidance on setting up and configuring GitHub Actions workflows for the Faker.js library. The GitHub link shows changes made to the GitHub Actions workflow configuration file.

Developer Prompt 1:

Write a GitHub Action yml file that blocks the PR from merging when there is a label named "do NOT merge yet" or "s: on hold"

Developer Prompt 2:

How can I check "s: on hold" with "contains" using "github.event.pull_request.labels.*.name"

Below is the final code snippet generated by ChatGPT and the portion integrated into the GitHub pull request.

Patch Not Applied (PN)

Example 1: Adaptation and Tailored Solutions

This example shows how ChatGPT’s conceptual advice was adapted for customized solutions to fit unique project needs.

• PR link
• ChatGPT link

Conversation Summary

• Initial ChatGPT Suggestion: ChatGPT provided a regex for ULID: ^[0-9A-HJKMNP-TV-Z]{26}$ to match a 26-character string using base32 encoding.

• Reviewer’s Suggestion: The reviewer (@lindyhopchris) recommended using Laravel’s whereUlid() regex for consistency, aligning with how ULIDs are handled in the Laravel framework. The Laravel regex provided by the reviewer is

    [0-7][0-9a-hjkmnp-tv-zA-HJKMNP-TV-Z]{25}
  

  This regex specifies that the first character is a digit between 0-7, followed by 25 base32 characters.

• PR Author’s Acceptance: The author (@Ashk2a) acknowledged the feedback and updated the code (File MatchedIDs.php) to reflect the Laravel-style regex:

    public function ulid(): self
    {
        return $this->matchAs('[0-7][0-9a-hjkmnp-tv-zA-HJKMNP-TV-Z]{25}');
    }
  

Conclusion

This pull request demonstrates how the developer adapted ChatGPT’s suggested regex to meet the specific requirements of the project. The task falls under the Software Engineering activity of code customization, as the developer further modified the generated code to align with project standards and feedback. This aligns with the theme of adaptation to project needs, where conceptual advice from ChatGPT was tailored to ensure consistency with Laravel’s framework.

Example 2: Methodological Guidance

This theme focuses on how ChatGPT’s advice informs approaches or refined solutions, rather than being directly applied as patches. The emphasis is on broader strategies influenced by ChatGPT.

• PR link
• ChatGPT Link

The developer’s interaction with ChatGPT in this pull request demonstrates a progression from a complex to a simpler, more efficient solution. Here’s a breakdown:

Initial Code (Before ChatGPT’s Suggestion)

In the initial code, the developer uses System.Char.ConvertToUtf32 to convert a char to its Unicode (UTF-32) code point. While effective, this method is mainly suited for handling UTF-16 surrogate pairs and is unnecessary for basic char values. It’s an overly complex solution for ASCII or basic Unicode characters.

The reviewer (@pbiggar), after consulting ChatGPT, noted that casting the char to an int would achieve the same result in a simpler and more efficient way. ChatGPT suggested casting char to int, which directly returns the Unicode (or ASCII) code point for the character—perfect for most simple characters.

Refined Code (After ChatGPT’s Suggestion)

The refined code can be found in the file backend/src/BuiltinExecution/Libs/Char.fs.

In the updated version:

• Simplified Conversion: The developer replaced System.Char.ConvertToUtf32 with int c.[0], which is a simpler and direct way to get the integer (ASCII/Unicode) value of a character.
• Added Validation: Additional logic was included to handle edge cases:
  • Digit Check: The code checks if the character is a digit using System.Char.IsDigit(c[0]).
  • Range Validation: It verifies that charValue is within the [0, 256) range, ensuring the value is within the ASCII range.
  • Conditional Handling: If charValue is valid, it returns Dval.optionSome (DInt charValue), otherwise it returns Dval.optionNone.

Conclusion

The interaction in this pull request is a clear example of methodological guidance, where ChatGPT’s advice helps refine the developer’s approach instead of being directly applied as a patch. This refinement aligns with the Software Engineering task of program repair, as it involved simplifying and improving the initial code to address inefficiencies.

By following ChatGPT’s advice, the developer made the code more efficient and easier to maintain, demonstrating how methodological guidance can lead to better solutions in software engineering.

None Existing Patch (NE)

Example 1: Conceptual Guidance & Theoretical Advice

This theme emphasizes programming concepts, design principles, and optimization strategies, focusing on best practices for readability and maintainability without specific code implementations.

• PR link
• ChatGPT Link

Let’s break down the situation to understand how the developer changed their code based on the ChatGPT conversation:

Original Problem

The developer was using the variable name pricingFrequencyClicked, which was problematic because the name implied an action (a click) rather than a state (the selected frequency). This didn’t accurately reflect what the variable was being used for: to store the selected pricing frequency (e.g., ‘monthly’ or ‘yearly’).

Developer’s Implementation

The final updates in the PR can be viewed in the file: app/controllers/pay.js.

After the ChatGPT conversation, the developer renamed pricingFrequencyClicked to selectedPricingFrequency. This new name more accurately reflects that the variable holds the user’s selection for pricing frequency, improving both readability and maintainability of the code.

ChatGPT’s Advice

ChatGPT highlighted the issue with the variable name, explaining that pricingFrequencyClicked was not appropriate because it didn’t align with the variable’s purpose. It suggested using a name like selectedPricingFrequency to more accurately reflect that the variable holds a selection, not an action.

Conclusion

This interaction falls under the “Conceptual Guidance & Theoretical Advice” theme, where ChatGPT’s recommendations helped improve the clarity and maintainability of the code by suggesting better naming conventions. This aligns with the software engineering tasks of code review and code quality enhancement, as it involved refining naming practices to enhance readability and maintainability.

In this case, ChatGPT played a key role in informing the process of improving code structure and long-term maintainability, aligning with these software engineering tasks.

Example 2: Debugging and Optimization Strategies

This theme focuses on debugging methods, performance optimization, and refining algorithms, offering strategic insights into problem-solving approaches without providing specific code snippets.

• PR link
• ChatGPT Link

Let’s break down how the developer improved their code based on a conversation with ChatGPT:

Original Code

Initially, the developer was using pushBack, which appends an element to the end of a list. However, repeatedly using this operation can be inefficient, as it may require traversing the entire list, especially in larger datasets.

ChatGPT’s Advice

After consulting ChatGPT, the developer received a suggestion to optimize the list operation by switching to push (which likely adds elements to the front of the list) and then reversing the list at the end. This approach is generally more efficient because adding to the front of the list is an O(1) operation in many functional languages, whereas pushBack can be an O(n) operation, where n is the length of the list.

Updated Code

The optimized code can be found in this PR file: backend/testfiles/execution/stdlib/result.dark.

In this updated code:

• The developer switched to using push (adding to the front) and planned to reverse the list at the end.
• This new approach enhances performance, as adding to the front of the list is typically O(1), whereas using pushBack could result in O(n) complexity.

Conclusion

This interaction falls under the Debugging & Optimization Strategies theme, where ChatGPT provided advice to improve performance. Rather than directly offering a code solution, ChatGPT suggested a strategy—appending elements to the front of the list and reversing it later—to optimize performance. This aligns with the Software Engineering task of performance optimization, where the developer refined the algorithm based on ChatGPT’s advice, ultimately enhancing the efficiency of the code.

ChatGPT played a key role in guiding the developer toward a more efficient solution for handling list operations, demonstrating how strategic advice can lead to performance improvements in software engineering.

Closed (CL): Architectural Misalignment and Scope Constraints

This example shows how AI-assisted reasoning surfaced a deeper architectural issue, but the pull request was closed without merging.

• Repository/PR: gemini-hlsw/scheduler — PR #428
  • PR (conversation)
  • PR (files changed)
• Reviewer who initiated AI use: @sraaphorst
• PR author: @stroncod

What the PR changed (local fix)

One of the core changes replaced object-based membership checks with ID-based checks:

has_resources = all([resource in nc[night_idx].resources for resource in obs.required_resources()])
has_resources = all([resource.id in [r.id for r in nc[night_idx].resources] for resource in obs.required_resources()])

Motivation: membership checks were failing after pickling/unpickling Resource objects, so the PR worked around the issue by comparing IDs instead of objects.

How ChatGPT Was Used (Reviewer-Driven)

The reviewer linked two ChatGPT conversations directly inside the PR discussion:

How ChatGPT Was Used (Reviewer-Driven)

Link 1 — Performance / Correctness Check
Link 1

Goal:

• Confirm that [r.id for r in ...] is rebuilt repeatedly inside all(...).
• Suggest computing the collection of IDs once (e.g., using a set) to improve efficiency and clarity.

Link 2 — Architectural Root Cause
Link 2

Goal:

• Explain why pickling/unpickling breaks flyweight identity.
• Clarify how __contains__, __hash__, and __eq__ affect membership checks in frozenset.
• Propose preserving canonical Resource instances by routing unpickling through ResourceManager.get_resource(...) (e.g., via __reduce__ or __setstate__).

Why the PR Was Closed (Key CL Insight)

The reviewer identified a deeper architectural issue involving flyweight consistency and unpickling behavior.

Addressing this properly would require structural changes (e.g., customizing unpickling and adjusting ResourceManager usage), extending beyond the scope of the submitted PR.

Outcome:
The PR author chose to close the pull request rather than expand it into a broader architectural refactor.

Understanding ChatGPT Conversation Sharing

In several case studies examined in this lab, developers shared their ChatGPT conversations directly inside GitHub pull requests.

This practice supports:

• Transparency in AI-assisted development
• Clear reasoning during code review
• Traceability of AI-generated suggestions
• Collaborative discussion of AI output

You are not required to use Generative AI in your project.
However, you are expected to understand how AI conversations are documented and discussed in professional workflows.

How to Generate a Shareable ChatGPT Link

If you choose to use ChatGPT:

1. Open your conversation.
2. Click Share.
3. Generate a public link.
4. Copy the link.

Shared links anonymize your personal profile information.

How Developers Use Shared Links in GitHub

Developers may include AI conversation links in:

• Pull request descriptions
• Code review comments
• Issue discussions

This allows others to:

• Understand the reasoning behind changes
• Evaluate AI suggestions critically
• Discuss architectural implications

Examples from real pull requests are shown below: