Use Personalized AI Agents to Speed Up Software Development & Improve Code Quality [Testμ 2024]

AI’s Growing Impact on Software Development

Erhan opened the session by highlighting the significant shift in software development, where AI is beginning to play a pivotal role. He illustrated this by demonstrating how, in the near future, developers might simply point to a JIRA ticket in their IDE, and AI would autonomously generate a fully functional application like a trading app—based on those requirements. While we’re not entirely there yet, we’re moving toward a reality where AI assists with an increasing number of tasks in the software development life cycle (SDLC).

He emphasized that AI is already delivering substantial productivity gains, ranging from 20% to 50%, by supporting various stages of the SDLC. Currently, human engineers drive the process, with AI acting as an assistant across different steps such as coding, testing, code review, and deployment. However, the future holds a more streamlined process involving three key phases: ideation, generation, and validation.

In this evolving landscape, the human engineer collaborates with AI during the ideation phase, defining designs and requirements. The generation phase sees AI taking over the code creation entirely, leaving humans to focus on refining tasks. The final phase validation becomes crucial as the bottleneck in ensuring software quality. As AI-generated code scales rapidly, so must the validation tools and processes, making the role of quality assurance more vital than ever.

Delegating Tasks to AI: The Fundamental Theorem of Generativity

Continuing from his initial discussion, Erhan delved deeper into the concept of delegating tasks to AI within the SDLC, framing it as a “delegation problem.” He introduced what he called the “Fundamental Theorem of Generativity.” This theorem suggests that it only makes sense to delegate a task to an AI agent if the combined cost of instructing the agent and verifying its output is significantly lower than the cost of doing the task manually.

Erhan explained that for AI to be truly useful in the SDLC, two critical aspects need to be addressed:

• Efficient Communication: The process of telling the AI agent what you want and how you want it must be straightforward and time-efficient. If it takes too long to communicate these instructions, the benefits of using AI diminish.
• Trust in Results: Once the AI completes the task, its output must be reliable in terms of security, compliance, and performance. If there’s any doubt about the quality or accuracy of the AI’s work, developers may prefer to handle the task manually.

However, AI test agents like KaneAI by TestMu AI addresses these challenges head-on by simplifying the creation, debugging, and management of tests through natural language commands.

KaneAI is a GenAI smart test agent, guiding teams through the entire testing lifecycle. From creating and debugging tests to managing complex workflows, KaneAI uses natural language to simplify the process. It enables faster, more intuitive test automation, allowing teams to focus on delivering high-quality software with reduced manual effort.

With the rise of AI in testing, its crucial to stay competitive by upskilling or polishing your skillsets. The KaneAI Certification proves your hands-on AI testing skills and positions you as a future-ready, high-value QA professional.

Building Trust Through Human-AI Collaboration

Erhan expanded on the concept of trust in AI-assisted software development by emphasizing the importance of “tight loop integration” between the human engineer and the AI assistant.

He described this collaboration as a continuous feedback loop where the human provides lightweight specifications, and the AI generates code based on those inputs. The human engineer then reviews the generated code, applying their judgment to decide whether to accept, modify, or reject it. He highlighted two key points on Human-AI collaboration:

• Lightweight Specification: Erhan explained that the specification process in AI-assisted development is often minimal. It can involve simple actions like typing a small comment, code completion as you type, or brief instructions in a chat, which the AI then uses to generate a portion of the code.
• Human Review: Once the AI generates the code, typically in small chunks (e.g., 50-200 lines), the human engineer reviews it. This review process is crucial as it involves applying common sense and technical judgment to determine whether the generated code is suitable for integration into the project.

Erhan provided a practical example to illustrate this process. Suppose a developer writes a function signature, such as get_tweet. The AI might offer two possible completions: one using a well-known library like Tweepy, and another performing an HTTP request directly using the Requests library. The human engineer then evaluates these options, considering factors like the reliability and quality of the libraries involved, and makes a decision based on their judgment.

Enhancing Code with AI—Human Judgment Remains Key

Erhan continued by discussing scenarios where a developer might ask the AI to improve or refactor existing code. For instance, a simple function that converts temperatures from Fahrenheit to Celsius could be refactored by the AI to include a common conversion function, making the code more concise and maintainable.

For instance, Erhan presented the example of an AI-generated refractor where he replaced repetitive code with a common function. However, he acknowledged that in many cases, the decision to accept AI-generated refactoring may come down to personal preference or specific project requirements.

Erhan emphasized that while AI can significantly assist in enhancing code, the final judgment always rests with the human developer, ensuring that the code meets the project’s standards and requirements. This balance between AI assistance and human oversight is essential for maintaining trust in the development process.

The Role of Generation and Validation Agents in Enhancing AI Efficiency

In discussing the future of AI in software development, Erhan emphasized a crucial separation within the system: the roles of generation and validation agents.

Here’s a closer look at these components:

• Generation Agents: These agents are responsible for creating code or unit tests. They automate the production of new code, handling tasks like coding and generating tests based on given specifications.
• Validation Agents: Their role is to review and validate the output from generation agents. They also ensure the generated code meets standards for security, compliance, performance, and quality.
• The Context Engine: The context engine gathers data from various sources to provide a comprehensive understanding of the organization. It offers visibility similar to that of a human engineer, utilizing historical support tickets, stack traces, and best practices to inform the validation process.
• Control Layer: This layer allows users to set parameters and guidelines for the AI agents. It further defines best practices, standards, and policies for what the validation agents should check, ensuring that the AI operates within defined boundaries and maintains quality.

As AI-generated code scales, so must the validation processes. This balance between generation and validation is critical for maintaining high standards and efficiency in software development.

Building the Ideal AI Engineer: Erhan’s Wishlist

In his talk, Erhan laid out what he envisions as the ideal AI engineer.

Here’s a breakdown of the key qualities he hopes for when hiring an AI agent:

• Personalized Onboarding: The AI engineer should be tailored to fit seamlessly into the organization. Without proper onboarding, the AI won’t reach its full potential.
• Concise Communication: Effective interaction with the AI should be straightforward. Erhan wants the AI to communicate clearly and concisely, avoiding lengthy explanations or results.
• Inquisitive Nature: The AI should ask for clarification when needed. For instance, if given a task like implementing a calculator, it should seek further details if the request is ambiguous (e.g., whether a scientific or standard calculator is needed).
• Reliability: Trustworthiness is crucial. The AI engineer must consistently deliver dependable results and perform reliably in various tasks.

Erhan’s vision highlights the importance of a personalized, efficient, and reliable AI engineer who can communicate effectively and seek clarification when necessary.

The Four Pillars of Personalizing AI Engineers

Moving on with the session, Erhan highlighted the importance of personalizing AI engineers to ensure they effectively integrate and perform within an organization.

Here’s a closer look at the four key aspects of personalization:

• Context Awareness: Personalization begins with the AI engineer’s ability to understand and leverage existing work. This includes being familiar with everything on the local machine, such as existing code and documentation. The AI engineer needs to grasp both the problem domain it is working within and the solutions it is meant to provide.
• Connection: The AI engineer should connect with various sources of information within the organization. This means having access to organizational repositories, JIRA tickets, and other relevant resources. By connecting to these sources, the AI gains better context and visibility, which helps it perform its tasks more effectively.
• Coaching: Coaching involves setting explicit guidelines and instructions for the AI. Quality leads and test engineers can provide specific directives on coding standards and compliance. This ensures that the code generated adheres to the required guidelines and maintains high quality across the system.
• Customization: Customization allows for the adjustment and fine-tuning of AI models using the organization’s entire codebase as training data. This level of personalization ensures that the AI engineer is tailored to meet the specific needs and preferences of the organization, enhancing its effectiveness and reliability.

Best Practices for Enhancing Code Quality

Erhan discussed how to effectively use AI agents to enforce coding standards and best practices:

• Coding Rules in Natural Language: Users can input coding guidelines in natural language, such as “Don’t connect to the database without using the appropriate connector.” This enables AI agents to understand and enforce specific do’s and don’ts.
• Golden Repository of Code: Providing a golden repository of code and expert solutions allows the AI to prioritize these best practices. This repository can include code snippets and examples of high-quality solutions.
• Integration and Flagging: The AI agent integrates seamlessly with source control systems. It flags deviations in pull requests, merge requests, and command line integrations. This helps ensure that the code aligns with established rules and practices.
• Custom Rules and Enforcement: Users can create custom rules in natural language to enforce organizational standards. For instance, you can set rules to use standard or internal libraries and prevent the use of outdated APIs. This customization helps maintain consistency and prevents new legacy issues.
• Preventing New Legacy Code: By guiding developers away from old libraries and APIs, the AI helps avoid the creation of legacy code, ensuring that new developments adhere to current best practices.

In a Nutshell…

Erhan’s session highlighted key aspects of leveraging AI agents to enhance code quality. He stressed the importance of context, noting that AI performance improves with rich contextual information from code bases and organizational tools like JIRA. The onboarding process involves careful selection of relevant systems to ensure effective operation and avoid legacy code.

The discussion also touched on balancing reliability, accuracy, and test coverage, with current efforts focusing on improving coverage. Overall, Erhan’s insights offer a path to using AI agents for better coding standards and higher-quality software development.

Time for Some Q&A

• What are the two potential risks of relying on AI agents for code generation?

  Erhan: Code generation with AI agents carries certain risks. Firstly, the quality of the generated code depends significantly on human judgment. If developers accept code without careful review, it can compromise the overall code quality. Secondly, AI agents might produce duplicate or similar pieces of code, leading to technical debt.

This redundancy can create refactoring challenges and clutter the codebase. While efforts are underway to mitigate these issues, it’s essential for users to be aware of these risks and apply diligent oversight when integrating AI into their development workflows.

• Can AI agents assist in automating complex task scenarios or edge cases that are difficult to cover with traditional testing methods?

  Erhan: AI agents excel at generating test data and can significantly boost test coverage quickly. They are particularly effective in taking coverage from a low to a high level by creating diverse and interesting test cases that might not be immediately obvious to humans. For instance, an AI agent might add tests for scenarios like a future date of birth, which a human might overlook.

However, these agents may not be as effective in addressing the final, complex edge cases that require nuanced human reasoning. Their strength lies in rapidly enhancing coverage rather than fine-tuning the last details of test scenarios.

• How can I test agents such as validate the correctness of their generated tests?

  Erhan: One challenge with generating tests from existing source code is that the AI agent often has to infer specifications rather than work from clear, high-level requirements. This can lead to gaps between what the code is expected to do and what the tests actually cover.

However, by integrating with tools like JIRA to access detailed specifications and descriptions, future AI agents will be better able to align tests with defined test scenarios, bridging this gap and ensuring that tests accurately reflect the intended specifications.

• How do you find a good balance between reliability, accuracy and state of the test?

  Erhan: Currently, the focus is on generating tests to improve coverage rather than optimizing test latency. The team is aware of the importance of considering latency but has not yet integrated this factor into the process. This is a valuable point for future consideration, and they are open to exploring how to address it.

Please don’t hesitate to ask questions or seek clarification within the TestMu AI Community.