blem reproducible at-will?

Does the problem occur with the current as well as the previous version of the software?

If the software runs on multiple operating systems, does the problem appear when the program is running on every OS, or only certain ones?

Does the program interact with third party software? If so, using SQL for database queries or native third-party clients, can you ensure that the other software isn’t at fault?

Has any software or hardware recently been installed on or removed from the system that exhibits the problem? If so, does the problem still exist if the system is restored to its previous state?

If the program runs across the network, does the problem occur when it is run locally on a single machine?

Does the problem occur when run in clustered and non-clustered Domino environments?

If the symptom appears client-side, does it occur in both the Notes client and web browser?

Does the problem only appear when the system is accessed via HTTP?

Is the problem load-related. That is, does it only occur when the system is under heavy load? If so, what appears to be the threshold?

* Note: These questions assume that the program is running in a supported environment. That is, the required operating system version and patches, third party software versions, minimum system requirements, etc are in place.

You may not be able to answer every one of these questions, but the more relevant information you supply, the better the developer will understand the scope of the problem.

Also, if the problem is reproducible, include a list of the steps required to recreate it. This will help the developer determine the cause and, when the time comes, it will help whoever has to determine whether or not the problem has been fixed.

Proper assignment For commercial software developers, database developers, IT professionals and those that test their programs, assigning a bug to the proper dev

IT professionals, software developers, support analysts, quality engineers, and their managers – virtually anyone who works professionally with software – will eventually, if not frequently, have to submit a defect or software problem report. When software acts unexpectedly, you’ll have to determine if the problem is your own or someone else's; if it is environmental, or user error; if it is even a problem at all. This ability to winnow out the root of a problem and report it accurately is a rare and valuable skill. Making sure that the cause is treated and not the symptom will save time, effort and money

Be sure it’s a bug

Is it functioning as-designed?

Software development is a relatively young and amorphous endeavor. As such, there are few standards to proscribe the way any given program should act. Developers build what they understand to be the correct solution to a particular problem, but what they build isn’t always what users of the program expect.

When users attempt to perform an operation, they expect a certain outcome. If the program doesn’t produce that outcome the person using it might naturally presume the program has a bug. While this is sometimes the case, the program may be doing what the developer designed it to do. In this case, the program is said to be ‘functioning as-designed’.

If you file a defect or software problem report (SPR) on a program that is functioning as-designed, you may receive a note from the developer describing the intricacies of the particular operation. Hopefully, you’re working with an articulate developer who has patience and a good sense of humor. If so, the information you get from him or her will be enlightening and useful. If your developer friend is having a bad day/week/year, you may get a less-than erudite explanation as to why your SPR is not worth the paper it’s written on.

In either case, it’s best to avoid writing SPR’s on features that are functioning as-designed. To that end, be as certain as you can be that you understand the functionality in-question. Read the documentation, talk to other users and use tech support to make sure that the program is designed to do what you’re expecting. You should even talk to the developer if you have access. A quick discussion can save a lot of emails, memos and erroneous SPR filings.

If you file an SPR that comes back as ‘functioning-as-designed’, you might want to have the documentation team make note of the behavior somewhere in the user manual. After all, if you – a seasoned software professional – thought it was a problem, the average user will probably run into it as well.

The difference between enhancements and bugs.

Enhancement requests are similar to bugs in that they are requests for a program to function differently than the way it currently does. If you thought you found a bug, but the program is actually functioning as designed, you may want to file an enhancement request. The process for doing this varies between companies, but these requests often go through the marketing department. One caveat, however; bugs usually get fixed before enhancement requests, so if you need a fix/feature as soon as possible, you should try to pursue it as a bug.

Is the problem environmental?

All software is sensitive to the environment in which it operates. If a program is certified to run under Windows NT and you’re running it on Windows 2000, the problem you’re seeing may be the result of using the wrong version of Windows. And it isn’t just the operating system; many factors comprise the environment. If your software is designed for use with Domino 6.0 and you’re running into problems under Domino 4.6, you might have environmental issues. If the program calls for LEI 6 and you have NotesPump 2.5, you’re asking for trouble. Other sources of environmental issues include duplicate copies of, incorrect versions of or completely missing runtime libraries (DLL’s in Windows); erroneous or missing environment variables; inadequate user or program security access; the system not meeting minimum requirements; network connectivity issues (not being able to access a network resource); incorrect versions of the JVM or JDK; etc.

An easy way to isolate a problem from its environment is to try to reproduce the error on a machine that meets the program’s requirements. If you don’t see the problem when running in the correct environment, bingo! You should certainly avoid filing software problem reports until you are sure the environment is not the cause.

Is it user error?

We are all fallible. And when it comes to using certain software, we’d have to be superhuman to never make any mistakes. But there’s nothing worse in the life a software professional than having an SPR rejected as ‘user error’. Some developers take a measure of glee in marking SPR’s this way. Maybe they’re happy not to have a real problem to fix, or maybe their motivation is more malevolent. But before you file an SPR, be absolutely certain that your expectations as to how the program should perform are in-line with the program’s intent. Otherwise, your SPR’s risk the possibility of receiving the dreaded ‘user error’ status. And if you get too many of these, developers will not take you seriously, and your SPR’s – even the valid ones – may wind up being ignored.

Pinpointing the problem

An accurate description.

I once worked with a QA manager who threatened severe bodily harm if the words ‘doesn’t work’ ever appeared in an SPR. Vagaries like ‘doesn’t look right’, ‘stopped working’ and ‘won’t run’ don’t actually describe a problem. They usually just frustrate someone along the line who is responsible for fixing the SPR or assigning it to the right team.

A better approach is to clearly define the exact nature of the problem. If a field in a screen was blank that should have contained a calculated value, you might be tempted to say something like ‘Bonus calculation didn’t work.’ While the statement may be true, it doesn’t describe the real cause and effect. Rather you should provide a detailed account of what was supposed to happen and what happened instead. Include the events leading up to the error, the action that triggered it, the expected result and the actual result, any error text the program displayed, anything that changed in your environment shortly before the problem occurred and any other details that will help the developer pinpoint the problem. A better description of the calculation problem might be:

‘After filling in all required fields of the Employee Bonus screen (date hired 1-13-2000, current salary 45,000 and bonus rating 4), I selected the Calculate Bonus option from the Actions menu. The Windows hourglass spun very briefly, less than one second, but the Bonus field remained blank. No error was displayed and the program behaved as if nothing was wrong.

This problem began after I updated to version 2.3 of the Salary module in our finance application. It appears when run from both the Notes client and a browser. Additionally, it only appears to happen if the employee is a manager.’

This description gives the developer all he or she needs to understand the nature of the problem.

The problem scenario.

Succinctly describing the process and circumstances that produced the problem is as important as identifying the problem itself. Simply put, you should include all relevant details and omit all irrelevant details. This isn’t always easy and winnowing the problem down to its least common denominators is a skill in itself. Some questions to answer that will help you define the problem domain:

1. Is the problem reproducible at-will?
2. Does the problem occur with the current as well as the previous version of the software?
3. If the software runs on multiple operating systems, does the problem appear when the program is running on every OS, or only certain ones?
4. Does the program interact with third party software? If so, using SQL for database queries or native third-party clients, can you ensure that the other software isn’t at fault?
5. Has any software or hardware recently been installed on or removed from the system that exhibits the problem? If so, does the problem still exist if the system is restored to its previous state?
6. If the program runs across the network, does the problem occur when it is run locally on a single machine?
7. Does the problem occur when run in clustered and non-clustered Domino environments?
8. If the symptom appears client-side, does it occur in both the Notes client and web browser?
9. Does the problem only appear when the system is accessed via HTTP?
10. Is the problem load-related. That is, does it only occur when the system is under heavy load? If so, what appears to be the threshold?

* Note: These questions assume that the program is running in a supported environment. That is, the required operating system version and patches, third party software versions, minimum system requirements, etc are in place.

You may not be able to answer every one of these questions, but the more relevant information you supply, the better the developer will understand the scope of the problem.

Also, if the problem is reproducible, include a list of the steps required to recreate it. This will help the developer determine the cause and, when the time comes, it will help whoever has to determine whether or not the problem has been fixed.

Proper assignment For commercial software developers, database developers, IT professionals and those that test their programs, assigning a bug to the proper deve

that are functioning as-designed. To that end, be as certain as you can be that you understand the functionality in-question. Read the documentation, talk to other users and use tech support to make sure that the program is designed to do what you’re expecting. You should even talk to the developer if you have access. A quick discussion can save a lot of emails, memos and erroneous SPR filings.

If you file an SPR that comes back as ‘functioning-as-designed’, you might want to have the documentation team make note of the behavior somewhere in the user manual. After all, if you – a seasoned software professional – thought it was a problem, the average user will probably run into it as well.

The difference between enhancements and bugs.

Enhancement requests are similar to bugs in that they are requests for a program to function differently than the way it currently does. If you thought you found a bug, but the program is actually functioning as designed, you may want to file an enhancement request. The process for doing this varies between companies, but these requests often go through the marketing department. One caveat, however; bugs usually get fixed before enhancement requests, so if you need a fix/feature as soon as possible, you should try to pursue it as a bug.

Is the problem environmental?

All software is sensitive to the environment in which it operates. If a program is certified to run under Windows NT and you’re running it on Windows 2000, the problem you’re seeing may be the result of using the wrong version of Windows. And it isn’t just the operating system; many factors comprise the environment. If your software is designed for use with Domino 6.0 and you’re running into problems under Domino 4.6, you might have environmental issues. If the program calls for LEI 6 and you have NotesPump 2.5, you’re asking for trouble. Other sources of environmental issues include duplicate copies of, incorrect versions of or completely missing runtime libraries (DLL’s in Windows); erroneous or missing environment variables; inadequate user or program security access; the system not meeting minimum requirements; network connectivity issues (not being able to access a network resource); incorrect versions of the JVM or JDK; etc.

An easy way to isolate a problem from its environment is to try to reproduce the error on a machine that meets the program’s requirements. If you don’t see the problem when running in the correct environment, bingo! You should certainly avoid filing software problem reports until you are sure the environment is not the cause.

Is it user error?

We are all fallible. And when it comes to using certain software, we’d have to be superhuman to never make any mistakes. But there’s nothing worse in the life a software professional than having an SPR rejected as ‘user error’. Some developers take a measure of glee in marking SPR’s this way. Maybe they’re happy not to have a real problem to fix, or maybe their motivation is more malevolent. But before you file an SPR, be absolutely certain that your expectations as to how the program should perform are in-line with the program’s intent. Otherwise, your SPR’s risk the possibility of receiving the dreaded ‘user error’ status. And if you get too many of these, developers will not take you seriously, and your SPR’s – even the valid ones – may wind up being ignored.

Pinpointing the problem

An accurate description.

I once worked with a QA manager who threatened severe bodily harm if the words ‘doesn’t work’ ever appeared in an SPR. Vagaries like ‘doesn’t look right’, ‘stopped working’ and ‘won’t run’ don’t actually describe a problem. They usually just frustrate someone along the line who is responsible for fixing the SPR or assigning it to the right team.

A better approach is to clearly define the exact nature of the problem. If a field in a screen was blank that should have contained a calculated value, you might be tempted to say something like ‘Bonus calculation didn’t work.’ While the statement may be true, it doesn’t describe the real cause and effect. Rather you should provide a detailed account of what was supposed to happen and what happened instead. Include the events leading up to the error, the action that triggered it, the expected result and the actual result, any error text the program displayed, anything that changed in your environment shortly before the problem occurred and any other details that will help the developer pinpoint the problem. A better description of the calculation problem might be:

‘After filling in all required fields of the Employee Bonus screen (date hired 1-13-2000, current salary 45,000 and bonus rating 4), I selected the Calculate Bonus option from the Actions menu. The Windows hourglass spun very briefly, less than one second, but the Bonus field remained blank. No error was displayed and the program behaved as if nothing was wrong.

This problem began after I updated to version 2.3 of the Salary module in our finance application. It appears when run from both the Notes client and a browser. Additionally, it only appears to happen if the employee is a manager.’

This description gives the developer all he or she needs to understand the nature of the problem.

The problem scenario.

Succinctly describing the process and circumstances that produced the problem is as important as identifying the problem itself. Simply put, you should include all relevant details and omit all irrelevant details. This isn’t always easy and winnowing the problem down to its least common denominators is a skill in itself. Some questions to answer that will help you define the problem domain:

1. Is the problem reproducible at-will?
2. Does the problem occur with the current as well as the previous version of the software?
3. If the software runs on multiple operating systems, does the problem appear when the program is running on every OS, or only certain ones?
4. Does the program interact with third party software? If so, using SQL for database queries or native third-party clients, can you ensure that the other software isn’t at fault?
5. Has any software or hardware recently been installed on or removed from the system that exhibits the problem? If so, does the problem still exist if the system is restored to its previous state?
6. If the program runs across the network, does the problem occur when it is run locally on a single machine?
7. Does the problem occur when run in clustered and non-clustered Domino environments?
8. If the symptom appears client-side, does it occur in both the Notes client and web browser?
9. Does the problem only appear when the system is accessed via HTTP?
10. Is the problem load-related. That is, does it only occur when the system is under heavy load? If so, what appears to be the threshold?

* Note: These questions assume that the program is running in a supported environment. That is, the required operating system version and patches, third party software versions, minimum system requirements, etc are in place.

You may not be able to answer every one of these questions, but the more relevant information you supply, the better the developer will understand the scope of the problem.

Also, if the problem is reproducible, include a list of the steps required to recreate it. This will help the developer determine the cause and, when the time comes, it will help whoever has to determine whether or not the problem has been fixed.

Proper assignment For commercial software developers, database developers, IT professionals and those that test their programs, assigning a bug to the proper dev

, incorrect versions of or completely missing runtime libraries (DLL’s in Windows); erroneous or missing environment variables; inadequate user or program security access; the system not meeting minimum requirements; network connectivity issues (not being able to access a network resource); incorrect versions of the JVM or JDK; etc.

An easy way to isolate a problem from its environment is to try to reproduce the error on a machine that meets the program’s requirements. If you don’t see the problem when running in the correct environment, bingo! You should certainly avoid filing software problem reports until you are sure the environment is not the cause.

Is it user error?

We are all fallible. And when it comes to using certain software, we’d have to be superhuman to never make any mistakes. But there’s nothing worse in the life a software professional than having an SPR rejected as ‘user error’. Some developers take a measure of glee in marking SPR’s this way. Maybe they’re happy not to have a real problem to fix, or maybe their motivation is more malevolent. But before you file an SPR, be absolutely certain that your expectations as to how the program should perform are in-line with the program’s intent. Otherwise, your SPR’s risk the possibility of receiving the dreaded ‘user error’ status. And if you get too many of these, developers will not take you seriously, and your SPR’s – even the valid ones – may wind up being ignored.

Pinpointing the problem

An accurate description.

I once worked with a QA manager who threatened severe bodily harm if the words ‘doesn’t work’ ever appeared in an SPR. Vagaries like ‘doesn’t look right’, ‘stopped working’ and ‘won’t run’ don’t actually describe a problem. They usually just frustrate someone along the line who is responsible for fixing the SPR or assigning it to the right team.

A better approach is to clearly define the exact nature of the problem. If a field in a screen was blank that should have contained a calculated value, you might be tempted to say something like ‘Bonus calculation didn’t work.’ While the statement may be true, it doesn’t describe the real cause and effect. Rather you should provide a detailed account of what was supposed to happen and what happened instead. Include the events leading up to the error, the action that triggered it, the expected result and the actual result, any error text the program displayed, anything that changed in your environment shortly before the problem occurred and any other details that will help the developer pinpoint the problem. A better description of the calculation problem might be:

‘After filling in all required fields of the Employee Bonus screen (date hired 1-13-2000, current salary 45,000 and bonus rating 4), I selected the Calculate Bonus option from the Actions menu. The Windows hourglass spun very briefly, less than one second, but the Bonus field remained blank. No error was displayed and the program behaved as if nothing was wrong.

This problem began after I updated to version 2.3 of the Salary module in our finance application. It appears when run from both the Notes client and a browser. Additionally, it only appears to happen if the employee is a manager.’

This description gives the developer all he or she needs to understand the nature of the problem.

The problem scenario.

Succinctly describing the process and circumstances that produced the problem is as important as identifying the problem itself. Simply put, you should include all relevant details and omit all irrelevant details. This isn’t always easy and winnowing the problem down to its least common denominators is a skill in itself. Some questions to answer that will help you define the problem domain:

1. Is the problem reproducible at-will?
2. Does the problem occur with the current as well as the previous version of the software?
3. If the software runs on multiple operating systems, does the problem appear when the program is running on every OS, or only certain ones?
4. Does the program interact with third party software? If so, using SQL for database queries or native third-party clients, can you ensure that the other software isn’t at fault?
5. Has any software or hardware recently been installed on or removed from the system that exhibits the problem? If so, does the problem still exist if the system is restored to its previous state?
6. If the program runs across the network, does the problem occur when it is run locally on a single machine?
7. Does the problem occur when run in clustered and non-clustered Domino environments?
8. If the symptom appears client-side, does it occur in both the Notes client and web browser?
9. Does the problem only appear when the system is accessed via HTTP?
10. Is the problem load-related. That is, does it only occur when the system is under heavy load? If so, what appears to be the threshold?

* Note: These questions assume that the program is running in a supported environment. That is, the required operating system version and patches, third party software versions, minimum system requirements, etc are in place.

You may not be able to answer every one of these questions, but the more relevant information you supply, the better the developer will understand the scope of the problem.

Also, if the problem is reproducible, include a list of the steps required to recreate it. This will help the developer determine the cause and, when the time comes, it will help whoever has to determine whether or not the problem has been fixed.

Proper assignment For commercial software developers, database developers, IT professionals and those that test their programs, assigning a bug to the proper dev

R or assigning it to the right team.

A better approach is to clearly define the exact nature of the problem. If a field in a screen was blank that should have contained a calculated value, you might be tempted to say something like ‘Bonus calculation didn’t work.’ While the statement may be true, it doesn’t describe the real cause and effect. Rather you should provide a detailed account of what was supposed to happen and what happened instead. Include the events leading up to the error, the action that triggered it, the expected result and the actual result, any error text the program displayed, anything that changed in your environment shortly before the problem occurred and any other details that will help the developer pinpoint the problem. A better description of the calculation problem might be:

‘After filling in all required fields of the Employee Bonus screen (date hired 1-13-2000, current salary 45,000 and bonus rating 4), I selected the Calculate Bonus option from the Actions menu. The Windows hourglass spun very briefly, less than one second, but the Bonus field remained blank. No error was displayed and the program behaved as if nothing was wrong.

This problem began after I updated to version 2.3 of the Salary module in our finance application. It appears when run from both the Notes client and a browser. Additionally, it only appears to happen if the employee is a manager.’

This description gives the developer all he or she needs to understand the nature of the problem.

The problem scenario.

Succinctly describing the process and circumstances that produced the problem is as important as identifying the problem itself. Simply put, you should include all relevant details and omit all irrelevant details. This isn’t always easy and winnowing the problem down to its least common denominators is a skill in itself. Some questions to answer that will help you define the problem domain:

1. Is the problem reproducible at-will?
2. Does the problem occur with the current as well as the previous version of the software?
3. If the software runs on multiple operating systems, does the problem appear when the program is running on every OS, or only certain ones?
4. Does the program interact with third party software? If so, using SQL for database queries or native third-party clients, can you ensure that the other software isn’t at fault?
5. Has any software or hardware recently been installed on or removed from the system that exhibits the problem? If so, does the problem still exist if the system is restored to its previous state?
6. If the program runs across the network, does the problem occur when it is run locally on a single machine?
7. Does the problem occur when run in clustered and non-clustered Domino environments?
8. If the symptom appears client-side, does it occur in both the Notes client and web browser?
9. Does the problem only appear when the system is accessed via HTTP?
10. Is the problem load-related. That is, does it only occur when the system is under heavy load? If so, what appears to be the threshold?

* Note: These questions assume that the program is running in a supported environment. That is, the required operating system version and patches, third party software versions, minimum system requirements, etc are in place.

You may not be able to answer every one of these questions, but the more relevant information you supply, the better the developer will understand the scope of the problem.

Also, if the problem is reproducible, include a list of the steps required to recreate it. This will help the developer determine the cause and, when the time comes, it will help whoever has to determine whether or not the problem has been fixed.

Proper assignment For commercial software developers, database developers, IT professionals and those that test their programs, assigning a bug to the proper dev

blem reproducible at-will?

Does the problem occur with the current as well as the previous version of the software?

If the software runs on multiple operating systems, does the problem appear when the program is running on every OS, or only certain ones?

Does the program interact with third party software? If so, using SQL for database queries or native third-party clients, can you ensure that the other software isn’t at fault?

Has any software or hardware recently been installed on or removed from the system that exhibits the problem? If so, does the problem still exist if the system is restored to its previous state?

If the program runs across the network, does the problem occur when it is run locally on a single machine?

Does the problem occur when run in clustered and non-clustered Domino environments?

If the symptom appears client-side, does it occur in both the Notes client and web browser?

Does the problem only appear when the system is accessed via HTTP?

Is the problem load-related. That is, does it only occur when the system is under heavy load? If so, what appears to be the threshold?

* Note: These questions assume that the program is running in a supported environment. That is, the required operating system version and patches, third party software versions, minimum system requirements, etc are in place.

You may not be able to answer every one of these questions, but the more relevant information you supply, the better the developer will understand the scope of the problem.

Also, if the problem is reproducible, include a list of the steps required to recreate it. This will help the developer determine the cause and, when the time comes, it will help whoever has to determine whether or not the problem has been fixed.

Proper assignment For commercial software developers, database developers, IT professionals and those that test their programs, assigning a bug to the proper developer or team for fixing is critical in getting it resolved quickly. If the SPR is assigned to the wrong team, it could sit unnoticed or ignored indefinitely. Additionally, loading up a developer’s queue with problems that aren’t his responsibility will almost certainly get you, at the very least, a not-so-nice email asking you to do your homework before assigning him any more defects.

Some development groups only allow certain individuals to assign SPR’s but, in most operations that I’ve worked with, the person who opens the bug report also assigns it to a developer who will be responsible for fixing it. If you have to assign a defect to an individual or team, do your best to pick the right one. If you don’t know, ask around. The developers will thank you.

Don’t overlook intermittent problems Running into a problem that isn’t readily reproducible can be extremely frustrating. Our inclination is to write these bugs off as our own mistakes or some other anomaly. Often times, however, they are valid and important bugs to fix. And because they are elusive by nature, keeping track of them becomes extra important. If you document one of these bugs in the form of a problem report, even if it doesn’t get fixed, the next person to run into it will see your SPR and should be able to build on it.

Before filing the report, be sure you’ve followed the proper procedures for installing the software, including the environment. And, as always, include all of the pertinent details. Also, make note of the fact that the problem is intermittent and not reproducible at-will. The developers may ask for more details and may want you to do more exploration. Keep track of all your findings in the problem report. If the bug remains unfixed for a while, you’ll save someone a lot of time recreating the conditions when the time comes to fix it.

Verifying the fix Once you’ve identified a problem, it’s important that you articulate the means by which you will verify its solution. If you omit this step, you or the person trying to determine if the bug has been fixed could have a hard time reaching a conclusion.

With some bugs, the verification will be obvious. In others, it will be require a complex and detailed series of steps that will confirm or refute that the problem has been solved. If you followed the directions outlined above when you filed the SPR, the groundwork should be laid for the verification work with the steps to reproduce. These steps are crucial to verifying that the developer has indeed fixed the bug. To verify the fix, simply follow the steps and see if the problem still occurs. If the steps to reproduce were omitted when the SPR was written, or they don’t have the appropriate level of detail, you’ll have to determine exactly how to test the fix and what evidence you’ll use to say whether or not the problem is solved.

Take, for example, the hypothetical bug that we described earlier in the article. When you get a new version of the program, you’d want to test that the bug has been corrected. You might be tempted to say that, if the bonus field calculates correctly when you select the action from the menu, the problem is solved. But there’s a problem with the word ‘correctly’. If the bonus field calculates at all, the original SPR has, in fact, been fixed. If the field calculates an incorrect amount, then the original fix uncovered a new problem, as often happens, which warrants its own SPR. By clearly and accurately defining the verification criteria, you’ll insure that your SPR’s cover a specific problem and only that problem.

Conclusion Problem reporting is a key component of the software development lifecycle. Accurate defect descriptions, complete scope definitions, correct assignment and appropriate verification all contribute to timely and complete problem resolutions. That means bugs are fixed quicker, so your software gets in users’ hands faster, and they’ll be happier with it.