So seeing who my top artist was last week is pretty cool and all, but I’ve noticed that the site overall lacks some really useful features.

1. First, I’d really like to see more realtime updating of top artists this week (instead of saying “this week” but showing last week’s numbers).

2. What about showing one artist’s weekly playcount over a 3 month time period? There’s just so much you can do with that much data, it’s astounding.
But really, all that data stuff is just kind of cool to look at, and it would be pretty difficult to implement. Last.fm would have to beef up their features with no direct monetization. Now you can argue that making the site better leads to more people using it more frequently which leads to more people buying music through the site, and I’ll agree. However, when you’re talking about development for web applications, there’s only so much you can do in the hopes of attracting new users. Sometimes you really need something that will directly help you make money.

Okay, Last.fm, are you listening? This is a doozie. It looks like to me that quite a bit of your revenue comes from affiliate links to Amazon and such, where people buy music they like and you get a cut. If that’s so, I’ve got a useful product feature for you that will boost the number of albums you sell.

It’s pretty simple, actually. You scan the top 100 artists of a user. Then you discover which albums the user owns from that artist, and which ones they don’t. Then you have an album suggestion page which shows the user which albums from their favorite artists they don’t own. I know I would really benefit from this, since it can be difficult to keep up with as many artists as there are these days.

So how about it? We get a useful feature, and you get an easy revenue stream. It’s not often you get something that works for both sides that well.

Documentation

It’s great to find bugs, but if you can’t or don’t explain them properly to the person responsible for fixing the bug, at best you’ll confuse/frustrate him while wasting his time, and at worst you could prevent the bug from being fixed at all. There are two key principles in documenting a bug. Specificity and perspicuity (or being clear).

Be Specific

When writing up a bug, be as detailed as possible. Where exactly did you find the issue, how did you get there, what did you do? These are all very important questions. The more information you give, the less time the person correcting the problem has to spend figuring out what you mean. Since we’re working on a website, give URLs, breadcrumb trails, and anything else that could help. Write out step by step how to reproduce the error, starting with the homepage for customers, and the administration home page for the back-end.

Be Clear

Perspicuous means easily understood or comprehended. In the case of bug documentation being clear with your instructions is crucial. No matter how detailed you are, if your documentation is filled with typos, awkward sentence structure, or confusing or ambiguous language, it helps no one. The harder your documentation is to understand, the more time the developer has to spend to figure it out before he can fix it. In many cases, the developer may have to ask you what you meant, wasting even more time. Take the time to get it right the first time so you can be efficient.

Verification

Once you’ve submitted a bug and the developer has (supposedly) corrected the issue, you may be notified and asked to verify that the bug is indeed corrected. If this is the case, please take time to go and make sure you cannot reproduce the bug you reported. Do not take it for granted that the bug is fixed. Also, do not try to reproduce the bug before you are notified it has been fixed. It may be that your bug was determined to be a low priority or a non-issue, which means it may be a long time before it’s looked, if at all.

Following these guidelines for documentation and verification can drastically streamline the process and increase productivity.

There are two primary methods for discovering bugs and. The first method is natural testing, and the second is systematic testing.

Be Natural

There are two perspectives we usually test from. The customer and the back end user. Both of these perspectives need to be tested extensively so we can be sure orders can be placed and all normal functions performed. So when you start testing, pick a perspective and test from it. Don’t meander around, aimlessly pushing buttons and links. While you may find bugs this way, it’s generally a waste of your time. We need to target the important processes and move down from there.

If you’re operating from the customer perspective, try to find a specific product that you want to look at, try to find the information you want, try to add to cart, checkout, etc. If you’re operating from the back end user perspective, try to do some of the same things but from the back end. Try crediting an order, etc. It’s imperative to focus on things that have to be done constantly so that we get the most important processes fixed and perfect first. After you’re sure a process is steady, move on to something less significant. The key is to operate how you’d operate if you weren’t testing, but using.

Systematic Testing

For more strenuous and detailed testing, the systematic approach is employed. When using this method, the tester will pick one function or a group of similar functions and make sure they work properly in every location across the site. This method can be tedious, but it is vital to finding rarely occurring bugs that might only affect one product or page. Systematic testing is important, but takes a lower priority than Natural testing.

Principles for Bug Discovery

Pay Attention

When you’re running processes, pay attention to what you’re doing. Try not to get into a rhythm but to test each component individually. Many times testers run a process and expect it to work because they’re in a rhythm and every similar process up to that point has worked. Don’t make this mistake. If you expect everything to be broken, you won’t miss anything. And you’ll be pleasantly surprised every time something works.

Think Stupid

Yes, think stupid. You may be very familiar with the system and feel fairly confident using it. In the case of testing from the back-end user perspective, this can be a strength. In the case of testing from the customer perspective, it’s a weakness. Since the site will be brand new (or newly tweaked), customers will not know how to use it. That means they’re not going to know how to do things you do. It also means they will not know not to do something you recognize as being “dumb.” Think less in terms of intelligence and more in terms of ignorance. When testing from the customer perspective try to behave as if you know nothing about the site.

Keep Track

As you’re testing various things, keep a mental track of what you’ve done up to that point. When/if something breaks, you’ll need to know how you did it so that you can reproduce it. The easiest way to accomplish this is to simply be aware of what you’re doing. Don’t try to multi-task while beta testing. Focus on it and pay attention to what you’re doing and what you’ve done. That way when something goes wrong, you will probably already know what caused it.

Reproduce

When you find an issue. Don’t just report it. Make sure you saw it right. And make sure you can reproduce it. Submitting an issue that isn’t really an issue is a waste of everyone’s time. Retrace your steps and find out exactly caused the error/malfunction to occur. Obviously, in the case of cosmetic issues, this won’t be as much of a factor. But in functional/system critical errors, it is absolutely essential. There could be many different reasons why something broke. Many times it is a combination of several things that causes bugs. So don’t assume. Narrow down the reason as much as possible by trying different methods to produce the same result. Chances are you won’t narrow it enough, but that’s okay. Whoever is making the correction will find out exactly why it’s breaking, but by narrowing it down, you make his job a lot easier so he can focus on fixing, not finding.

Reproducing the problem is key to the next step, documentation.

Beta Testing – For the purpose of this series, this will refer to testing a website to find bugs (see definition 2) and document them.

Bug – Anything that deviates from the standard or normal function. It’s anything that is not how it’s supposed to be, not just something that generates an error.

Bug Categories

Problems with a website usually come in one of the following categories:

Cosmetic

This is an issue where something doesn’t look right. It could be a misplaced button, improper tab order, or typos. These are issues that won’t stop the site from being used properly but is a distraction. More serious cosmetic issues are ones that lead a customer/user to believe something about the site that is incorrect. For instance a free shipping offer for hard drives that’s actually supposed to be for memory, etc.

Functional

This is an issue where something doesn’t work. For instance, a button that when clicked, doesn’t do anything. Or a product that won’t add to the cart. Anything that is supposed to do something but doesn’t do it correctly. These can range from minor errors to very serious ones.

System Critical

These are considered to be the most serious issues. A system critical bug will shut the user/customer out of the site, freeze up the host computer, or break the site completely. These are usually rare on websites, but when they occur they are devastating.

That’ll do it for now. Our next installment will cover methods for bug discovery.

Even though I tested a software program, many of the same principles and guidelines apply to launching or relaunching a new site. I’ve seen lots of companies that do minimal testing on a website just to be overwhelmed as soon as the site goes live. So how do we find and fix these problems? Over the next few weeks I’ll be examining some basic guides and principles for web development and beta testing. Any feedback or ideas you might have would (as always) be welcome.

So with no further adieu, here’s the first installment.