Human Testing vs. Automated Testing | Magento Consulting

It’s possible that I’m paranoid. But one of the most frustrating things in the technology world is when something that was working perfectly yesterday… stops working today. But nobody notices for a day or two.

Automated testing scripts don’t always catch this. And it’s not ideal to rely on your customers to tell you that things aren’t working.

This inspired another daily process for us here at Customer Paradigm.

Read more below >>

Automation vs. Human Testing.

About a month ago on the Customer Paradigm website, the contact forms stopped working properly. (I was the one responsible. I did a minor security update, and didn’t test it out 100%).

To the end user, they filled it out, and it looked like it was working. They received a success page message.

But submissions weren’t being saved properly or sent to our team. Ouch.

(I did have one more low-level backup system in place that allowed us to recover this information.)

Here’s the main problem: Our automated systems didn’t detect this. It wasn’t until I noticed nothing had come through the site in 24 hours that I decided to test it out.

And then I saw that the alerts weren’t coming through:

Customer Paradigm Alert Email

I have an automated system that tells me if the site is down. Or if the processor on the Webserver is running at 90% capacity or more for longer than a minute, another system sends me an email. Or if the disk space is full. Or even if the name servers change, or a dozen other things we can automatically check.

Automation is wonderful. And that’s a big reason why Websites and technology are force multipliers.

But sometimes, what you really need is a human looking at the site to make sure everything is working properly.

Automatic Software Updates.

Automatic Software Updates
Don’t you hate it when your computer / iPhone / tablet / DVR / thermostat does an automatic update… but then something else breaks? Yes, this actually did happen to my Internet-connected thermostat.

As computers update themselves with security updates and new functionality, sometimes this can break existing systems. But sometimes it’s not clear what isn’t working 100%.

Our Solution: Human Testing.

Every morning, one of the members of our team runs through our Website with a checklist:

Does the website load properly? Has any content been changed (i.e. has the site been hacked or changed)?
How fast does the site load? (We do an external speed test)
Does the main slider work properly?
How does the site look in three different browsers + a mobile device? (Browsers update themselves all the time, and that can break things.)
Do 4-5 random links from the navigation work okay?
Does the contact form work properly? (We fill it out, and make sure that someone receives the alert message.)
Do the phone numbers on the site work? (We actually dial the numbers and make sure that the phone system is working. It’s also a computer system, too.)
Is the site set to restrict access to search engines? (We’ve seen this happen to sites where a setting is turned on by accident, and then the site is removed from all search engines.)
Does our Twitter feed load (or does it show an error message)?
Does the site search function work properly? (This is something that could be broken for days or weeks before someone notices.)

Five days a week, we run through a human testing script for our site. We’ve gotten it down to about 10-15 minutes to test everything, and document it.

Is this overkill? Maybe. But now I don’t have to worry about things not working properly on the site. It’s just one less thing that I have to worry about each day.

Our Website and phone system is a primary way that new and existing customers reach us. Without them, we might as well start communicating via manual typewriters and snail mail.

Want Your Site Tested?

Want your site tested every day or week? We can set up both automated and human-tested checklists to make sure that problems are identified quickly. Let me know…

Until next time,
Jeff FinkelsteinFounder, Customer Paradigm