Email checker tools gets yet more new features!

We’ve just added visual reports (including cross-tab tables and charts) to the email intelligence service (Email Checker Tools). Now you can view and download the results of your email validations in pretty, easy to understand web, PDF, Word, Excel and image formats.

This facility is an excellent way to speed up interpreting and downloading results and is provided at no extra charge to our customers.

Enjoy!

Our subscription and 3 day pass services currently rely on PayPal UK Website Payments Pro for payment and collection of fees both by PayPal and credit card. We selected PayPal primarily as it supported “major credit cards” inlcluding Amex (or so we thought). PayPal UK acceptance logos and other documentation show Amex is supported for payment.

Customers Can’t Pay With Amex?

We received an email from one of our customers complaining that they were unable to pay for our service with Amex. The payment screen (hosted by PayPal) gave no clue as to why their Amex credit card was being declined. As far as our customer was concerned, there was nothing wrong with the card.

Amex in the UK? The Real Truth.

We were puzzled by the problem and undertook to investigate and resolve the billing problem. After many hours on the phone to PayPal and other payment providers we found that the situation is fairly unsatisfactory for merchants operating out of the UK.

The Problem with Amex in the UK

On phoning PayPal to investigate the reasons why we were unable to setup recurring payment in Virtual Terminal, we were told that PayPal does not support Amex for native (i.e. Non PayPal accounts) because of a “dispute” between PayPal UK and Amex over the high fees being charged by Amex. We were told that Amex support would be coming “soon” but no commitment of dates could be made.

We also found that Google Checkout (UK) does not support payment by Amex to UK merchants, presumably for the same reasons (fee) as PayPal.

PayPal UK + Amex = Lost Opportunity

Learning that PayPal UK did not support Amex has been a shock to us. Had it not been for the fact that a customer had taken the time to complain, we would have continued to be blissfully unaware that there was a problem at all. We would have happily carried on with PayPal but how many lost opportunities would we have no knowledge of as customers go to checkout but then have to abort as their Amex card cannot be processed.

Having researched the PayPal UK /Amex problem extensively, we believe that PayPal is not making it clear enough to merchants that Amex is not supported. It is indeed the case that much of PayPal’s own material (e.g. implementation guides and acceptance logos) leads one to the conclusion that Amex is supported. For example, on the PayPal payment screen for credit cards, there is even a drop down option for Amex which then does not work (no helpful errors, just “card declined” – Nice!).

Our findings certainly raise some interesting questions for anyone considering using PayPal as a viable payment provider if you are based in the UK and having international customers wanting to pay by Amex.

UK Merchant? Your (Limited) Options Exposed

So, PayPal and Google are out of the running if you need to accept Amex.

Your options seem to be fairly limited. We found some possibilities:

• Up sticks and move our business to the US (Yes, we seriously considered this)
• Open a US subsidiary, open a US PayPal account
• Tell our customers that they can’t use Amex (an absurd idea & not even an option)
• Use RBS WorldPay (who have not been exactly speedy answering our enquiry)
• Use our bank

Ultimately, we opted to use the merchant services provided by our business bank (HSBC). There are many hoops to jump through in terms of compliance, integration work (just what we need;)) and more fiery hoops with Amex UK (who need a separate application process and yet more integration work). On top of this is the added expense (setup fees and operating costs) but what can we do?

One thing is for sure though, when we complete the HSBC integration work, we will seriously look at dropping PayPal as a payment provider on the basis of:

a) It doesn’t make financial sense to maintain 2 providers

b) We feel let down by PayPal UK merchant services. Why can’t PayPal just be transparent and honest? Up until today, we would have recommended PayPal as a decent payment provider without hesitation. How things can change so quickly huh?

It’s snowing (heavily) today in the UK. In the chaos of the travel problems resulting from snow & ice, we’ve managed to put in some huge effort to get the new Email Checker available for general release.

So, here it is. After a year+ of development, testing and refining we now have what is (based on our market research), the best email address checking service on the planet.

“Best?” Bold claims huh? Let’s back the claims up with some facts by looking at the “competition”:

Really “The Best”? The Competition Exposed

Desktop Software

“Desktop” software is software that you buy, download and install. We found at least 3 examples of this software for checking email addresses.

Our verdict:

• Too complicated – IT expertise needed to configure and integrate
• Highly inaccurate results

Bureaux Solutions

By “bureaux” we mean services that you manaully send your lists of email to. They do the validation work and send you the results.

Our verdict:

• Expensive! We found one company charging around $5 per thousand email addresses! Our service works out a fraction of this.

Behind The Scenes – What It Took to Get To Release

This project started based on the numerous requests that we were getting for bulk email checking capability. Conceptually, it should be easy to deliver bulk email address checking right? Yes, we thought so too all that time ago.

Way back at the beginning of the design, we thought: “just take our single email address checker and scale it up”:-no problem! How wrong we were. It has been anything but “simple”.

Along the way, we’ve had to deal with what seems like an endless stream of technical challenges ranging from getting it to run fast enough to achiveing the accuracy of results that we feel is acceptable.

For the technically minded, here are some of the cutting edge technologies that we’ve had to use to get to a viable service:

• SQL
• MSMQ
• Windows Communication Foundation (WCF)
• Windows Workflow Foundation (WF)
• SQL Broker & Notification Services
• PayPal Instant Payment Notification (IPN). We use our own IPN component to simplify things enormously (PayPal IPN IS complex).
• SQL Management Objects (SMO)

The service itself consists of 4 service programs that span multiple servers, 1 client (the web interface), and a whole host of external service such as fast DNS resolution (we use Google DNS).

To get the performance up (it takes 0.07 seconds on average to validate one email), we’ve had to write custom multi-threaded server software. Anyone who is in the software game will tell yout that this multithreading can be a complete nightmare!

On top of designing, building and testing this lot, there is the challenge of configuration – email checker relies heavily on external resources (e.g. email servers, DNS etc). It took many weeks of tweaking to get the right configuration settings (e.g. timeouts etc) to deliver maximum accuracy.

On top of this lot, there has been the dedicated web site (i.e. this website that you’re reading now:)) to design, build, test and populate with content.

Hopefully this gives some idea of the amount of work that’s gone into this project.

Although the amount of work far exceeded what we had expected and planned for, there is one driving principle that has not changed for us:-

We believe that our service offers the best value for money!

During all of the budget and timescale overruns on this project, we kept going because we firmly believe that our competitors do not offer either cost effective or fit-for-purpose solutions. It is therefore our sincere hope that our service can meet the demands and expectations of discerning customers.

Ultimately, we’re proud of this service and stand by our belief that it is “the best” option for anyone that has email lists to validate.

^*Average figures based on those email that can be checked remotely.

In the last 48 hours hours, we’ve been tinkering with DNS settings as well as working out what settings for time out settings deliver the best average results.

Time out settings determine how long our service waits for a remote mail server to return the answer to our query “does this email address exist?”. A time out of too long slows the overall validation process whereas a setting of too short runs the increased risk of missing results from mail servers that do not respond within the set time out.

Getting the best settings is a balancing act between performance and accuracy. With a lot of analysis, we arrived at some numbers that work well in in this application.

If you have used desktop based email checkers before, you would have configured your own time out settings. With our service, you don’t have to as we’ve done it for you. Yet another example of the benefits of doing email validation on a hosted platform rather than your desktop!

• Speed: Check
• Performance: Check
• Accuracy: Check

And so on. Every so often, we hit a check item and we get, what we call in the office, an “oh sh*t” moment! For example, this morning, we hit the “Check DNS” item. On checking our SPF records we found that they were screwed up big time and had been this way for a long time (probably “for ever”). Our SPF entries are (thankfully) fixed now which should make email address checking more accurate from our servers when we query external email servers that implement SPF checking.

Hindsight is a wonderful thing, so what did we learn from this? Two things:

• SPF wizards suck! In our case, the SPF wizard that we used a couple of years ago spat out the wrong text to paste into our DNS records. To fix the problem, we resorted to manually generating the records after having read the Wiki on SPF records. The idea behind SPF wizards is great (i.e. make it easy to create records). Unfortunately, the implementations of all the wizards that we tried were confusing (poorly worded), ambiguous and created duplicate entries for A:, MX, PTR and every other conceivable DNS record. Ultimately, we boiled the required SPF entry to a simple “v=spf1 a mx -all” and have found this to work well for all of our server infrastructure. “Wizards” came back with completely different results and, on testing, we found that none of them worked as we had expected.
• With DNS, don’t assume anything! In hindsight, we should have checked the original SPF records for validity. We didn’t and so have been running with no SPF records for over two years! Doh!

Having correct SPF records is essential if you operate services that need to cooperate with external email servers. For us, it’s very important that we get this right as it directly affects the results that we see when checking email addresses on servers that implement aggressive anti-spam policies.

Getting SPF records wrong is, at best inconvenient and may result in some of your emails not getting delivered. At it’s worst, incorrect SPF records can mean that most of your emails (or email address checks in our case) wont get through.

A lesson learnt for us.

We are in the business of email address validation. Email addresses and mail boxes exist on servers and use the services provided by these servers, services and The Internet in general.

Validating email addresses in real time is a tricky and complex operation. For each and every email address, our services have a “conversation” with the remote email server hosting the email address that’s being queried. Therefore, our service is to a large extent dependent on accurate information coming back from the remote email server. In most cases, things work great and we get back an accurate response as to whether an email address exists or not. However, there are a few instances where remote email servers just don’t play “nicely” with the net result that they refuse to cooperate, converse or simply provide incorrect information.

Ultimately, the accuracy of our email address checking services rely to a large extent on remote email servers being setup in a “standard” configuration that works with our services. Unfortunately, but in a relatively small number of cases, email servers are set up in non-standard configurations and can cause problems for an email address checking service such as ours. It’s not just our service that is affected by these non-standard email servers but every other email address checking solution on the market.

What this means in real terms is that you can expect accuracy of between 85-90% from our email address checking services compared to just 60-70% from our competitors products.

Scope

So, what’s the scope of the problem and how is it managed?

In our experience (based on historical data), we estimate that (approximately) between 10 and 15% of remote email servers have configurations that prevent accurate email address checking. We go into the technical details and reasons behind these issues in our Terms and Conditions.

This means that, for those 10% or so of email servers, it is simply not possible to validate email addresses. We manage this as follows:

• Providing an indication (where possible – i.e. know problem servers) of email addresses that cannot be validated by including a “flag” with the result returned from checking an email address. In our bulk email address checker service this “flag” is the “blocked/rejected” status. Please consult product documentation for details of the “flag” in other products / services.
• Providing clear policies and terms of operation/sale describing what you can expect from our services.

How This Affects Accuracy

Putting all this in perspective:- We estimate that, as a result of remote mail servers that do not cooperate with our service, you can expext a reduction of between 10 and 15% accuracy in email address checks. Looking at it from a different perspective, this means that, with our service, that you can expect between 85 and 90% accuracy when checking a typical list of email addresses.

Accuracy of between 85-90% is, as far as we have been able to determine in our market research, unprecedented. Other email address checker products will quote you accuracy figures somewhere in the region of 60-70%. In many cases, accuracy figures are not stated and it’s only after purchasing that customers of some of our competitors products find that accuracy is far less than anticipated.

Not 100%? What Good Is It Then?

…Is a question that we’ve had from time to time (usually from users of our free email address checker).

Let’s look at the benefits of email list cleansing. Ultimately, we are providing a service to reduce business operational costs by:

• Reducing email bounces from sending email (e.g. newsletters) to your prospects or clients.
• Increase the value (e.g. resale) value of your email address list. Fresh/”Validated” lists should command a higher resale fee.
• Reduce infrastructure costs such as network, storage, processing and backup associated with storing, processing and backing up old/”dead” email addresses.

We believe that, with our stated accuracy of 85-90%, your organization can still make significant time & cost savings. The cost savings to your organization should far outweigh the loss of 10-15% accuracy. It’s your decision whether your organization can tolerate the relatively small reduction in accuracy versus the potentially large gains in operational cost saving. For organizations that require nothing less than 100% accuracy, our service is not for you.

I Need 100% Accuracy!

We understand and we would love to be able to deliver 100%! However, the reality is that we live in a less than perfect Internet world where email servers can be offline, mis-configured or simply uncooperative. No process is able to handle all of these scenarios and, if anyone says (or implies that) 100% is achievable on any signifficant scale, they are not being entirely honest.

Alternatives to Automated Email Address Checking

Looking at the alternatives, checking of email addresses can rarely be 100%. Perhaps you would consider a “manual” process that might involve (for example): picking up the phone, emailing or sending customers questionnaires in the post. Even a manual process is subject to inaccuracy as clients don’t respond, provide fake email addresses or simply make a mistake in answering your question.

Policy Documentation

We treat our customers with respect and believe that solid business relationships are built not just on offering great products and services but also on trust and transparency.

To this end, we provide a comprehensive set of policies that describe what you can expect from our services as follows:

• Terms & Conditions of Sale
• Privacy Policy

It is our sincere wish that, through comprehensive and transparent policies, we are able to deliver products and services that meet and exceed your expectations.