How hard can it be to measure phishing?

Last Friday I went to a workshop organised by the Oxford Internet Institute on “Mapping and Measuring Cybercrime“. The attendees represented many disciplines from lawyers, through ePolicy, to serving police officers and an ex Government minister. Much of the discussion related to the difficulty of saying precisely what is or is not “cybercrime“, and what might be meant by mapping or measuring it.

The position paper I submitted (one more of the extensive Moore/Clayton canon on phishing) took a step back (though of course we intend to be a step forward), in that it looked at the very rich datasets that we have for phishing and asked whether this meant that we could usefully map or measure that particular criminal speciality?

In practice, we believe, bias in the data and the bias of those who are interpret it means that considerable care is needed to understand what all the data actually means. We give an example from our own work of how failing to understand the bias meant that we initially misunderstood the data, and how various intentional distortions arise because of the self-interest of those who collect the data.

Extrapolating, this all means that getting better data on other types of cybercrime may not prove to be quite as useful as might initially be thought.

As ever, reading the whole paper (it’s only 4 sides!) is highly recommended, but to give a flavour of the problem we’re drawing attention to:

If a phishing gang host their webpages on a thousand fraudulent domains, using fifty stolen credit cards to purchase them from a dozen registrars, and then transfer money out of a hundred customer accounts leading to a monetary loss in six cases: is that a 1000 crimes, or 50, or 12, or 100 or 6 ?

The phishing website removal companies would say that there were 1000 incidents because they need to get 1000 domains suspended. The credit card companies would say there were 50 incidents because 50 cardholders ought to have money reimbursed. Equally they would have 12 registrars to “charge back” because they had accepted fraudulent registrations (there might have been any number of actual credit card money transfer events between 12 and 1000 depending whether the domains were purchased in bulk). The banks will doubtless see the criminality as 100 unauthorised transfers of money out of their customer accounts; but if they claw back almost all of the cash (because it remains within the mainstream banking system) then the six-monthly Financial Fraud Action UK (formerly APACS) report will merely include the monetary losses from the 6 successful thefts.

Clearly, what you count depends on who you are — but crucially, in a world where resources are deployed to meet measurement targets (and your job is at risk if you miss them), deciding what to measure will bias your decisions on what you actually do and hence how effective you are at defeating the criminals.

5 thoughts on “How hard can it be to measure phishing?

  1. Great post and paper Richard. I’m curious why you think that measuring phishing activity, regardless of how measured, tells you anything.

    What is that number useful for, and what does it help you drive? It tells us a little about the total volume of these scams reaching end-users, but doesn’t really tell us anything about how much we should spend stopping it, what the total economic damage it, etc.

  2. @Andy
    Iโ€™m curious why you think that measuring phishing activity, regardless of how measured, tells you anything.

    If the measurement increases then you should be looking for more policing, better security mechanisms, more investment in back-office control systems.

    If the measurement decreases then you may be over-investing in preventative measures and looking for ways to make things simpler for customers, with less hoop-jumping when they want to do something unusual with their money.

    If phishing measurements decrease but bank losses continue to grow, then you have a key logging problem! and work is needed on countermeasures for that.

    It [..] doesnโ€™t really tell us anything about how much we should spend stopping it, what the total economic damage it, etc.

    The real danger for the banks is that people will lose confidence in online banking. That’s potentiall disastrous — they’d need to repurchase all the trendy wine bars and refill them with all the staff they’ve been downsizing. Serious money, whereas phishing losses are still down in the noise.

  3. Thanks for the reply. It isn’t clear to me though that there is a correlation between phishing mails delivered, and people visiting those sites and giving up their credentials. If we did have clear evidence of that then phishing mails delivered to the inbox is a reliable predictor of losses. Without that we’re still a little lost aren’t we? I haven’t seen any literature demonstrating the correlation above.

    Mind you, I think there probably is one, I’m just hoping for more evidence than intuition ๐Ÿ™‚

    And, totally agree on why this is an issue, and why tracking consumer confidence is critically important.

  4. From the ex-banker point of view, we actually used to count, for our own stats, the email waves – it was the easiest thing to do (and the way the attacks were usually detected). It also meant (very good for your metrics) that you could declare an attack “over” once you had crushed the sites, even though data stolen through them would be (and was ๐Ÿ™ ) usable for crimes in the future (which was sometimes irritating because it complicated the post theft analysis.)

    On crimes – I’d make that between 112 and 150 (I would count each fraudulent credit card payment and each unauthorised transfer as 1 x theft), but we never considered or relied on any numerical correlation (never mind equality) between “security incident” and actual criminality.

    BTW – can I take issue with your “now with impeccable grammar“? For anecdote, as opposed to evidence, may I submit the most recent phish from my spam bin?:

    Abbey National has changed the Online System Security in the last two days and your account was deactivated for security reasons. We suggest you check your balance
    as soon possible, Just in case your savings was affected
    please contact us:

  5. Sorry, I forgot the Computer Misuse Act s2 offences. Probably doubling (ish) the total there – but I have no idea how our police / CPS / Procurator Fiscal colleagues count those? Per access inappropriately gained (so on the order of three per transfer); per account illegally access (so 100); per (bank) website accessed; per bank etc?

    I would note that one of the many things that annoyed me about the take-down companies (you know who you are, b@stards!) was that they would count a site / URI as down (and to be paid for) as soon as they couldn’t access it (often because of connectivity issues as it was swamped by the duped – in the days before botnet hosting) and then count it again and again as it popped in and out of reach.

    Money lost, money moved, money at risk were all also measured and were important for understanding the actual risk the bank faced. But the bank’s metrics, important as they are for taking security decisions shouldn’t be the same as society’s metrics (crimes, in this case), which reflect a necessarily different view of what is important.

    On the point in your conclusion – it is “2 + multiple” crimes – 1 x TWOC and (1 + multiple) criminal damage. But don’t worry, he’ll get a conditional discharge or a community service order. Which, unfortunately, is more than the phishing fraud gang are likely to receive ๐Ÿ™

Leave a Reply

Your email address will not be published. Required fields are marked *