Category Archives: Cybercrime

Is Apple’s NeuralMatch searching for abuse, or for people?

Cybercrime, Internet censorship, Legal issues, News coverage, Operating systems, Privacy technology, Security economics, Security engineering, Security psychology

Apple stunned the tech industry on Thursday by announcing that the next version of iOS and macOS will contain a neural network to scan photos for sex abuse. Each photo will get an encrypted ‘safety voucher’ saying whether or not it’s suspect, and if more than about ten suspect photos are backed up to iCloud, then a clever cryptographic scheme will unlock the keys used to encrypt them. Apple staff or contractors can then look at the suspect photos and report them.

We’re told that the neural network was trained on 200,000 images of child sex abuse provided by the US National Center for Missing and Exploited Children. Neural networks are good at spotting images “similar” to those in their training set, and people unfamiliar with machine learning may assume that Apple’s network will recognise criminal acts. The police might even be happy if it recognises a sofa on which a number of acts took place. (You might be less happy, if you own a similar sofa.) Then again, it might learn to recognise naked children, and flag up a snap of your three-year-old child on the beach. So what the new software in your iPhone actually recognises is really important.

Now the neural network described in Apple’s documentation appears very similar to the networks used in face recognition (hat tip to Nicko van Someren for spotting this). So it seems a fair bet that the new software will recognise people whose faces appear in the abuse dataset on which it was trained.

So what will happen when someone’s iPhone flags ten pictures as suspect, and the Apple contractor who looks at them sees an adult with their clothes on? There’s a real chance that they’re either a criminal or a witness, so they’ll have to be reported to the police. In the case of a survivor who was victimised ten or twenty years ago, and whose pictures still circulate in the underground, this could mean traumatic secondary victimisation. It might even be their twin sibling, or a genuine false positive in the form of someone who just looks very much like them. What processes will Apple use to manage this? Not all US police forces are known for their sensitivity, particularly towards minority suspects.

But that’s just the beginning. Apple’s algorithm, NeuralMatch, stores a fingerprint of each image in its training set as a short string called a NeuralHash, so new pictures can easily be added to the list. Once the tech is built into your iPhone, your MacBook and your Apple Watch, and can scan billions of photos a day, there will be pressure to use it for other purposes. The other part of NCMEC’s mission is missing children. Can Apple resist demands to help find runaways? Could Tim Cook possibly be so cold-hearted as to refuse at add Madeleine McCann to the watch list?

After that, your guess is as good as mine. Depending on where you are, you might find your photos scanned for dissidents, religious leaders or the FBI’s most wanted. It also reminds me of the Rasterfahndung in 1970s Germany – the dragnet search of all digital data in the country for clues to the Baader-Meinhof gang. Only now it can be done at scale, and not just for the most serious crimes either.

Finally, there’s adversarial machine learning. Neural networks are fairly easy to fool in that an adversary can tweak images so they’re misclassified. Expect to see pictures of cats (and of Tim Cook) that get flagged as abuse, and gangs finding ways to get real abuse past the system. Apple’s new tech may end up being a distributed person-search machine, rather than a sex-abuse prevention machine.

Such a technology requires public scrutiny, and as the possession of child sex abuse images is a strict-liability offence, academics cannot work with them. While the crooks will dig out NeuralMatch from their devices and play with it, we cannot. It is possible in theory for Apple to get NeuralMatch to ignore faces; for example, it could blur all the faces in the training data, as Google does for photos in Street View. But they haven’t claimed they did that, and if they did, how could we check? Apple should therefore publish full details of NeuralMatch plus a set of NeuralHash values trained on a public dataset with which we can legally work. It also needs to explain how the system it deploys was tuned and tested; and how dragnet searches of people’s photo libraries will be restricted to those conducted by court order so that they are proportionate, necessary and in accordance with the law. If that cannot be done, the technology must be abandoned.

Cybercrime gangs as tech startups

Academic papers, Cybercrime, Security economics

In our latest paper, we propose a better way of analysing cybercrime.

Crime has been moving online, like everything else, for the past 25 years, and for the past decade or so it’s accounted for more than half of all property crimes in developed countries. Criminologists have tried to apply their traditional tools and methods to measure and understand it, yet even when these research teams include technologists, it always seems that there’s something missing. The people who phish your bank credentials are just not the same people who used to burgle your house. They have different backgrounds, different skills and different organisation.

We believe a missing factor is entrepreneurship. Cyber-crooks are running tech startups, and face the same problems as other tech entrepreneurs. There are preconditions that create the opportunity. There are barriers to entry to be overcome. There are pathways to scaling up, and bottlenecks that inhibit scaling. There are competitive factors, whether competing crooks or motivated defenders. And finally there may be saturation mechanisms that inhibit growth.

One difference with regular entrepreneurship is the lack of finance: a malware gang can’t raise VC to develop a cool new idea, or cash out by means on an IPO. They have to use their profits not just to pay themselves, but also to invest in new products and services. In effect, cybercrooks are trying to run a tech startup with the financial infrastructure of an ice-cream stall.

We have developed this framework from years of experience dealing with many types of cybercrime, and it appears to prove a useful way of analysing new scams, so we can spot those developments which, like ransomware, are capable of growing into a real problem.

Our paper Silicon Den: Cybercrime is Entrepreneurship will appear at WEIS on Monday.

Hiring for iCrime

Cybercrime, Jobs

We are hiring two Research Assistants/Associates to work on the ERC-funded Interdisciplinary Cybercrime Project (iCrime). We are looking to appoint one computer scientist and one social scientist to work in an interdisciplinary team reporting to Dr Alice Hutchings.

iCrime incorporates expertise from criminology and computer science to research cybercrime offenders, their crime type, the place (such as online black markets), and the response. We will map out the pathways of cybercrime offenders and the steps and skills required to successfully undertake complex forms of cybercrime. We will analyse the social dynamics and economies surrounding cybercrime markets and forums. We will use our findings to inform crime prevention initiatives and use experimental designs to evaluate their effects.

Within iCrime, we will develop tools to identify and measure criminal infrastructure at scale. We will use and develop unique datasets and design novel methodologies. This is particularly important as cybercrime changes dynamically. Overall, our approach will be evaluative, critical, and data driven.

If you’re a computer scientist, please follow the link at: https://www.jobs.cam.ac.uk/job/30100/

If you’re a social scientist, please follow the link at: https://www.jobs.cam.ac.uk/job/30099/

Please read the formal advertisements for the details about exactly who and what we’re looking for and how to apply — and please pay special attention to our request for a covering letter!

10/06/21 Edited to add new links

COVID-19 test provider websites and Cybersecurity: COVID briefing #22

Academic papers, Cybercrime, Web security

This week’s COVID briefing paper (COVIDbriefing-22.pdf) resumes the Cybercrime Centre’s COVID briefing series, which began in July 2020 with the aim of sharing short on-going updates on the impacts of the pandemic on cybercrime.

The reason for restarting this series is a recent personal experience while navigating through the government’s requirements on COVID-19 testing for international travel. I observed great variation in the quality of website design and cannot help but put on my academic hat to report on what I found.

The quality of some websites is so poor that it hard to distinguish them from fraudulent sites — that is they have many of the features and characteristics that consumers have been warned to pay attention to. Compounded with the requirement to provide personally identifiable information there is a risk that fraudulent sites will indeed spring up and it will be unsurprising if consumers are fooled.

The government needs to set out minimum standards for the websites of firms that they approve to provide COVID-19 testing — especially with the imminent growth in demand that will come as the UK’s travel rules are eased.

Cybercrime is (still) (often) boring

Academic papers, Cybercrime

Depictions of cybercrime often revolve around the figure of the lone ‘hacker’, a skilled artisan who builds their own tools and has a deep mastery of technical systems. However, much of the work involved is now in fact more akin to a deviant customer service or maintenance job. This means that exit from cybercrime communities is less often via the justice system, and far more likely to be a simple case of burnout.

Continue reading Cybercrime is (still) (often) boring →

Infrastructure – the Good, the Bad and the Ugly

Cybercrime, Internet censorship, Legal issues, News coverage, Politics, Security economics, Social networks

Infrastructure used to be regulated and boring; the phones just worked and water just came out of the tap. Software has changed all that, and the systems our society relies on are ever more complex and contested. We have seen Twitter silencing the US president, Amazon switching off Parler and the police closing down mobile phone networks used by crooks. The EU wants to force chat apps to include porn filters, India wants them to tell the government who messaged whom and when, and the US Department of Justice has launched antitrust cases against Google and Facebook.

Infrastructure – the Good, the Bad and the Ugly analyses the security economics of platforms and services. The existence of platforms such as the Internet and cloud services enabled startups like YouTube and Instagram soar to huge valuations almost overnight, with only a handful of staff. But criminals also build infrastructure, from botnets through malware-as-a-service. There’s also dual-use infrastructure, from Tor to bitcoins, with entangled legitimate and criminal applications. So crime can scale too. And even “respectable” infrastructure has disruptive uses. Social media enabled both Barack Obama and Donald Trump to outflank the political establishment and win power; they have also been used to foment communal violence in Asia. How are we to make sense of all this?

I argue that this is not simply a matter for antitrust lawyers, but that computer scientists also have some insights to offer, and the interaction between technical and social factors is critical. I suggest a number of principles to guide analysis. First, what actors or technical systems have the power to exclude? Such control points tend to be at least partially social, as social structures like networks of friends and followers have more inertia. Even where control points exist, enforcement often fails because defenders are organised in the wrong institutions, or otherwise fail to have the right incentives; many defenders, from payment systems to abuse teams, focus on process rather than outcomes.

There are implications for policy. The agencies often ask for back doors into systems, but these help intelligence more than interdiction. To really push back on crime and abuse, we will need institutional reform of regulators and other defenders. We may also want to complement our current law-enforcement strategy of decapitation – taking down key pieces of criminal infrastructure such as botnets and underground markets – with pressure on maintainability. It may make a real difference if we can push up offenders’ transaction costs, as online criminal enterprises rely more on agility than on on long-lived, critical, redundant platforms.

This was a Dertouzos Distinguished Lecture at MIT in March 2021.

Friendly neighbourhood cybercrime: online harm in the pandemic and the futures of cybercrime policing

Academic papers, Cybercrime, Politics

As cybercrime researchers we’re often focused on the globalised aspects of online harms – how the Internet connects people and services around the world, opening up opportunities for crime, risk, and harm on a global scale. However, as we argue in open access research published this week in the Journal of Criminal Psychology in collaboration between the Cambridge Cybercrime Centre (CCC), Edinburgh Napier University, the University of Edinburgh, and Abertay University, as we have seen an enormous rise in reported cybercrime in the pandemic, we have paradoxically seen this dominated by issues with a much more local character. Our paper sketches a past: of cybercrime in a turbulent 2020, and a future: of the roles which state law enforcement might play in tackling online harm a post-pandemic world.

Continue reading Friendly neighbourhood cybercrime: online harm in the pandemic and the futures of cybercrime policing →

WEIS 2020 – Liveblog

Authentication, Cybercrime, Legal issues, Politics, Security economics, Security psychology

I’ll be trying to liveblog the seventeenth Workshop on the Economics of Information Security (WEIS), which is being held online today and tomorrow (December 14/15) and streamed live on the CEPS channel on YouTube. The event was introduced by the general chair, Lorenzo Pupillo of CEPS, and the program chair Nicolas Christin of CMU. My summaries of the sessions will appear as followups to this post, and videos will be linked here in a few days.

Three paper Thursday: Online Extremism and Radicalisation

Cybercrime, Three Paper Thursday,

With the recent United States presidential election, I have chosen to focus the theme of this Three Paper Thursday on extremism and radicalisation. This topic has got increasing media attention during the past six years in the United States, through both a general rise in the public prominence of far-right, racist rhetoric in political culture (often attributed to the Trump presidency), and a series of high-profile violent events associated with far-right extremism. These events range from the riots in Charlottesville, Virginia (which turned violent when rally attendees clashed with counter-protesters and a vehicle drove into a crowd marching through downtown, killing one protester (Heim, Silverman, Shapiro, & Brown, 2017), to the recent arrest of individuals plotting a kidnap of the Governor of Michigan. This far-right violence brought to light the continued existence of right-wing extremism in the United States. This has historical roots in well-known organisations such as the Ku Klux Klan (KKK), a secretive, racist, terrorist organisation founded in 1865 during Reconstruction as part of a backlash against the acquisition of civil rights by African-American people in the South (Bowman-Grieve, 2009; Martin, 2006).

In contemporary online societies, the landscape and dynamics of right-wing extremist communities have changed. These communities have learned how to exploit the capacities of online social networks for recruitment, information sharing, and community building. The sophistication and reach of online platforms has evolved rapidly from the bulletin board system (BBS) to online forums and now social media platforms, which incorporate powerful technologies for marketing, targeting, and disseminating information. However, the use of these platforms for right-wing radicalisation (the process through which an individual develops and/or accepts extreme ideologies and beliefs) remains under-examined in academic scholarship. This Three Paper Thursday pulls together some key current literature on radicalisation in online contexts.

Maura Conway, Determining the role of the internet in violent extremism and terrorism: Six suggestions for progressing research. Studies in Conflict & Terrorism, 40(1), 77-98. https://www.tandfonline.com/doi/full/10.1080/1057610X.2016.1157408.

The first paper comments on future directions for research in understanding and determining the role of the Internet in violent extremism and terrorism. After guiding readers through an overview of current research, the author argues that there is a lack of both descriptive and explanatory work on the topic, as the field remains divided. Some view Internet as mere speech platforms and argue that participation in online radicalised communities is often the most extreme behaviour in which most individuals engage. Others acknowledge the affordances of the Internet but are uncertain in its role in replacing or strengthening other radicalisation processes. The author concludes that two major research questions remain to be answered: whether radicalisation can occur in a purely online context, and if so, does it contribute to violence? In that case, the mechanisms merit further exploration. The author makes six suggestions for future researchers: a) widening current research to include movements beyond jihadism, b) conducting comparison research (e.g., between platforms and/or organisations), c) studying individual users in extremist communities and groups, d) using large-scale datasets, e) adopting an interdisciplinary approach, and f) examining the role of gender.

Yi Ting Chua, Understanding radicalization process in online far-right extremist forums using social influence model. PhD thesis, Michigan State University, 2019. Available from https://d.lib.msu.edu/etd/48077.

My doctoral dissertation examines the impact of participation in online far-right extremist groups on radicalisation. In this research, I applied social network analysis and integrated theories from criminology (social learning theory) and political science (the idea of the echo chamber) to understand the process of attitudinal changes within social networks. It draws on a longitudinal database of threads saved from eight online far-right extremist forums. With the social influence model, which is a regression model with a network factor, I was able to include the number of interactions and attitudinal beliefs of user pairs when examining attitudinal changes across time. This model allows us to determine if, and how, active interactions result in expression of more radical ideological beliefs. Findings suggested that online radicalisation occurred at varying degrees in six of seven forums, with a general lowered level of expressed extremism towards the end of observed time period. The study found strong support the proposition that active interactions with forum members and connectedness are predictors of radicalisation, while suggesting that other mechanisms, such as self-radicalisation and users’ prior beliefs, were also important. This research highlighted the need for theory integration, detailed measures of online peer association, and cross-platform comparisons (i.e. Telegram and Gab) to address the complex phenomena of online radicalisation.

Magdalena Wojcieszak, ‘Don’t talk to me’: effects of ideologically homogeneous online groups and politically dissimilar offline ties on extremism. New Media & Society, 12(4) (2010) pp 637-655. https://journals.sagepub.com/doi/abs/10.1177/1461444809342775.

In this article, the author is interested in answering two questions: 1) does participation in ideologically homogeneous online groups increase extreme beliefs, and 2) how do offline strong and weak ties with dissimilar beliefs affect extreme beliefs? The author uses online survey data and posts from neo-Nazi online forums. The outcome is measured by respondents’ responses to 10 ideology-specific statements. Other variables in the analysis included the level of participation in online groups, perceived dissimilarity of offline ties, news media exposure and demographics. Findings from a multivariate regression model indicate that participation in online groups was a strong predictor of support for racial violence after controlling for demographic factors and news media exposure. Forum members’ attitudes are subjected to normative influences via punitive or rewarding replies. For individuals with politically dissimilar offline ties, the author finds a weakened participation effect.

Together, these papers highlight the complexity of assessing the role played by the Internet in the radicalisation process. The first paper encourages researchers to tackle whether online violent radicalisation occurs via six different approaches. The other two papers show support for online radicalisation while simultaneously calling attention to the effect of other variables, such as the influence of offline relationships and users’ baseline beliefs prior to online participation. All of these papers cross academic disciplines, highlighting the importance of an interdisciplinary perspective.

References

Bowman-Grieve, L. (2009). Exploring “Stormfront”: A virtual community of the radical right. Studies in Conflict & Terrorism, 32(11), 989-1007.

Heim, J., Silverman, E., Shapiro, T. R., Brown, E. (2017, August 13). One dead as car strikes crowds amid protests of white nationalist gathering in Charlottesville; two police die in helicopter crash. The Washington Post. Retrieved from https://www.washingtonpost.com/local/fights-in-advance-of-saturday-protest-in-charlottesville/2017/08/12/155fb636-7f13-11e7-83c7-5bd5460f0d7e_story.html?utm_term=.33b6686c7838.

Martin, G. (2006). Understanding Terrorism: Challenges, Perspectives, and Issues. Thousand Oaks, California: Sage Publications.

How an Illicit Cybercrime Market Evolves: A Longitudinal Study

Academic papers, Cybercrime, , ,

Online underground marketplaces are an essential part of the cybercrime economy. They often act as a cash-out market, enabling the trade in illicit goods and services between pseudonymous members. To understand their characteristics, previous research mostly uses vendor ratings, public feedback, sometimes private messages, friend status, and post content. However, most research lacks comprehensive (and important) data about transactions made by the forum members.

Our recent paper (original talk here) published at the Internet Measurement Conference (IMC’20) examines how an online illicit marketplace evolves over time (especially its performance as an infrastructure for trust), including a significant shift through the COVID-19 pandemic. This study draws insights from a novel, rich and powerful dataset containing hundreds of thousands contractual transactions made by members of HackForums — the most popular online cybercrime community. The data includes a two-year historical record of the contract system, originally adopted in June 2018 as an attempt to mitigate scams and frauds occurring between untrusted parties. As well as contractual arrangements, the dataset includes thousands of associated members, threads, posts on the forum, which provide additional context. To study the longitudinal maturation of this marketplace, we split the timespan into three eras: Set-up, Stable, and COVID-19. These eras are defined by two important external milestones: the enforcement of the new forum’s policy in March 2019, and the declaration of the global pandemic in March 2020.

We applied a range of analysis and statistical modelling approaches to outline the maturation of economic and social characteristics of the market since the day it was introduced. We find the market has centralised over time, with a small proportion of ‘power users’ involved in the majority of transactions. In term of trading activities, currency exchange and payments account for the largest proportion of both contracts and users involved, followed by giftcards and accounts/licenses. The other popular products include automated bots, hacking tutorials, remote access tools (RATs), and eWhoring packs. Contracts are settled faster over time, with the completion time dropping from around 70 hours in the early months to less than 10 hours during the COVID-19 Era in June 2020.

We quantitatively estimate a lower bound total trading value of over 6 million USD for public and private transactions. With regards to payment methods preferably used within the market, Bitcoin and PayPal dominate the others at all times in terms of both trading values and number of contracts involved. A subset of new members joining the market face the ‘cold start’ problem, which refers to the difficulties of how to establish and build up a reputation base while initially having no reputation. We find that the majority of these build up their profile by participating in low-level currency exchanges, while some instead establish themselves by offering products and services.

To examine the behaviours of members over time, we use Latent Transition Analysis to discover hidden groups among the forum’s members, including how members move between groups and how they change across the lifetime of the market. In the Set-up Era, we see users gradually shift to the new system with a large number of ‘small scale’ users involved in one-off transactions, and few ‘power-users’. In the Stable Era, we see a shift in the composition and scale of the market when contracts become compulsory, with a growth of ‘business-to-consumer’ trades by ‘power-users’. In the COVID-19 Era, the market further concentrates around already-existing ‘power-users’, who are party to multiple transactions with others.

Overall, the marketplace provides a range of trust capabilities to facilitate trade between pseudonymous parties with the control is becoming further centralised with administrators acting as third-party arbitrators. The platform is clearly being used as a cash-out market, with most trades involving the exchange of currencies. In term of the three eras, the big picture shows two significant rises in the market’s activities in response to two major events that happened at the beginning of Stable and COVID-19 eras. Particularly, we observe a stimulus (rather than transformation) in trading activities during the pandemic: the same kinds of transactions, users, and behaviours, but at increased volumes. By looking at the context of forum posts at that time, we see a period of mass boredom and economic change, when some members are no longer at school while others have become unemployed or are unable to go to work. A need to make money and the availability of time in their hand to do so may be a factor resulting in the increase of trading activities seen at this time.

Some limitations of our dataset include no ground truth verification, in which we have no way to verify if transactions actually proceed as set out in the contractual agreements. Furthermore, the dataset contains a large number of private contracts (around 88%), in which we only can observe minimal information. The dataset is available to academic researchers through the Cambridge Cybercrime Center‘s data-sharing agreements.