License Plate Reader Searches Should Require a Warrant

So while I work with police departments regularly, I think it is critically important that technology be used reasonably.

While this may be off-putting to some of my clients, I worked with the Institute for Justice as an expert witness in their trial Schmidt v City of Norfolk. (Any opinions herein are my own and not those of IJ, to be clear.) The gist of that case was whether searches of historically cached ALPR data (automated-license-plate-reader) constituted an illegal search.1

The judge ruled against plaintiffs in that case. Here is a quote from the judgment:

Consistent with Plaintiffs’ claims in this case and controlling precedent involving mass surveillance in public spaces, ALPR surveillance could become too intrusive and run afoul of [constitutional privacy standards] at some point. But when? While a definitive answer to that question is elusive, what is readily apparent to this Court is that, at least in Norfolk, Virginia, the answer is: not today.

The important point to note about this quote is “not today”. This will be a long winded post, but to try to keep it simple:

  • I think cameras will become ubiquitous in the foreseeable future. So the question is not if this data will require a warrant, it is when. It is going to happen eventually under current case law.
  • I think cameras are good, and can be used to reduce crime in a cost effective manner.
  • There is a difference between active flags (e.g. this car is stolen and it pings the PD when it drives past a camera) vs historical searches (e.g. look to see where license plate XYZ1000 was the last 30 days).
  • Requiring a warrant for historical searches will not seriously impede police investigations.
  • The current status quo of not retaining data is VERY BAD; it does not prevent illegal searches, and currently limits the utility of actually using that data for legitimate investigations.
  • Current standards to prevent abuse of the searching ALPR data systems are laughable.

Long story short in my opinion everyone would be better off if states just mandated warrant procedures through state statutes.

To try to not get too much into the weeds of what historically constitutes a search, I think the easiest place to start is via Carpenter vs US. So current US case law requires police departments to obtain a warrant to request cellular providers provide law enforcement with cell phone tower pings (cell-site location information, CSLI).

This deviated from historical precedent in requiring a warrant mainly because it was private companies that had the information. Before Carpenter, mostly it was argued you did not have a reasonable expectation of privacy if a private company could access the same data. The court in Carpenter basically made a determination that cell phone data was so comprehensive it justified a different standard – that you could track the whole of a person’s movements with the detailed CSLI data. And that this level of invasiveness violated a reasonable person’s expectation of privacy. Even if Google had all that info, you did not expect them to give it away.

This opinion was reaffirmed with the recent Chatrie decision (for geofence warrants, e.g. give me a ping for all cell phones in area X and datetime-range Y). Another relevant decision to be aware of is also Beautiful Struggle v Baltimore, in which searching historical aerial imagery via drones also constituted a search.

So this is why I am saying the question is when, not if, ALPR data will require a warrant. If a city happened to have a camera on literally every intersection (which I think will happen in the future), under current case law it would clearly be the same situation as you have for your cell phone data.

Cameras are Good

To be brief, again I mostly work with police departments in my career and was a former crime analyst. I do think ALPR cameras are good investments, mainly because they are cheap enough to have a reasonable return on investment. (Note I do not think this about all police tech, I am particularly critical of the price tag for acoustic-gun-shot-detection.)

So ALPRs are well under $3,000 per camera. The machine learning models, camera, and computation necessary to flag a plate when it passes can easily fit on current cell phones. (The harder part is powering the phone and protecting it from the elements.) ALPRs for the most part just take static images and then extract out the license plate (and for some vendors extract out additional information, like car make and color).

The overall evidence that ALPRs reduce crime is pretty meh at the moment (see my slides at a Wake Libertarian talk I did in 2024), but because they are so cheap they really only need to increase a few arrests per camera to likely have a positive return on investment.

It is pretty hand-wavy, as we do not have estimates for the value of increased clearances I find persuasive. But I think saying “I would pay $500 to help solve one case” is on the low side if anything. So a single camera if it helps catch just a handful of crimes a year is likely in my opinion to be a positive ROI.

I think cameras in all public spaces are going to happen. Imagine Ring comes out with a nicer camera system for homeowners that has more comprehensive views around your house and is just as cheap. And we will ultimately be safer for it. So even for folks advocating that cities do not pay for Flock, this is coming anyway in the near future.

Historical Searches vs Active Flags

ALPRs have been around a long time. The first ones I worked with at Troy, NY when I was an analyst were in-car cameras. Basically a go pro attached to the window that alerted when an officer drove by a stolen plate.

While ALPRs initial use was always pitched as this active flagging of stolen vehicles, they were used right away to retroactively search the historical locations of plates. They had a log of every plate, lat/lon, and timestamp of when that car passed a camera.

So imagine you are conducting an investigation of Joe Schmo, you know his license plate, and then you can type in his plate and see where his car passed a camera. Based on this information, same as CSLI data, you can basically trace where Joe went, where he repeatedly visited, where he likely slept, etc. (The first time I used this at Troy, we figured out a particular individual we were actively investigating was living with his girlfriend for example. I was honestly amazed how densely filled in the map was of hits for a single plate based on the in-car cameras.)

You technically do not need to cache any data at all to accomplish this “flag a stolen vehicle” (or any other scenario where you are actively looking for a specific license plate). There are legitimate scenarios though where ALPR searches for recent data in a real time context can be very helpful.

One of the more common examples – someone robs a gas station, and they drove a vehicle. You don’t know the plate, but can look at the images that passed by the fixed location ALPRs in the time range, and then especially if you have a car description from the gas station attendant can figure out the plate associated with the vehicle.

To be clear I am not a lawyer, but in my opinion I think exigent circumstances make searching a few minutes of cached ALPR location data totally reasonable. In practice, New Hampshire’s 3 minute data retention is far too short. I could see arguments for several hours (imagine “I found a dead body on the side of the road”, that requires more time for it to be reported.) But we are meandering into the territory where it is not an active emergency “need to find someone who may have a gun and hurt people” that would justify those exigent circumstances. Those are the scenarios where getting a warrant is reasonable (no different than a geofence warrant if you do not have a plate and want to just search what cars passed by a camera within a certain date-time window, or no different than a CSLI warrant if you have an active suspect and want to search for a specific license plate).

Most states are retaining ALPR data for longer periods. While the Norfolk case was ongoing, Virginia set a standard across the state at 21 days. Before that it was up to the individual agency. It varies state by state, but states often mandate data retention around 30 days, or leave it up to the discretion of the police department.

Deleting Data does not prevent abuses

These data retention statutes are argued as a mechanism to prevent abuse. They do not accomplish this.

If you look through the cases in which officers abused the system to search for individuals, all of them searched for specific plates over-and-over again, sometimes hundreds of times.

If you retain data for 20 days, you can just go and do a search every 20 days, keep notes on the data as you so wish, and then do another search 20 days later. Getting rid of old data, in-and-of-itself, does nothing to prevent that abuse. In fact if someone is actively stalking a person, you would expect them to regularly do searches, seeing where their victim is going on a regular basis while they have access to the system.

Simultaneously, deleting data does prevent its legitimate use in long term law enforcement investigations. It is totally normal for a murder investigation to take more than 30 days to identify a suspect. Gosh, sure would be nice to be able to then query the ALPR data to show whether a person was in the vicinity of the murder. Simultaneously it could be used by the defense for exculpatory purposes (which assuredly would take longer than 30 days).

So folks advocating for deleting data as a mechanism to prevent abuse are making things worse. It does not prevent abuse, and limits the utility of ALPR for historical investigations. The only way data retention by itself prevents abuse is if you do not cache data at all (like in New Hampshire), and only use ALPRs for the active alert situation.

What Smart Regulation Looks Like

One of the reasons I say that the current standards to prevent abuse are laughable is that data retention policies and internal PD policies on when the data should be searched have been in place in most departments for years (if not a decade) at this point. The examples where searching ALPR data to stalk an intimate partner were obviously not prevented via data retention policies.

Alas, my suggestion that some data is cached for real time investigations (longer than 3 minutes), and that a warrant should be required outside of this window, does not prevent that type of abuse either. Most departments have in place reasons why a search can be conducted, and some states have specific statutes identifying impermissible reasons for conducting searches. In the Norfolk IJ case, officers, when entering a reason for a search (which was often omitted), sometimes supplied reasons that appeared prima facie illegal, such as “protest”.

Departments, even if they have a standard to do internal audits, often do not follow them. It took Tyler Dukes asking Raleigh PD for their audit results for them to even conduct their first audit.

This is a long standing problem for PDs, not just with ALPRs, but also with searching criminal history illegally. IJ collating a dozen cases of arrests of ALPR misuse across the country is not evidence these systems are working, as it is likely the case that only the most egregious abuses are ever caught.

In addition to creating state statutes to mandate that a warrant be used for historical ALPR searches, states should, at a minimum, have clear punishments for illegal searches. These should include at a minimum losing your job, and being banned from accessing the system forever. When I was a crime analyst in New York (and ditto for when I worked at DCJS), this was the standard for misusing the criminal history search database.

If there is a standard for just retaining active search data for less than 24 hours, it does present a potential simple check that should be flagged – if a specific plate or specific camera is searched twice within 2 days, it should be flagged to review more closely. Flock does have their own system to identify suspicious search history.

The bigger issue to me though is who is doing the reviewing. It does not make sense to put this on vendors, and PDs just have not seriously devoted resources to this, even in response to public criticism. This audit mechanism should be delegated to a third party, either a specific group in the state attorney general’s office, or a state criminal justice agency (like DCJS in New York).

So that of course needs to be explicitly set by state statute as well. Who is doing the auditing?

My focus so far has been on abuses via police departments themselves, but smart regulation should also specify auditing of the vendors themselves, as well as punishments if they fail to meet data standards. (I am not thinking so much TEMPEST attacks here, but more so “I left an unauthenticated endpoint willy nilly on the internet”.)

Indeed, many of the requirements I am suggesting are likely already on the books; the problem is that the entity responsible for auditing is often unspecified or lacks the resources to do the work. (Also it is often unclear what the punishments are for failing to abide by statutes. That also needs to be specifically stated.)

The Future

So while I hope (although I have no expectation) that my blog post can somehow influence current standards across the country, I think it is important to keep in mind surveillance not just as the world exists now, but how it may look in the foreseeable future.

I think states should just pull the band aid off and create statutes that require a warrant to search the historical ALPR data. (And this makes data sharing between agencies mostly moot, the real time searches only need to be done within your own jurisdiction.) Like I said at the beginning, the current case law on being able to reconstruct the whole of a person’s movements (which I think is quite reasonable) will eventually be met if the ALPR cameras become dense enough. So states can either create the statutes to dictate that a warrant is necessary themselves, or eventually have the court system thrust it upon them.

In a world filled with privately owned cameras in public spaces, I think these suggestions are still relevant. So similar to Carpenter for CSLI data, and Chatrie for geofence warrants, there should just be warrant standards for historically searching any surveillance footage. There need be no special distinction between ALPR data (public or private) or video cameras.

Even if the groups calling for the banning of Flock cameras get their way, this does not stop private owners from collecting the data. So banning Flock, by itself, does not prevent abuse of searching private cameras. Again I think it is better to just let the government retain the data (same as private vendors will retain the data), and have consistent warrant standards for police to obtain that historical data.

This, of course, is a burden to detectives. I believe that trade-off in protecting our personal liberties while still allowing police effective means to investigate cases is a reasonable one.


  1. There are some technicalities between whether just collecting the data is a search (which was the scenario in the Norfolk case) or whether doing an active search (e.g. an officer querying the system for license plate ABC1234). The Norfolk case was the former, but for this post I am focusing on officers actually searching the data (the latter scenario).↩︎

VerusCite: checking academic articles for hallucinations

I have a new app out, VerusCite. With the recent rise in popularity of GenAI tools like ChatGPT and Claude, this has also come along with academics writing slop articles.

One of the ways to check that slop is via looking at the articles citations. LLMs have some predictable failure modes in writing papers whole cloth – they tend to get details like complicated author lists wrong, or swap out incorrect journal titles. VerusCite is a tool for editors and reviewers to use to verify citations in a fast and cheap application.

It costs $2 to review a paper (and you get two free reviews on sign-up). If you want to see the output of a single example though, check out https://veruscite-data.com/share/E1-rn3TBwksENO_3hlmsRD0IG-EnXd6vfQg-upQZWQM

In addition to hallucinations, I have made many parts of the application just useful to editors in general. Many papers have minor errors in their bibliographies; typos, years off, author swaps, bad URLs, etc. Here is an example – no hallucinations that signal poor writing, but has seven different errors in the bibliography.

This is par for the course (it is quite possible 5% of citations have errors that look like this). The website has convenient tools to edit citations and export the fixed citations (whether minor errors or gross hallucinations) in various formats.

This makes much lighter work of the tedious job of formatting and checking citations for editors. One of the ways I think is critical to build generative AI tools is to consider the human in the loop from the start. My tool will ultimately make some errors (I error rate estimates in my public benchmark). I want it to be as fast for a human to confirm (or refute) the LLM label.

If you are an editor or a reviewer, I highly suggest you check the application out. Peer review journals (and pre-print servers that review the applications before posting), will need to use a tool like this as a first pass to ensure slop is not being posted.

Notes on document processing with LLMs

One of the recent papers that came across my X feed was a researcher digitizing a corpus of old Sears catalogs to estimate inflation more accurately:

The technical skills to do this work are closely related to many of the projects I am working on at Gainwell – large-scale document processing using LLMs. So if you are a PhD student and your thesis involves work like this, I would likely want to hire you for a six-figure job as a data scientist at Gainwell.

I wanted to have a quick blog post on some of the tools to process documents. So this researcher processed 180 Sears catalogs that are typically well over 1000 pages, so they likely dropped well over $10k on this project.

Like I said in the X post, for this type of volume processing, you shouldn’t be using the main models. You should first consider the cheaper models, like the flash-lite models from Gemini (not even the flash for this), or the mini/nano models from OpenAI. Using these cheaper models, which I suspect would be of similar accuracy, would reduce the cost to more like $2k in this project.

For those even more budget-conscious (I am processing this volume often daily at Gainwell for different projects), here are a few of my notes on open source models. (Also I am concerned OpenAI will entirely drop the mini/nano models in the future, hence I want to make sure I have plenty of options.)

OCR and Markdown

So this project used the images to help classify the information, but if you are dealing with pure text, a common approach is to first OCR the document, and then apply structured extraction from the text. This works well to save costs even if using served models, as we are talking about an image of a page being 10k tokens, but the extracted text often being well under 1k tokens.

In LLMs for Mortals, I show using the docling library to do this:

If the PDF is already OCRed (it has the underlying text) there are Python libraries to directly read the text. This example uses pypdf, but a better default library is pypdfium2 (it is much faster).

The difference between converting to Markdown vs. reading the text is that Markdown conversion will be a bit nicer in converting tables, graphs, and page chrome (e.g. footers/headers).

I have had good success with docling (which you can have it convert to Markdown with just the page text), but it does load in a PyTorch model (which adds a bit of latency). You can turn it off to only use the text in the PDF (and not the image), which does save time but still has a fair bit of latency.

If the PDF has the text embedded and you do not want a big model, I have been happy with the results of the liteparse library. This does not use an LLM at all to do the conversion to Markdown, so is quite fast.

Due to the increased cost, if you do need a served model, (sometimes you just have latency requirements and you don’t want to deal with your own server), Mistral OCR is worth checking out. This is a good cost-effective alternative if you are using AWS Textract and need things like table extraction.

There are many different models and providers coming out in the space (OvisOCR2 is one of the recent models for OCR, for served models the group that created liteparse (LlamaIndex) also has served model options).

Structured Extraction

OK, so now you have your pages in text that you can use. At this stage, you will want to extract out information. My go-to local model for named entity recognition (NER) is GLiNER, but the GLiNER2 library has examples that do the full structured extraction.

For a primer on the distinction between NER and structured extraction, pretend you had a free-text narrative “PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA. Pt tolerated assessment well, no acute distress noted.”.

For NER, if you extracted vitals you would get back something like:

from gliner import GLiNER

# load GLiNER model
model = GLiNER.from_pretrained("urchade/gliner_small-v2")

# Physician Assistant narrative
narrative = """PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA. 
Pt tolerated assessment well, no acute distress noted."""

# define entities to extract
labels = ["heart_rate","oxygen"]

# extract entities
entities = model.predict_entities(narrative, labels)
print(entities)

And this prints out:

[{'start': 25, 'end': 34, 'text': 'HR 78 bpm', 'label': 'heart_rate', 'score': 0.9038965106010437}, {'start': 36, 'end': 43, 'text': 'SpO 98%', 'label': 'oxygen', 'score': 0.8429999351501465}]

So it identifies the specific text. Structured extraction can actually identify additional information. So here I just say collect all vitals, and identify if the note is from a physician assistant or nurse.

from gliner2 import GLiNER2

# Load GLiNER2 model
extractor = GLiNER2.from_pretrained("fastino/gliner2-base-v1")

# Physician Assistant narrative
narrative = """PA note: Vitals checked: HR 78 bpm, SpO₂ 98% on RA. 
Pt tolerated assessment well, no acute distress noted."""

schema = {"clinical_note": [
            "clinician_role::[nurse|physician assistant]::str::Role of the healthcare professional",
            "vitals::list::Vital signs collected"
          ]}

# Define structured extraction schema
results = extractor.extract_json(narrative,schema)
print(results)

And this prints out:

{'clinical_note': [{'clinician_role': 'physician assistant', 'vitals': ['SpO 98% on RA', 'HR 78 bpm']}]}

So it filled in the clinician role without the actual words “physician assistant” being in the actual note (it smartly inferred that from the “PA note” part).

NER is actually one of the techniques that local models are really the only viable approach, but structured extraction is still a case where I default to the small frontier models. But small open source models are worth testing (these can easily be run on a CPU).

They are just different use cases. NER is good for redacting info and then forwarding to an LLM (see OpenAI’s model, although I have had good success using GLiNER for this exact use case as well). But most of the time I want actual structured extraction – so turning PDFs/images into actual data you can put into a table.

If you want to skip the OCR part entirely and just submit images, see the NuExtract model (and follow Gio’s work, he is often writing examples of these).

If you are using Databricks, it has similar structured extraction tasks compiled down to UDFs that are directly available in SQL. (Also FYI, Teradata has a similar option to take models as an ONNX or PMML file and create a UDF function.)

Advice

So I talked a lot; let’s try to recap in some simple advice.

  1. Do not use the main frontier models (e.g. Sonnet, Sol) to do document processing and extraction tasks. Should default to trying models like Gemini Flash-Lite and gpt-nano at first to see if they work. Frontier models are both more expensive and often not any better than the tinier models.

  2. If you do not need to extract info out of images, converting documents to text first (and maybe converting that text to markdown) is a way to make extraction cheaper/faster than using images directly. There are multiple local models that run reasonably fast on CPU that can do this.

  3. If you need NER or structured extraction, check out GLiNER2. I would still default to using the cheaper frontier models for this often. But if they are too expensive (or you are a cheap grad student), the local models may be totally sufficient.

I am often using the frontier models at work, as I have throughput requirements and I default to not running a server as much as I possibly can. For production APIs, the latency requirements are often hard.

But in the example you are a grad student and have a corpus of 30k pages you want to churn through, you can just have your laptop burn through them overnight for a week if you want to cut costs.

Podcast with Jeff Asher, AI blog posts

Just a quick update on a few things. I was interviewed on Jeff Asher’s Jeffalytics Podcast, mostly about AI and the current landscape in criminal justice:

Using Technology to Improve Crime Analysis with Andy Wheeler by Jeff Asher

I don’t always link to my recent Crime De-Coder posts (follow the Crime De-Coder RSS feed or follow me on Crime De-Coder LinkedIn to stay up to date). But for a few recent technical ones:

AI is not going away, as so folks looking to learn about the tech, you should buy my book, Large Language Models for Mortals: A Practical Guide for Analysts with Python.

If you are an analyst and have questions always feel free to send me an email. Happy to expand those blog posts to material that is helpful.

Have been working on a new project very heavily in the past month+, so will be able to share that soon with the wider audience. (If you want to get some sneak peaks, follow me on X to see what I am working on.)

AI writing is better than no writing

AI disclosure – this post was entirely written by myself.

I know AI writing is still pretty cringey – so I get that people are quite opposed to it. For people like me though (academics promoting their work, more technical oriented) I would like to proffer a slight defense of (even cringey) AI writing. Having an LLM tool help you write a blog post is better than not writing at all.

I have come to the personal opinion I just want you to disclose when you use AI. I am starting to get peer review requests for academic papers that are clearly LLM written, and they are not obviously worse than the typical (mostly horrid) way academics write papers (they may actually be better to be honest). Blog and social media posts I think are strictly worse to my personal tastes when using LLM writing (across many dimensions, for now anyway). But it is better to write something than nothing if you have something worth saying.

Where this matters for technical folks (and academics) is that your default SEO is awful. Most academic papers are behind paywalls. LLM research tools are not picking up peer reviewed papers. So if you have something worth saying, having LLMs write out a blog post for you is worth it relative to having no writing at all.

For examples of LLM writing I have on this site:

And then my book, Large Language Models for Mortals: A Practical Guide for Analysts with Python, is around 50% AI generated.

None of these examples I would have finished without the help of AI; either entirely writing for the example blog posts, or writing the first draft in the case of the LLM book. (The LLM book is good by the way, you would not be able to tell I generated that first draft at all with Claude.)

My suggestion is to not let AI entirely take the wheel, but to create a detailed outline and have the LLM review your prior writing. Those two things improve posts by a wide margin (in addition to making sure AI is not too verbose – keep those blog posts simple!). And then you still need to take the time to review your own writing (for references you need to check those for hallucinations).

To be clear again, AI writing is better than nothing if you have something actually useful to say to the world. The bigger issue with AI writing are slop merchants just wasting space. That happened before with LLM tools, it is just much easier and more prevalent now. Just own it when you use AI to help you write.

Notes on Valuing the Cost of Crime

AI disclosure – I used AI to write this blog post. I figure having an AI blog post is better than not writing it at all. I will always disclose though if I use AI to heavily write any content on this blog. (I use it for minor copy editing all the time.)

For the tech details, I used gemini flash 3.5 with medium reasoning in the Antigravity IDE, using the same advice I said in this blog post. (Minor preference to Claude Code for writing blog posts for those who care.) It is the outline of the thread I did on X (which I wrote entirely by hand). Using this approach, e.g. I give a detailed outline and prior examples, Pangram says this is only lightly AI assisted.

Notes on Valuing the Cost of Crime

We often hear eye-popping figures about the “cost of crime.” For example, that a single aggravated assault costs society $100,000, or that a statistical life is worth $10 million. But if you look under the hood of these estimates, they are built on a house of cards: Willingness-to-Pay (WTP) surveys.

WTP estimates wildly inflate the costs of crime. For realistic policy decisions and police budgeting, we should be using concrete measures that are easier to calculate and verify.

The Three Buckets of Crime Costs

To evaluate criminal justice interventions, we can break costs into three broad categories:

  • A) Cost to the individual: Personal hospital bills, lost work, and physical trauma.
  • B) Cost to public sector agencies: Police labor, court proceedings, jail/prison operations, and public healthcare programs like Medicaid.
  • C) Cost to society: Reduced business activity in high-crime areas and the loss of workers to the economy.

Most cost-of-crime estimates do not calculate these countable categories. Instead, they use survey estimates of willingness-to-pay to approximate the costs of crime to individuals. I believe WTP estimates themselves are junk and should not be used to guide operations.

The Scaling Problem of Willingness-to-Pay

If you have heard the phrase “a statistical life costs $10 million,” you are seeing a WTP estimate in action.

The scaling math is straightforward, but the resulting estimates themselves are junk. Researchers ask survey respondents questions like: “Would you pay $100 in increased taxes to fund sidewalk improvements that reduce pedestrian fatalities?” If the safety measures are estimated to reduce pedestrian deaths by 1 in 100,000 annually in a city, the math scales up simply:

100 × 100, 000 = $10, 000, 000

People are thus deemed “willing to pay” $10 million to reduce one death.

This methodology yields massive, noisy estimates. You can see these WTP metrics compiled on the RAND Cost of Crime site. The primary limitation is that survey respondents will agree to pay almost any seemingly small amount when they do not actually have to pay it. In one street lighting survey I reviewed, participants were paid $1 to participate and claimed they were willing to pay $200 on average for better streetlights. It is highly doubtful that someone who sells their time for $1 to complete a survey will actually pay $200 in taxes for streetlights. As Andrew Gelman has pointed out, valuing lives based on ability to pay reveals how detached these hypothetical exercises are from real-world resource constraints.

Countable Costs vs. Theoretical Valuations

When we rely on concrete cost estimates that can be verified—such as labor hours and medical bills—the figures are much lower.

For instance, while a WTP estimate for an aggravated assault is close to $100,000, Priscilla Hunt’s study on law enforcement costs estimates the actual police labor cost for an assault is closer to $10,000.

I cannot prove what people are hypothetically willing to pay. But I can show a police chief that reducing ten assaults in a specific sector will save $100,000 in labor and overtime.

This distinction matters for other public costs too. Serious physical assaults can easily generate six-figure medical bills. In New York, more than 70% of gun violence hospitalizations are paid for via Medicaid. While it is reasonable for state or federal governments to weigh these medical costs, a local county or police department does not bear them. It makes no sense for a local police department to justify its budget by claiming it is reducing Medicaid expenses.

Example Cost-Benefit Case Studies

When we restrict our analysis to tangible costs, how do common interventions stack up?

Hotspots Policing

Because crime is highly concentrated, we can identify specific geographic areas that generate massive public costs. I have previously written about locating Million-Dollar Hotspots in Baltimore and Dallas. In my research on redrawing hotspots, I show how spatial concentration makes 24/7 hotspots policing cost-effective based purely on offsetting tangible labor costs.

For code examples of this, check out my crimepy python library (DBSCAN with weights for cost of crime estimates).

ShotSpotter

I am much less bullish on acoustic gunshot detection systems like ShotSpotter due to their high cost, as detailed in my ShotSpotter cost-benefit analysis. I estimate that ShotSpotter saves approximately 1 life for every 100 shooting victims it covers by dispatching emergency services faster. If you value a life at $10 million using WTP, the system easily looks cost-effective. If you use tangible costs, the math changes. ShotSpotter has not shown consistent evidence that it increases case clearances or prevents victimization. In fact, saving a shooting victim via faster response generates higher medical bills than if they had died, highlighting the complex economics of reactive vs. proactive interventions.

Business Improvement Districts (BIDs)

A great example of societal cost-shifting is Business Improvement Districts (BIDs). As shown in John MacDonald and colleagues’ study on BIDs in Los Angeles, BIDs demonstrate that commercial businesses are actually willing to spend their own money to improve safety in their areas through private security, cleaning services, and physical improvements. This is not hypothetical willingness-to-pay; it is a real-world, out-of-pocket expenditure by local merchants who calculate that reducing crime is directly worth their private investment.

Gun Violence Interventions (READI)

When looking at community-based interventions, the cost-benefit models face a different hurdle. Monica Bhatt and her colleagues evaluated Chicago’s READI program in their study on predicting and preventing gun violence. They claim a massive benefit of around $180,000 per participant (translating to a 3:1 benefit-cost ratio).

However, this estimated benefit of $180,000 is derived by mixing up WTP estimates and lifetime projections of individual offending (specifically, the Cohen & Piquero lifecourse model). As I discussed in my analysis of limits on gun violence interventions, extrapolating high-risk youth crime savings over an entire lifecourse using inflated WTP values creates a benefit estimate that is completely detached from the immediate budget realities of local governments.

The Missing Metric: The Value of an Arrest

This brings us to a major gap in criminology: we do not have good estimates for what it is worth to clear a crime.

Because crime is highly concentrated among a small number of chronic offenders, an arrest is often worth more than preventing a single crime. Apprehending a chronic offender can prevent dozens of future offenses.

This is why tools like automated License Plate Readers (LPR) are interesting. As Ozer’s study on LPR effectiveness shows, they are much cheaper than ShotSpotter and are highly cost-effective even if they only generate a small percentage increase in arrests. However, to truly calculate their ROI, we need a better grasp on the actual monetary value of a clearance.

To build better policy, we need to stop relying on WTP surveys and start measuring the real, tangible savings that police departments and local governments can actually bank.

References

  • Bhatt, M. P., Heller, S. B., et al. (2024). Predicting and preventing gun violence: An experimental evaluation of READI Chicago. The Quarterly Journal of Economics, 139(1), 1-56.

  • Cohen, M. A., & Piquero, A. R. (2009). New evidence on the monetary value of saving a high risk youth. Journal of Quantitative Criminology, 25(1), 25-49.

  • Hunt, P., Saunders, J., & Kilmer, B. (2019). Estimates of law enforcement costs by crime type for benefit-cost analyses. Journal of Benefit-Cost Analysis, 10(1), 95-123.

  • MacDonald, J., Golinelli, D., Stokes, R. J., & Bluthenthal, R. (2010). The effect of business improvement districts on the incidence of violent crimes. Injury Prevention, 16(5), 327-332.

  • Ozer, M. (2016). The impact of automatic number plate recognition (ANPR) technology on crime. Police Journal, 89(2), 117-132.

  • Wheeler, A. P., & Reuter, S. (2021). Redrawing Hot Spots of Crime in Dallas, Texas. Police Quarterly, 24(2), 159-184.

How long to conduct your experiment: Talk at ASEBP

Upcoming at the American Society of Evidence Based Policing Conference, I have a talk Thursday morning (9:45-10:00), How long to conduct your experiment.

The talk goes over some of the simple metrics I have created to help plan how long to conduct your intervention. Such as how long to evaluate your hot spots intervention, or purchase to increase arrest rates, etc.

I have prepared a ton of different resources. The main one is a web-based application (a WASM-based app with R as the backend) where you can enter your inputs and generate a graph showing how precise your parameter estimates are:

The help page includes citations and additional materials, but here is a brief rundown:

  • I have the math details in this github repo, see the methodology.pdf. It also includes notes on how I used different LLM tools to produce the webpage and the method materials. Each of the applications allows you to download the R code used to generate the graphs and tables.

  • I have created a series of YouTube videos demonstrating the application (WDD, IRR, Proportion tests)

  • I have posted my slides for the ASEBP talk

See you all in DC at ASEBP in a few weeks!

xAI voice cloning API

xAI has just released an API to clone your voice. It is pretty simple, read a script, and then an API where you can have text to speech in that voice.

Here is the python code after you have cloned your voice.

import os
import requests
voice_id = os.environ['ANDY1_VOICE'] # my demo voice ID
text = '''this is a test demo of my voice. Be excited!
OK, how about a list of things; one, two, three.
Lets see where this takes us.'''
response = requests.post(
"https://api.x.ai/v1/tts",
headers={
"Authorization": f"Bearer {os.environ['XAI_API_KEY']}",
"Content-Type": "application/json",
},
json={
"text": llm_book,
"voice_id": voice_id,
"language": "en",
},
)
response.raise_for_status()
with open("AndyTest1.mp3", "wb") as f:
f.write(response.content)

I need to figure out my audio set up a bit better (my mic set up is probably not optimal and it produces some echo). But does a good job imitating my boring voice right out of the box!

And here is an example for longer speech from my intro to LLMs book:

# intro to llm book
llm_book = '''
Large language models (LLMs) are transforming how we work. Some of these examples include using LLMs to help write computer code, using LLMs to extract out information from irregular text sources, and creating chat-bots that can interact with various data sources and documents.
Most analysts, however, do not have any experience with these tools. This book is meant to be a general introduction to realistic examples of how individuals can use these tools; either in general software applications, or to help analysts write code to create software itself. Given the rapid pace of advancement in this area, a general introduction to help individuals who work in the knowledge economy understand the capabilities of these tools I believe is in order.
Here is a simple example of using an LLM API (*Application Programming Interface* -- just a standard way to send information and get information back on the web) using the anthropic library in python to extract key information from a free text crime narrative:
'''
response = requests.post(
"https://api.x.ai/v1/tts",
headers={
"Authorization": f"Bearer {os.environ['XAI_API_KEY']}",
"Content-Type": "application/json",
},
json={
"text": llm_book,
"voice_id": voice_id,
"language": "en",
},
)
response.raise_for_status()
with open("AndyTest_LLMIntro.mp3", "wb") as f:
f.write(response.content)

The LLM intro messed up *Application Programming Interface* section (start listening at 50 seconds in). But otherwise it is very nice.

For those worried about security, xAI did something smart here — you need to input text live into the API given their prompts. You cannot have a pre-recording audio input to do this. So cloning someone elses voice is pretty hard.

Costs are around $4 per million characters in the text to speech API. So say narrating my entire book should be under $10 I believe.

Took me a total of less than an hour to set up a voice, create the python code, and write this blog post!

Pangram is good

Many of the initial wave of “AI writing detectors” were quite bad. The biggest issue you need to be concerned about with an AI writing detector is false positives. If you are a professor and want to check students’ writing, it is very bad to falsely accuse a student.

The Pangram product, though, is quite good, and I suggest folks check it out.

The other main competitor on the market, GPTZero, is clearly lower quality (such as saying the Constitution is AI generated).

GPTZero in their documentation says they are the most accurate AI detector. One of the reasons you don’t really care about accuracy is that you cannot know the underlying rate of AI writing in any corpus except in the scenario where it is artificially generated. And that is the only scenario in which you can know the accuracy for sure. What you care about is specifically the false positive rate and the false negative rate.

Unlike GPTZero, Pangram appears to have very low false positive rates. A simple way to estimate the false positive rate is to just submit writing prior to 2022 to the tool and see how many it flags as AI. ChatGPT came out in late 2022, the tools to generate writing before that were just not even close for people to use in any serious way. So any writing flagged in the older corpus as AI is a false positive.

Here is an example examining legal briefs.

It is an independent assessment. We cannot really know the capture rate (were there more than 66 briefs generated via LLMs in that sample). We can know the false positive rate though. And it is 1/800 in this sample with Pangram.

Pangram says it has a 1 in 10,000 false positive rate across a wide array of writing samples. They even report in their own internal tests that GPTZero has a 2% false positive rate (I am pretty sure GPTZero’s false positive rate is much higher than 2%, hence the Constitution error.)

Many other checks for false negative rates involve people having various models generate writing and then classifying it. It is hard to know if those are very good benchmarks for estimating the false negative rate. But we can easily estimate the false positive rate, and in that respect Pangram is clearly better than other AI writing detectors on the market.

Should we care if writing is AI?

I have used AI tools to help me write. I promise to be forthcoming if I use AI to help me write any substantive sections of writing (in blog posts, books, social media posts, etc.) Currently I am almost always using the LLMs to copy-edit, which is often simply a prompt “check for spelling and grammar issues”.

I do not use it all the time for writing. This post was all written by hand (and then just copy-edited with Gemini CLI).

It is really not that hard to bring your own voice and use AI to aid your writing. Have the LLM read your prior work, then give it a detailed outline, and then iterate. See my transcript on a prior post for an example.

I’d note I have used Pangram to see if my LLM writing is too obviously AI, and it is not. To me, when the writing is clearly AI, this often signals a clear lack of care and effort in the writing. AI writing can be valuable, but it is quite frequently low value slop.

So you get people larping as tech experts.

You can trivially have Claude or whatever software write a Skill file, and then have an LLM write how it is super awesome. This does not make it so.

And you have salespeople write posts that literally make no sense.

This, to be clear, is obviously AI slop.

So these individuals could actually generate useful content if they spent any more than a trivial amount of time. But they don’t, and it shows.

Gathering interest in tech courses

Quick post this morning — I have a survey up gathering input on interest in short, technical courses.

Think 2-3 days, potentially in person/synchronous.

If you have taken a course with Paul Allison at Horizon’s, or an ICPSR summer course, those are similar examples. But, the main difference will be these courses are to prepare you for pursuing private sector roles.

These will be aimed at:

  • grad level social science students
  • current professors looking to pursue private sector roles
  • current data analysts looking to get into data science
  • undergrads with some more technical background

Survey lists potential courses (python for data analysis, intro to LLM APIs, SQL + Dashboards, using agent based tools for analysis), the course medium (in person vs video), price points.

If you are a university or organization interested in hosting such sessions for your students, let me know as well. Happy to chat to you about bringing this to your campus.