Types of websites and trade-offs

For some minor updates on different fronts. I have a new blog post on Crime De-Coder about how to figure out the proper ODBC connection string to query your record management system. I have done this song and dance with maybe a dozen different PDs at this point (and happened to do it twice in the prior week), so figured a post would make sense.

Two, article with Kim Rossmo has been published, The journey-to-crime buffer zone: Measurement issues and methodological challenges. Can check out the github repo and CrimRXiv version for free.

Main reason I wanted to make a post today about the types of websites. I have seen several influencers discuss using GenAI to create simple apps. These tools are nice, but many seem to make bad architecture decisions from the start (many people should not be making python served websites). So I will break down a few of the different options for creating a website in this post.

The most basic is a static html site – this just requires you create the HTML and place it somewhere on a server. A free option is github pages. You can still use javascript apps, but they are run client side and there is no real ability to limit who can see the site (e.g. you cannot make it password protected to log in). These can handle as much traffic as you want. If you have data objects in the site (such as a dashboard) the objects just need to be stored directly on the server (e.g. in json files or csv if you want to parse them). You can build data dashboards using D3 or other WASM apps.

The other types of websites you can do anything you can in HTML, so I focus more on what makes them different.

PHP sites – this probably requires you to purchase an online server (ignoring self hosting in your closet). There are many low cost vendors (my Crime De-Coder site is PHP on Hostinger. But they are pretty low price, think $5 per month. These do have the ability to create password protected content and have server side functions hidden. My $5 a month website has a service level agreement to handle 20k responses per day, it also has a built in MySQL database that can hold 2 gig of data for my plan. (These cheap sites are not bad if all you want is a smallish database.) WordPress is PHP under the hood (although if you want a custom site, I would just start from scratch and not modify WordPress. WordPress is good to use a GUI to style the site with basic templates.)

IMO if you need to protect stuff behind a server, and have fairly low traffic requirements, using a PHP site is a very good and cheap option. (For higher traffic I could pay under $20 a month for a beefier machine as well, we are talking crazy site traffic like well over 100k visits per day before you need to worry about it.)

Node.js – this is a server technology, popular for various apps. It is javascript under the hood, but you can have stuff run server side (so can be hidden from end user, same as PHP). The tech to host a site is a bit more involved than the PHP hosting sites. You can either get a VPS (for typically less than $10 a month, and can probably handle close to the same amount of traffic as the cheap PHP), and write some code to host it yourself. Think of a VPS as renting a machine (so can be used for various things, not just webhosting.) Or use some more off the shelf platform (like FlyIO, which has variable pricing). You typically need to think about a separate database hosting as well with these tools though. (I like Supabase.)

Python – python has several libraries, e.g. django, flask, as well as many different dashboard libraries. Similar to Node, you will need to either host this on a VPS (Hostinger has VPS’s as well, and I know DigitalOcean is popular), or some other service. Which is more expensive than the cheaper PHP options. It is possible to have authentication in python apps, but I do not tend to see many examples of that. Most python websites/dashboards I am familiar with are self-hosted, and so limit who can seem them intrinsically to the companies network (e.g. not online in a public website for everyone to see).

Personal story, at one point my Dallas crime dashboard (which is WASM + python) on my Crime De-Coder site (which is served on the PHP site, so takes a few extra seconds to install), was broken due to different library upgrades. So I hosted the python panel dashboard on Google cloud app while I worked on fixing the WASM. I needed one up from the smallest machine due to RAM usage (maybe 2 gigs of RAM). Google cloud app was slower than WASM on start up, sometimes would fail, and cost more than $100 per month with very little traffic. I was glad I was able to get the WASM version fixed.

Dallas Crime Dashboard

It is all about trade-offs though in the architecture. So the WASM app you can right click and see how I wrote the code to do that. Even though it is on a PHP site, it is rendered client side. So there is no way to protect that content from someone seeing it. So imagine I wanted you to pay $5 a month to access the dashboard – someone could right click and copy the code and cancel the subscription (or worse create their own clone for $4 per month). For another example, if I was querying a private database (that you don’t want people to be able to see), someone could right click and see that as well. So the WASM app only makes sense for things that don’t need to be private. Google cloud app though that is not a limitation.

The mistake I see many people make is often picking Node/Python where PHP would probably be a better choice. Besides costs, you need to think about what is exposed to the end user and the level of effort to create/manage the site. So if you say to GenAI “I want to build a dashboard website” it may pop out a python example, but many of the examples I am seeing it would have been better to use PHP and say “I have a php website, build a function to query my database and return an array of crimes by month”, and then as a follow up question say, “ok I have that array, create a line chart in PHP and javascript using D3.js”.

So to me GenAI does not obviate the need to understand the technology, which can be complicated. You need a basic understanding of what you want, the constraints, and then ask the machine for help writing that code.

Stacking and clustering matplotlib bar charts

Just a quick post – recently was trying to see examples of both stacking and clustering matplotlib + pandas bar charts. The few examples that come up online do not use what I consider to be one of the simpler approaches. The idea is that clustering is the hard part, and you can replicate the stacked bars fairly easily by superimposing multiple bars on top of each other. For just a quick example, here is a set of similar data to a project I am working on, cases cleared by different units over several years:

import pandas as pd
import matplotlib.pyplot as plt

# data of cases and cleared over time
years = [20,21,22]
acases = [180,190,200]
aclosed = [90,90,90]
bcases = [220,200,220]
bclosed = [165,150,165]
ccases = [90,110,100]
cclosed = [70,100,90]

df = pd.DataFrame(zip(years,acases,aclosed,bcases,bclosed,ccases,cclosed),
                  columns=['Year','Cases_A','Closed_A','Cases_B','Closed_B',
                           'Cases_C','Closed_C'])
print(df)


# Transpose
dfT = df.set_index('Year').T
dfT.index = dfT.index.str.split("_",expand=True) # multi-index

The biggest pain in doing these bar charts is figuring out how to shape the data. My data initially came in the top format, where each row is a year. For this plot though, we want years to be the columns and rows to be the different units. So you can see the format after I transposed the data. And this is pretty much the first time in my life a multi-index in pandas was useful.

To superimpose multiple pandas plots, you can have the first (which is the full bar in the background) return its ax object. Then if you pass that to the subsequent bar calls in superimposes on the plot. Then to finish I need to redo the legend.

# Now can plot
ax = dfT.loc['Cases',:].plot.bar(legend=False,color='#a6611a',edgecolor='black')
dfT.loc['Closed',:].plot.bar(ax=ax,legend=False,color='#018571',edgecolor='black')

# Redo legend
handler, labeler = ax.get_legend_handles_labels()
hd = [handler[0],handler[-1]]
lab = ['Not Closed','Closed']
ax.legend(hd, lab)
ax.set_title('Cases Assigned by Unit 21-23')

plt.show()

To make this a percentage filled bar plot, you just need to change the input data. So something like:

den = dfT.loc['Cases',:]
ax = (den/den).plot.bar(legend=False,color='#a6611a',edgecolor='black')
(dfT.loc['Closed',:]/den).plot.bar(ax=ax,legend=False,color='#018571',edgecolor='black')

# Redo legend
handler, labeler = ax.get_legend_handles_labels()
hd = [handler[0],handler[-1]]
lab = ['Not Closed','Closed']
ax.legend(hd, lab,loc='lower left')
ax.set_title('Proportion Cases Closed by Unit 21-23')

plt.show()

This does not have any way to signal that each subsequent bar in the clustered subsequent is a new year. So I just put that in the title. But I think the part is pretty simple to interpret even absent that info.

Crime De-Coder Updates

For other Crime De-Coder updates. I was interviewed by Jason Elder on his law enforcement analysts podcast.

Also I am still doing the newsletter, most recent talks about my social media strategy (small, regular posts), an interesting research job analyzing administrative datasets at Chase, and some resources I have made for Quarto (which I much prefer over Jupyter notebooks).

Hillbilly Lament

I recently read JD Vance’s Hillbilly Elegy. I grew up in very rural Pennsylvania in the Appalachian mountains, and I am the same age as Vance. So I was interested in hearing his story. My background was different, of course not all rural people are a monolith, but I commiserated with many of his experiences and feelings.

I think it is a good representation of rural life, more so than reading any sociology book. The struggles of rural people are in many ways the same as individuals living in poverty in urban areas. Vance highly empathized with Wilson’s Truly Disadvantaged, and in the end he focuses on cultural behaviors (erosion of core family, domestic violence, drug use). These are not unique to rural culture. Personally I view much of the current state of rural America through Murray’s Bell Curve, which Vance also discusses.

This is not a book review. I will tell some of my stories, and relate them to a bit of what Vance says. I think Murray’s demographic segregation (brain drain) is a better way to frame why rural America looks the way it does than Vance’s focus on cultural norms. This is not a critique of Vance’s perspective though, it is just a different emphasis. I hope you enjoy my story, the same way I enjoyed reading Vance’s story. I think they give a peak into rural life, but they aren’t a magic pill to really understand rural people. I do not even really understand rural people, I can only relate some of my personal experiences and feelings at the time.

It also is not any sort of political endorsement. I do endorse people to read his book – even if you do not like Vance’s current politics the book has what I would consider zero political commentary.

Farming

Where I grew up is more akin to Jackson Kentucky than Middletown Ohio – I grew up in Bradford county Pennsylvania. The town I went to school in has a population of under 2,000 individuals, and my class size was around 80 students. A Subway opened in town when I was a teenager and it was a big deal (the only fast food place in town at the time).

My grandfather on my mother’s side had a small dairy farm. One of the major cultural perspectives I have on rural people is somewhat contra to Vance – farmers have an incredible work ethic. Again this is not a critique of Vance’s book, I could find lazy people like Vance discusses as well. I just knew more farmers.

Farming is Sisyphean – you get up and you milk the cows. It does not matter if you are sick, does not matter if it is raining, does not matter if you are tired. It would be like you not feeding your pets in the morning, and you do not get paid.

There was always an expectation of working. I was doing farm work at an early age. I always remember being in the barn, but even before I was 10 years old I was taught to operate heavy machinery (skidsteer, tractors). So I was doing what I would consider “real work” at that young, not just busy work you give a child to keep them preoccupied.

My grandfather retired and sold his farm when I was 11, but I went and worked for a neighbors farm shortly there after. We had a four-wheeler that I would drive up the road in the wee hours of the morning to go work. I was not an outlier among other kids my age. The modal kid in my school was not a farmer, but there were more than a dozen of my classmates who had the same schedule.

Farming, and manual labor in general, is brutal. When Vance talked about the floor tile company not being able to fill positions (despite the reasonable pay), I can hardly blame people for not taking those jobs. They are quite literally backbreaking.

Farming has become more consolidated/automated over time. The dairy farms I worked on were very small, having less than 100 cows. One of my memories is being exhausted stacking small square bales of hay. There is a machine that bundles up the hay and shoots it onto a wagon. I would be in the wagon stacking the bales. Then we would have to load the bales up a conveyor belt and stack them in the barn. The bales weighed around 50 pounds, it was crazy hard work.

Round bales were only starting to become more common in the area when I was a teenager. You can only move the larger round bales using farm equipment. I believe almost no one does the small square bales like I am describing anymore. The smaller square bales were also a much larger fire hazard. If the hay was wet when baled it would go through a fermentation process and potentially get hot enough in the center of the stack to catch fire.

This is one of the reasons I do not have much worry about automation taking jobs. A farmer not needing to hire extra labor to stack hay and instead use a tractor to do the same work is a good thing. This will apply to many manual labor jobs.

The Culture of Danger

One thing I had always been skeptical of when reading in sociological texts was the southern culture of violence. Saying southern people have an honor culture and then showing homicide rates in the south are higher is very weak evidence supporting that theory.

The first person who had told me of that hypothesis in my PhD program at Albany was a professor named Colin Loftin. I think I laughed when he said it in class and I straight up asked “is this a real thing.” He was from Alabama and assured me it is, and told some story of a student fighting after bumping into another person walking in a hallway as an example. For those who do not know Colin, imagine a nerdy grandpa accountant with a well groomed white beard stepped out of a Hallmark movie. And he tells you southern people are prone to violence over trivial matters of honor – I was not convinced. As I said I grew up in a very rural area; Pennsylvania is sometimes described as Philadelphia, Pittsburgh, and Alabama in between. But I did not experience that type of violence at all.

Vance’s book is the first confirmation of the southern culture of violence that I believed. The area I grew up in was substantively less violent. I would consider it more northeastern vibes than what Vance describes as southern honor culture. I am sure I could find some apocryphal violent stories similar to what Vance describes if I prodded my relatives enough, but I did not take that to be a core part of our identity the same way Vance does.

Culture is hard to define. The ideal of “you work at all costs” I take as part of farming culture. It was not an ideal more generally held among the broader population though. I definitely was familiar with adults who were lazy, or adults who had the volatile lifestyle of Vance’s mother he describes in his book. But I was personally familiar with more people who day in and day out performed incredibly hard manual labor.

Another aspect of growing up, which I have coined the culture of danger, is harder to associate specific behaviors with. But it is something that I now recognize was constantly in the background, something we all took for granted as a given, but is not something that is more broadly accepted in other parts of society.

Farming is incredibly dangerous. There are steel pipes that transport the milk from the cow to the bulk tank. After you are done with a round of milking, the interior of the pipeline gets washed with a round of acid. Then the pipes are rinsed with a second round of a much more concentrated caustic solution to get rid of the residual acid. When I was around 6 years old, me and my older sister were playing in the barn and I spilled the more caustic solution on myself. They were housed in large plastic containers that had a squirt top on them (no different than the pump on your liquid hand soap). All it took was a push, and it melted my nipple off (through the shirt I was wearing at the time). Slightly different trajectory and I would be missing half of my face.

Like I said previously, I learned to operate heavy machinery before I was 10. I distinctly remember doing things in both the skidsteer and the tractor when I was young that scared me. Driving them in areas and in situations where I was concerned I would roll the machines. Skidsteers are like mini tanks with a bucket on the front. I do not think it had a seat belt, but if I rolled the skidsteer I think my chances were better than 50/50 to make it out (it had a roll cage, but if ejected and it rolled over you, you are going to be a pancake). If I rolled a tractor (these are very old tractors, no cab), I think your chances are less than 50% you get out without serious injury.

Learn to operate here meant I was given a short lesson and then expected to go do work, by myself without supervision. Looking back it was crazy, but of course when I was a kid it did not seem crazy.

Another example was climbing up silos. At the farm that I worked on, he did not have a machine to empty out the silo. So I would need to climb the silo, and fork out the silage for the cows feed. Imagine you took your lawn mower over a corn field, the clippings (both the cobs and the stalks) are what silage is.

I would climb up a ladder, around 50 feet, carrying a pitchfork. When I was 12. This was not a rare occurrence; this was one of my main responsibilities as a farmhand, I did this twice a day at least.

The cows were also fed a grain like mixture (that is similar to cereal, it did not taste bad). I have mixed feelings about the grass fed craze now, since the cows really enjoyed the grain and the silage (although I do not doubt a grass fed diet could be better for their overall health). And I do not know if feeding them just baled hay counts as grass fed, or if they need to be entirely fed with fresh grass from the field.

Some may read this and think the child labor was the issue. I do not think that that was a problem at all. To me there is no bright line between doing chores around the house and doing chores in the barn. I was paid when I worked on the neighbors farm, and it was voluntary. It was hard work, but it did not directly impact my schooling in any obvious way. No more than when a kid does sports or scouts. The operating heavy machinery when I was a child was crazy though, and the working conditions were equally dangerous for everyone.

Even more broadly, just driving around the area I lived was dangerous. There were six males I went to high school with that died in traffic accidents. So in a group of less than 300 males nearby me in age (+/- two years), six of them died. I can not remember what the roads MPH was graded at, but they were winding. I am not sure it matters what the MPH is, there is no way reasonable way to enforce driving standards on all those back roads.

Death and danger was just part of life. Johnny died in a car accident, Mary is having a yard sale, and Bobby rolled the tractor, his femur broke the skin but he was able to crawl to the road where Donny happened to be driving by, so he will be ok. So it goes.

The culture of danger as I remember it does not have as many direct manifest negative behaviors as does “honor culture” or “I am too lazy and too high on drugs to keep a job”. So maybe some real ethnographers would quibble with my description of it as a culture.

I do think this is distinct from how individuals in certain areas have a constant specter of interpersonal violence. I do not have any PTSD type symptoms from my experience, like Vance describes based on his experience with child abuse. In the end I suspect the area I grew up had worse early death rates than many urban areas with violence on a per capita basis, but the nature of it doesn’t have quite the same effect on the psyche. Ignorance is bliss I suppose, you get used to driving tractors on steep hills.

Steaks are for rich people

One of the places I remember eating at growing up was Hoss’s Steakhouse in Williamsport. So if we went to get clothes for school at the Lycoming Mall as a family, we would often eat there on the way home. It is a place with a nice salad bar. You could load up a salad with a pound of bacon bits, and have a soft serve ice cream on the side if you wanted.

Walking into the restaurant before you are seated are pictures on the walls of the meals. I was maybe 14 at the time, and waiting to be seated I pointed to one of the steak meals and said “I would like to get that”. The immediate response from my mother was “No you are not. You are getting the salad bar like always.” One of the ironic parts of this story to me is that, because I was working as an independent farmhand, I had my own money. I could have certainly paid for a steak dinner with my own cash.

After I was 16 and could drive, I did end up doing the majority of my own clothes shopping. I remember splurging on some really ugly yellow and green Puma sneakers one year (totally worth it, they were $80 and did get the “whoa nice shoes” comments at school as intended).

I have only recently developed a palette for steak. My son likes it and requests we go to the local steak house on occasion. I have for awhile actively encouraged him to buy whatever expensive steak is on the menu when we go out to eat (Waygu beef at the sushi place, that sounds interesting you should try that). Makes me feel like the god damn king of the world.

Soda and Beer

When I was young (less than 10 years old), during summers I would spend most of my time on the farm, but once a week visit my other grandparents. I would do two activities with my grandfather: either golf or go fishing. For golf we had a par three 9 hole course in my town. I cannot hit a driver straight to save my life, but my iron game is at the level where I would not embarrass myself.

Fishing was just in little ponds in the area, mostly sunfish and bass. We would bring a sandwich, two Pepsis for myself, and two beers for Grandpa.

I tell this story both because it is a fond memory, as well as it highlights one of the stark differences in my lifestyle now (in terms of healthy eating) vs back then. I am pretty sure I drank more soda than water growing up. Our well water was sulfurous, so it was quite unpleasant to drink. Soda was just a regular thing I remember everyone having, including kids.

Charles Murray in his Bell Curve proactively addresses most of the negative commentary you hear about right in the writing. But one thing I thought was slightly cringe at the time I read the book (sometime while getting my PhD) was his “middle class values index”. To be clear these were not things about healthy eating, but were more basic “received a high school diploma” and “have a job”. The items I did not object to, but the moniker of “middle class” I thought was an unnecessary jab for something that was so subjective.

In retrospect though, “feeding kids soda like it is water” and “driving around with open containers of beer” is the most apropos “not middle class values” I can think of. So now I do not hold that name against Murray. These are not idiosyncratic to rural areas, you can identify people in poor urban areas who behave like this as well. But you definitely do not need to worry about being pulled over for an open container while driving where I grew up.

In Pennsylvania at this time, to buy beer you needed to go to a distributor. You could not get a six pack at the gas station. You needed to get at least a 24 pack. I figure this limited the number of people purchasing beer, but I do wonder for those who did buy beer if it increased the amount of binge drinking.

Role Models and Choices

For a crazy dichotomy with how I grew up versus what I know now, one of the only role model career choices I remember individuals talking about growing up were teachers. Getting a job as a teacher at the school district was a well respected (and well paying) career option in my town.

An aspect of this that can only be understood when you are outside of it is how insular this perspective is. A kid wants to be a teacher, because that is pretty much the only career they are exposed to. This is from the perspective “I may need to go to school, but I will come back here and get a job as a teacher”.

I personally did not have much respect for my teachers in high school, so I never seriously considered that as a career option. I was an incredibly snarky and sarcastic kid. My older sister (and then later my younger sister) were salutatorians of their classes. I (quite intentionally) did not try very hard.

For one story, my physics teacher (who was friends with my father) called home to ask if I was doing ok since I slept in class. My mother asked what my grade was, and since it was an A, she did not care. For another, which I am embarrassed about now, I would intentionally give wrong answers (it was history or civics I believe) because the teacher would get upset. I found it hilarious at the time, but I realize this is incredibly churlish now (he cared that we were learning, which I cannot say for all of my teachers). Sorry Mr. Kirby.

So, I was not a super engaged student.

Vance talks about it seeming like the choices you make do not matter. I can understand that, it did not seem to me I was making any choices at all when I think back on it. My parents did always have an expectation that I would go to college for myself and my siblings. Working as a farmhand (for other peoples farms at that point) was never a serious option.

Both my parents had associates degrees, and my sister (who was two years older than me) went to Penn State for accounting. That was about the extent of college advice I got – you should go. I never had a serious conversation about where I should go or what I should go for. Choose your own adventure.

I remember signing myself up for the SAT. I took the test on a Saturday morning in a testing center a few towns over. I finished each of the sections very fast, and I scored decently for not practicing at all (a 1200 I believe, out of a possible 1600). I have consistently done excellent on math and poorly on the English portions of tests in my life; I think I had 700 in math and 500 in English for the SAT.

One funny part of this is that, until graduate school, I actually did not understand algebra. Given my area of expertise (statistical analysis and optimization now), many people think I am quite a math geek. I did have good teachers in high school, but I was able to score this high on the SAT through memorization. For example I knew the rule for derivatives for polynomials, but if you asked me to do a simple proof at that point I would not have had any clue how to do that.

When I say I did not understand algebra, I mean when I was given a word problem that I needed to use mathematical concepts to solve, I just figured it out in my head. It was not until graduate school that at some point I realized you can take words and translate them into mathematical equations.

I know now that this is somewhat common for intelligent people when learning math. I home school my son, and I noticed the same thing for him. So it took active engagement (forcing him to write down the algebraic equivalent, even when he could figure out the answer in his head). But just rote memorization can get you quite a good score on the SAT.

Individuals who want to get rid of standardized testing because poor people score worse on average is the wrong way to think about it. You should want to improve the opportunities for individuals to get better education, not stick your head in the sand and pretend those inequalities do not exist.

SAT results in hand, I remember asking the guidance counselor about school information for Bloomsburg University (I chose Bloomsburg because it was not Mansfield and not Penn State, and was cheaper). And her response was simply to hand me a single page flier.

I can understand my parents not giving decent college advice; they did not know about scholarships or what opportunities were available. The high school guidance counselor in her sloth though makes me angry in retrospect. Our grade had less than 80 kids – she could have spent a few minutes reviewing each of those kids backgrounds, and provided more tailored advice.

I am positive none of the kids in my class went to undergrad at any more advance school than Penn State (and even that may have only been one student in my class). Cornell (an ivy league school in Ithaca, New York) is actually closer than Penn State to where I lived – I did not know it existed when I was in high school. To be fair, I do not know if the guidance counselor knew my SAT score, but she could have asked. She certainly had access to my grades, could see I did well in STEM courses, and could have easily given suggestions like “you can apply for partial scholarships to many different places.”

This goes both ways, I knew several of my classmates that should not have gone to college. My best friend in high school was a solid B/C student, went to Mansfield for journalism, and stopped going in his junior year. He is doing fine now as a foreman for a natural gas company. Going to get a four year BA degree for him was a bad idea and waste of money. And it was doubly bad going for journalism.

The brain drain had not happened yet in my high school. The level of discourse in my high school classes was excellent. I noticed a significant regression in the level of discussion in my first classes at Bloomsburg relative to those in my high school. (Bloomsburg had an average SAT score for entrants of 1000.)

I am intelligent, but I was not the most intelligent in my class. There were easily 10 other people in my class that had comparable intelligence to me. All of whom would have likely qualified for at least partial Pell grant assistance.

For those in my class that did go to college, pretty much everyone went to one of the PASSHE schools (these are state schools in Pennsylvania, originally founded as normal schools that were intentionally spread out in rural areas). Most went to the closest nearby (Mansfield), but a few spread out to various institutions across the state (Lock Haven, Shippensburg, Indiana, etc.).

I have hobnobbed with ivy league individuals (professors and students) since getting my PhD. There is no fundamental difference between kids who go to ivy league schools and the kids I went to high school with. With a semester of SAT test prep, and a not lazy guidance counselor helping apply to scholarships, we could have had double digit number of kids accepted to prestigious institutions for zero cost.

Do Not Talk About Money

I remember asking my grandfather why barns were red. He said it was because red paint was cheaper. That was the only conversation I can remember in my childhood that discussed money in any form.

When going to college I was filling out my FAFSA form, and asked my father how much money he made. His response was “enough”. Vance brings up the idea that, ironically, going to nicer colleges is cheaper for poor people. But they have no clue about that. I am fairly sure I would have qualified for partial Pell grant assistance – I just left the section on your parents income blank on the FAFSA form.

Besides inept counsel on college, even though I had worked all these different jobs I only remember actively thinking about pay when I was working different jobs in college. The floor board factory was over $8 per hour. The ribbon factory was $11 per hour. Later when I worked as a security guard for Bloomsburg University I made $13 per hour.

Similar to Vance’s experience in Middletown, $13 per hour is quite decent to get an apartment and put food on a table for a family in that area of PA (at least at that time, 2004-2008). You are not saving up for retirement, but you shouldn’t need to live in the dregs and go hungry either.

In retrospect the advice I needed at the time (but never received) was real talk about pursuing careers. This is wrapped up in college, you go to college to prepare yourself for a career (the expectation I go to college was certainly not only to obtain a liberal arts education!)

I ended up choosing Bloomsburg University because I knew many of my classmates were going to a closer school (Mansfield) and just wanted to be different. There was no thought into choosing criminal justice as a major either. When folks ask me the question “what did you want to be when you grew up”, I can not remember actively thinking about any specific career. Even when I was young and hitting baseballs in the back yard, I knew that I was not going to be a professional baseball player.

I remember at one point in the middle of undergrad at Bloomsburg realizing that a criminal justice degree is not really vocational, and I could quit if I really wanted to just go and be a police officer (the only vocation I likely associated with the major). Which I did not really want to do. So I was debating on transferring to Bucknell for sociology, or some community college for whatever degree you get to work on HVAC systems. (I do not know where the Bucknell idea came from, I must of thought “it was fancy” or something relative to Bloomsburg.)

I could have used other advice, like “you can negotiate your wage”, but likely understanding my career options was the one thing that negatively effected my long term career progression. I do not mean to denigrate the HVAC job (given my background and what I know now, I am pretty sure that would have been a better return on investment than sociology at Bucknell!)

Not that I would go back in time and change anything (I received an excellent education at Bloomsburg in criminal justice, and ditto at SUNY Albany). But if someone somewhere said “hey Andy, you are pretty good at math, you should look into engineering”, my life trajectory would likely be very different.

Factory Jobs Suck

After I was able to drive at 16 I started to take other jobs outside of being a farmhand. These included being a line cook and dishwasher at a local restaurant where my aunt-in-law was a chef, and working for a company that did paving and seal-coating before I was 18. Cooking wasn’t bad. The restaurant was actually a mildly fancy steak and seafood place you needed a membership to eat. Thinking back, I am honestly confused how that many people where I grew up could afford a membership that would make that business model work.

Paving and seal-coating was comparable to the level of effort of farming. It was safer in the short term than farming (in a “I probably will not be maimed way”), but breathing in the fumes I am guessing would be worse long term. I do not remember my hourly wage (it may have just been the minimum wage), but I did get a bunch of overtime in summer which was nice.

When I went to college I then did various jobs as well. I worked at Kentucky Fried Chicken at the cash register at one point. On campus, jobs intended for undergraduate college students through the university, I worked as a carpenter building theater sets and as a tutor for the stats classes in the criminal justice department.

The last time I moved home though over summer break (after sophomore year at Bloomsburg) I got a job stacking uncut floorboards in a factory. This was my first factory job – we would stand on a conveyor belt down the machine that cut the boards. Our palettes would be stacked with a single size and we would rotate sizes after a while. So sometimes I am stacking 4 inch wide boards, another time I am stacking 12 inch wide boards, etc.

This was monotonous and hard work, but not crazy bad and I enjoyed my coworkers. Stacking hay bales was harder. Many of the people I worked with were on work release from the county jail. I got paid $8 an hour, they only got $4. But they were happy to do the work and not be sitting in jail. It is absurd that they did not receive the same pay. Most of them were in jail for DUIs.

After about a month of doing this job I had inflammation in my elbow. My elbow only had minor pain, but my arm would fall asleep when I slept and I would wake up in quite a lot of pain from that (so lack of sleep was really the bigger issue than my arm hurting). I asked to take the day off to go to the doctor (one of my coworkers said he had the same issue, but still worth working rather than sitting in jail). The owner mistakenly thought I was trying to get workers compensation, so told me no to the day off and I would be fired if I went to the doctor. (That was not my intention, I just wanted to get some pain medication.) So I just ended up quitting. Doctor said it was “tennis elbow,” and that it would only go away with rest, so I would have needed to quit anyway.

I then moved back to Bloomsburg for the summer (the town I grew up in was incredibly boring, hanging out in an empty college town was certainly a step up for a 20 year old). I got a job at a ribbon factory in Berwick (a neighboring town) for $11 dollars an hour. You could consider Berwick a doppelganger for Middletown as Vance describes it.

Working at the ribbon factory was barely manual labor. I would sit on a conveyor belt and either count bags to fill in boxes, or look at ribbons as they rolled by only to throw away malformed ones. This was soul sucking work. There was only one other younger person I befriended while working there, most everyone else was middle aged. I wondered to myself how these people survived this existence. Of all the jobs I have had in my lifetime this was easily the worst.

Despite having the “work every day” mentality from when I was young, I just stopped going to this job alittle over a month after I started worked there. I did not tell my boss I quit, just literally stopped going. It wasn’t a hard job, the opposite, it was easy. A second grader could do the job.

So this is often what I think about when people say “the factory jobs are going away,” or Vance’s example that the tile company that cannot get people to work for them. You have a choice, break your back or watch ribbons go by on a conveyor belt. I recognize that having people just take a paycheck from the government is not good for people long term. I think people need something to work to strive for and take pride in. Working in a factory is not that.

It was at this point (in between being a sophomore and junior) I went from just doing the bare minimum to get by for my classes at Bloomsburg to being actively engaged in my course work and putting in real effort. Working at the ribbon factory was the nadir. Having the more advanced upper level classes did make me more engaged. It was around this time I began working as a security guard for the university. I made the most for that position of any job I had at that point in my life, $13 per hour.

I worked night shift for the security guard job. There was one point in my schedule where I needed to stay awake for over 48 hours. I would get off at 4AM, and if I went to sleep I would not wake back up (even with an alarm) for a 9AM class. So I would have to stay up, go to class, and then sleep for an extended period of time.

By my senior year of college, I was back in the working crazy all the time stage. At one point I had three jobs (college tutor for statistics classes, working as a security guard, and even had work to help with statistical analysis for the local school). This is in addition to being a full time student.

Trailers and Going to Grad School

In the summer between junior and senior year at Bloomsburg University, I had an internship with state parole. The officer I shadowed had an area that covered the counties around Scranton, so a mix of rundown rust belt towns (like Berwick) but also more rural areas. There were more people living in trailers on single lots than trailers in trailer parks.

The first house call I shadowed was an individual who only had a few more weeks on his sentence. He was very nervous and sweaty (which was my first house call I witnessed, so I did not think much of it at the time). The parole officer had the individual do a urine sample. I found out later that he failed (heroin), and the parole officer said the reason he was nervous is that they take multiple officers to arrest individuals, so he likely thought he was being taken back to prison. It probably was not the failed drug test, which him being that close to finished would just result in a warning.

Shadowing parole was an eye opening experience. I had lived in rural areas, but I had been mostly sheltered from the decrepit lifestyle some people lived. Some houses the parole officer would make his parolees meet us outside, as he would refuse to go inside the house. People would not let their animals out (so the house smelled of the strong ammonia scent from the urine, much worse than the barn). Houses with kids sleeping on mattresses in the living room and fleas. I knew people like this existed, but seeing them firsthand was different.

Matthew Desmond’s book, Evicted, in which he follows the lifestyle of various individuals trying to scrape by in Milwaukee, reminded me quite a bit of my time with parole. A bunch of people who could not make two good decisions in a row if their life depended on it. I presume getting drunk before your scheduled parole visit or not cleaning your sink and getting evicted are consequences of the same inability to make good long term decisions.

All of those individuals had no fundamental reason they needed to live in filthy conditions. You can take the cat litter out. People dig on trailers and trailer parks, but living in a trailer is not fundamentally bad. It is no different than living in a small apartment.

So I had planned on applying to be a parole officer after this experience. It was likely I would not be assigned the field area I did my internship, but either assigned a position in a prison (they have officers inside state prison help with offenders release plans). Or maybe be assigned in the field in the Philadelphia area. So I took the civil service exam to be a parole officer in the fall semester of my senior year.

I had made a mistake though, I had taken the exam too early. They called and asked if I could go to training in the spring. I said I could not do that, as I wanted to finish my degree. (The parole officer I shadowed had quit one semester early, and he did say he regretted that.) The civil service exams in Pennsylvania had a rule that you could not re-take them within a certain time frame, so I did not have the ability to take them again when the timing would have worked out better.

So at this point in the fall semester I decided to apply to graduate school, not really knowing what I was getting into. The other option was applying to different police departments in the spring (I remembered Allentown and Baltimore had come to classes to recruit). I did well on the GREs, and was accepted into SUNY Albany (my top choice) quite early. I had also applied to Delaware. SUNY was somewhat unique, in that you could apply straight into the PhD program from undergrad. I did not realize this at the time, but this was very fortunate, as PhD programs were funded. I would have racked up a bill for the masters degree at Delaware.

When I ended up getting into grad school at SUNY Albany, Julie Horney called me in the afternoon of one of my binge sleep sessions in my night security guard schedule to say I was invited for orientation. I do not remember what I said on the phone call, I remember getting up later and not being sure if that was a dream or it had really happened.

Later that spring when I visited Albany I headed straight up from my night shift to the orientation day. I remember being confused, thinking this was an interview and still not 100% guaranteed I was in. I said something to Dana Peterson and her response was along the lines of “you do not have to worry Andy, you have gotten in”.

Going to Albany ended up being one of the greatest decisions of my life. The academic atmosphere of a PhD program was totally different and fundamentally changed me. It would be a lie though if I said it was something I intentionally pursued, as opposed to a series of random happenstance factors in my life that pushed me to do that. I really had no clue what I was getting into.

Drugs

Growing up in Bradford county I had very little exposure to drugs. My friends and I would pinch beer and liquor from our parents on occasion, but in the grand scheme of things I was a pretty square kid. I knew some individuals smoked pot, but I did not know anyone in high school who did heroin or other opiates. Unlike Vance, serious drug or alcohol abuse was not something I personally witnessed in my family.

It was around the beginning of 2000’s that the gradual increase in opioid overdose deaths started to happen across the US. In the town with the ribbon factory, Berwick, heroin usage was an issue. My sister in law (who grew up in Bloomsburg) ultimately died due to long term heroin usage. I worked on a grant to help Columbia county analyze the Monitoring the Future survey (a behavioral health survey that all students in Pennsylvania took). There were a few students in each grade cohort (as young as seventh grade) who stated they used heroin.

If you look at maps of drug overdose deaths in this time period, you can see a cluster start to form around the Scranton area by around 2005. This area in western Pennsylvania is close enough to commute to New York City. It is possible the supply networks for heroin from more urban areas were established that way.

I suspect it is also related to working labor jobs though. One reason my grandfather retired from farming was because he had chronic shoulder pain. I would drive him to his visits to the VA hospital in Harrisburg on occasion, in a full size van, when I was a teenager. He was prescribed oxycodone, but knew that it was addictive, so he would take them one week and then abstain the following week.

I do not know how you work these labor jobs and not have some type of chronic pain. It is hard for me to imagine working these jobs for thirty years without them killing you. I recently had a kidney stone, and I was stuck waiting for several hours in the ER before I was able to get a fentanyl drip. Went from pulsating pain and throwing up to relief almost instantly. I was prescribed oxycodone to use at home before I passed the stone. I did not take it.

When I was a professor at the University of Texas at Dallas, a well respected qualitative criminologist came to give a talk. He discussed his recent work, an ethnography of methamphetamine users in rural Alabama. His main thesis in the talk, which is something I think only an academic sociologist could come up with, is that women were influenced by their boyfriends to begin taking meth. (This is true, but you could do the same talk and say men were introduced to drugs by their female partners.) I asked at the end of his talk whether he thought his findings extended to heroin users, and his response was that heroin is an urban drug problem.

Another main point of his talk was to take pictures of his subjects in realistic settings. The idea being that most drug users are depicted in the media in a negative light. So we should take pictures of them so people do not think they are monsters. The lecture was mostly pictures of peoples trailers, and pillow talk his interviews discussed that resulted in snickers from the audience at various points.

Taking pictures of trailers does not humanize poor people. It makes you look like you are Steve Irwin describing wildlife in the outback – I personally thought it was incredibly degrading.

The idea that you shouldn’t show the negative impacts of hard drug use is such a comically white knight perspective I am not sure whether it makes me want to laugh or cry. I did not take that oxycodone because I have seen, with my own eyes, what happens to people who are addicted to opioids. The most recent wave of fentanyl laced with xylazine can result in open sores and losing phalanges. I do not believe those people are monsters (does anyone?) but it is grotesque what drug addiction can do to people.

I suggest to read Vance’s discussion of his life growing up, over any sociologist, because of this. When what we have to offer is “some people think people who take drugs are monsters” and “you should take nice pictures of them”, people are well justified to ignore academics as out of touch and absurd.

Lament

If I had the chance to sit down with Vance, one thing I would ask him is his choice of elegy in the title of his book. I name my blogpost lament. I have moved on with my life, my work focuses on working with police departments on public safety. This work entirely focuses on urban areas.

I do not think anything my research relates to could, even at the margins, help materially improve the lives of people I grew up with. There are things I think could marginally improve individuals outcomes, such as getting better advice about colleges. But there is nothing reasonable to be done to prevent traffic accidents or improve farm safety. I mean you could attempt stricter safety regulations but realistically enforcing them is another matter.

Vance in his book does not really talk about politics, but towards the end gives some examples of policies he thinks could on the margins help – such as restricting the number of section 8 vouchers in a neighborhood (to prevent segregation). He is right that you cannot magically subsidize factory jobs and all will be well – it will not. These jobs, as I said above, suck.

I view the current state of rural America, as I experienced it, via Murray’s Bell Curve. Specifically the idea of brain drain, and more broadly intellectual segregation. One of Murray’s theses was that, historically, rural communities had a mix of intelligent (and not so intelligent) individuals. Gradually over time, the world has become more globalized, so it is easier to move from the farm to industrialized areas.

This results in intelligent people – those who can go to college and keep a job – to move away. What is left over is a higher proportion of the types of people Vance more focuses on in his book – individuals with persistent life problems. Criminologists will recognize this process as fundamentally the same with urban blight areas described by the Chicago school of crime. Vance focuses on the culture that is the end result of this demographic process – the only people who don’t move away are the ones who live hard lives that Vance describes in his book.

Automation is the long term progression of farming in America. Farming in rural areas will eventually be just the minimal number of humans needed to oversee the automated machinery. I am not sure the town I grew up in will exist in 100 years.

And this to me is not a bad thing. I left to pursue opportunities that were not available to me if I stayed in Bradford county. I am not sad that I do not need to sling square bales of hay. My suggestion, to help give better advice to students about pursuing college and careers, will only hasten the demise of rural areas, not save them. This is the lament.

Politics aside, I found Vance’s biography of his life growing up worth reading. If you find my stories interesting, I suspect you will find his as well.

Aoristic analysis, ebooks vs paperback, website footer design, and social media

For a few minor updates, I have created a new Twitter/X account to advertise Crime De-Coder. I do not know if there is some setting that people ignore all unverified accounts, but would appreciate the follow and reshare if you are still on the platform.

I also have an account on LinkedIn, and sometimes comment on the Crime Analysis Reddit.

I try to share cool data visualizations and technical posts. I know LinkedIn in particular can be quite vapid self-help guru type advice, which I avoid. I know being more technical limits the audience but that is ok. So appreciate the follow if you are on those platforms and resharing the work.

Ebooks vs Paperbacks

Part of the reason to start X account back up is to just try more advertising. I have sold not quite 80 to date (including pre-sales). My baseline goal was 100.

For the not pre-sales, I have sold 35% ebooks and 65% paperbacks. So spending some time to distribute your book paperback seems to me to be worth it.

Again feel like most academics who publish technical books self-publishing is a very good idea. So read the above linked post about some of the logistics of self-publishing.

Aoristic analysis in python

On the CRIME De-Coder blog, check out my post on Aoristic analysis. It has links to python code on github for those who just want the end result. It has several methods though to do hour of day and hour by day of week breakdowns. With the ability to do it by categories in data. And post hoc generate a few graphs. I like the line graphs the best:

But the more common heatmap I can understand why people like it

Website Design

I have a few minor website design updates. The homepage is more svelte. Wife suggested that it should be easier to see what I do right when you are on homepage, so put the jumbotron box at the bottom and the services tiles (with no pictures) at the top.

It does not look bad on mobile either (I only recently figured out that in Chrome’s DevTools they have a button to do turn on mobile view, very helpful!)

Final part is that I made a footer for my pages:

I am not real happy with this. One of the things you notice when you start doing web-design is everyone’s web-page looks the same. There are some basic templates for WordPress or Wix (and probably other CMS generators). Here is Supabases’s footer for example:

And now that I have shown you, you will see quite a few websites have that design. So I did the svg links to social media, but I may change that. (And go with no footer again, there is not a real obvious need for it.) So open to suggestions!

In intentionally made many of the decisions for the way the Crime De-Coder site looks not only to make it usable but to make it at least somewhat different. Avoid super long scrolls, sticky header (that still works quite well on phones). The header is quite dense with many sub-pages (I like it though).

I think alot of public sector agencies that are doing data dashboards now do not look very nice. Many are just iframed Tableau dashboards. If you want help with those data visualizations embedded in a more organic way in your site, that is something Crime De-Coder can help with.

Reducing folium map sizes

Recently for a crimede-coder project I have been building out a custom library to make nice leaflet maps using the python folium library. See the example I have posted on my website. Below is a screenshot:

This map ended up having around 3000 elements in it, and was a total of 8mb. 8mb is not crazy to put on a website, but is at the stage where you can actually notice latency when first rendering the map.

Looking at the rendered html code though it was verbose in a few ways for every element. One is that lat/lon are in crazy precision by default, e.g. [-78.83229390597961, 35.94592660794455]. So a single polygon can have many of those. Six digits of precision for lat/lon is still under 1 meter of precision, which is plenty sufficient for my mapping applications. So you can reduce 8+ characters per lat/lon and not really make a difference to the map (you can technically have invalid polygons doing this, but this is really pedantic and should be fine).

A second part of the rendered folium html map for every object is given a full uuid, e.g. geo_json_a19eff2648beb3d74760dc0ddb58a73d.addTo(feature_group_2e2c6295a3a1c7d4c8d57d001c782482);. This again is not necessary. I end up reducing the 32 length uuids to the first 8 alphanumeric characters.

A final part is that the javascript is not minified – it has quite a bit of extra lines/spaces that are not needed. So here are my notes on using python code to take care of some of those pieces.

To clean up the precision for geometry objects, I do something like this.

import re

# geo is the geopandas dataframe
redg = geo.geometry.set_precision(10**-6).to_json()
# redg still has floats, below regex clips values
rs = r'(\d{2}\.|-\d{2}\.)(\d{6})(\d+)'
re.sub(rs,r'\1\2',redg)

As most of my functions add the geojson objects to the map one at a time (for custom actions/colors), this is sufficient to deal with that step (for markers, can round lat/lon directly). It may make more sense for the set precision to be 10**-5 and then clip the regex. (For these regex’s I am showing there is some risk they will replace something they should not, I think it will be pretty safe though.)

Then to clean up the UUID’s and extra whitespace, what I do is render the final HTML and then use regex’s:

# fol is the folium object
html = fol.get_root()
res = html.script.get_root().render()
# replace UUID with first 8
ru = r'([0-9a-f]{8})[0-9a-f]{4}[0-9a-f]{4}[0-9a-f]{4}[0-9a-f]{12}'
res = re.sub(ru,r'\1',res)
# clean up whitespace
rl = []
for s in res.split('\n'):
    ss = s.strip()
    if len(ss) > 0:
        rl.append(ss)
rlc = '\n'.join(rl)

There is probably a smarter way to do this directly with the folium object for the UUID’s. For whitespace though it would need to be after the HTML is written. You want to be careful with the cleaning up the whitespace step – it is possible you wanted blank lines in say a leaflet popup or tooltip. But for my purposes this is not really necessary.

Doing these two steps in the Durham map reduces the size of the rendered HTML from 8mb to 4mb. So reduced the size of the file by around 4 million characters! The savings will be even higher for maps with more elements.

One last part is my map has redundant svg inserted for the map markers. I may be able to use css to insert the svg, e.g. something like in css .mysvg {background-image: url("vector.svg");}, and then in the python code for the marker svg insert <div class="mysvg"></div>. For dense point maps this will also save quite a few characters. Or you could add in javascript to insert the svg as well (although that would be a bit sluggish I think relative to the css approach, although sluggish after first render if the markers are turned off).

I have not done this yet, as I need to tinker with getting the background svg to look how I want, but could save another 200-300 characters per marker icon. So will save a megabyte in the map for every 3000-5000 markers I am guessing.

The main reason I post webdemo’s on the crimede-coder site is that there a quite a few grifters in the tech space. Not just for data analysis, but for front-end development as well. I post stuff like that so you can go and actually see the work I do and its quality. There are quite a few people now claiming to be “data viz experts” who just embed mediocre Tableau or PowerBI apps. These apps in particular tend to produce very bad maps, so here you can see what I think a good map should look like.

If you want to check out all the interactions in the map, I posted a YouTube video walking through them

Durham hotspot map walkthrough of interactions

LinkedIn posting and link promotion: impression vs reality

For folks who are interested in following my work, my advice is either email or RSS. This site you should see ‘follow blog via email’ and the RSS link on the right hand side. I sometimes post a note here on crimede-coder stuff, but not always, so just do the same (RSS, or use if-this-than-that service to turn RSS into email) on that site if you want to keep abreast of all my posts.

Another way to follow my work though is on LinkedIn. So feel free to connect with me or follow my content:

I post short form blogs/reactions on occasion (plus share my other posts/work). Social media promoting your work is often cringy, but I try to post informative and technical content (and not totally vapid self-help stuff). And I write things for people to view them, so I think it is important to promote my work.

One of the most recent things I have heard a few influencers mention how embedding links directly in LinkedIn posts they think de-promotes their work. See this discussion on HackerNews, or this person’s advice for two examples.

I formed a few opinions based on my regular postings over the past year+, but impressions of things over extended periods can often be wrong. So I actually downloaded the data to see! In terms of the thing about links and being de-promoted, I don’t see that in my data at all – this is a table of impressions broken down by the domain I linked to (for domains with at least 2+ posts over the prior year):

I did notice however two different domains – youtube and newsobserver (the Raleigh newspaper) tend to not have much engagement. So it may be certain domains are not as promoted. It is of course possible that particular content was not popular (I thought my crim observations on the Mark Rober glitterbombs would be more popular, but maybe not). But I think this is a large enough sample to at least give a good hint that they are not promoted in the same way my other links are. My no URL posts have slightly less engagement than my posts to this blog or the crimede-coder site, so overall the idea that links are penalized doesn’t appear to me to be true without more conditional statements.

Data is important, as again I think impressions can be bad for things that repeatedly happen over a long period of time. So offhand I though Tue/Thu I had less engagement, so stopped posting on those days. What does the data say?

| Day | Avg Impressions | Number Posts |
----------------------------------------
| Sun |         1,860   |         32   |
| Mon |         1,370   |         44   |
| Tue |         1,220   |         35   |
| Wed |         1,273   |         41   |
| Thu |         1,170   |         34   |
| Fri |         1,039   |         39   |
| Sat |         1,602   |         38   |

The data says Sun/Sat have higher impressions, and days of the week are lower. If anything Friday is the low day, not Tuesday/Thursday.

I have had other examples of practitioners argue with me in crime analysis or academic circles in my career that strike me as similar. In that perceptions (that people strongly believed in), did not align with actual data. So I just don’t think this idea of ‘taking the average of impressions over posts over the past year’ is something that you can really know just based on passive observation. Your perceptions are likely to be dominated by a few examples, which may be off the mark. Ditto for knowing how much crime happens at a particular location, or knowing how much different things impact survival rates for gunshots.

It is definately possible that my small page experience (currently at a few over 2700 followers on LinkedIn) is not the same as the large influencers. But without looking at actual data, I don’t trust peoples instincts on aggregate metrics all that much.

Another meta LinkedIn tip (I received from Rob Fornango) is to post tall images, so when people are scrolling your content stays on the screen longer. Here is an example post from Rob’s

It is hard for me to test this though, the links on LinkedIn sometimes expand the link to bigger images and sometimes not (and sometimes I edit the image it displays as well). And I think after a while they turn them into tiny images as well. Someone tell the folks on LinkedIn to allow us to use markdown!

So I mean I could spend a full time job tinkering, but looking at the data I have at hand I don’t plan on changing much. Just posting links to my work, and having an occasional comment as well if I think it will be of interest to more people than myself. Content over micro optimization that is (since the algorithm could change tomorrow anyway).

One of the things I have debated on is buying adverts to promote my python book. I think they are just on the cusp of a net loss though given clickthrough rates and margins on my book. So for example, LinkedIn estimates if I spend $140 to promote a post, I will get 23-99 clicks. My buy rate on the site is around 5%, so that would generate 1-5 book sales. My margins are not that high on a sale, so I would not make money on that.

I have been wondering if I posted direct adverts on Reddit for the book to the learn python forum how that would go. But I think it would be much of the same as LinkedIn (too low of clickthrough to make it worth it). But if I do those tests in the future will write up a blog post on my experience!


LinkedIn I can only find how to download my stats on the company crimede-coder page, not my personal page. Here is the script I used to convert the LinkedIn short urls back to the original domains I linked, plus the analysis:

'''
python Code to parse the domains from my
crimede-coder linkedin posts
run on 7/24/2024, so only has posts
from that date through the prior year
'''

import requests
import traceback
import pandas as pd
import time
from urllib.parse import urlparse

errors = {}

def get_link(url):
    time.sleep(2)
    try:
        res = requests.get(url)
    except Exception:
        er = traceback.format_exc()
        print(f'Error message is \n\n{er}')
        return ''
    if res.ok:
        it = res.text.split()
        it = [i for i in it if i[:4] == 'href']
        rl = it[3]
    else:
        print(f'Not ok, {url}, response: {r2.reason}')
        errors[url] = res
        return ''
    return rl[6:].replace('/">','').replace('">','')

# more often than not, linkedin converts the link in the post
# to a lnkd.in short url
def get_refer(txt):
    rs = txt.split()
    rs = [i for i in rs if i[:8] == 'https://']
    if rs:
        url = rs[0]  # if more than one link, only grabs the first
        if url[:15] == 'https://lnkd.in':
            return get_link(url)
        else:
            return url
    else:
        return ''


# this is data exported from LinkedIn on my Crime De-Coder page only goes back one year
df = pd.read_excel('crime-de-coder_content_1721834275879.xls',sheet_name='All posts',header=1)

# only need to keep a few columns
keep_cols = ['Post title','Post link','Created date','Impressions','Clicks','Likes','Comments','Reposts']
df = df[keep_cols].copy()

df['url'] = df['Post title'].apply(get_refer)

def domain(url):
    if url == '':
        return 'NO URL'
    else:
        pu = urlparse(url)
        return pu.netloc

df['domain'] = df['url'].apply(domain)

# caching out file, so do not need to reget url info
df.to_csv('ParseInfo.csv',index=False)

# Can aggregate to domain
agg_stats = df.groupby('domain',as_index=False)['Impressions'].describe()
agg_stats.sort_values(by=['count','mean'],ascending=False,ignore_index=True,inplace=True)
count_cols = list(agg_stats)[1:]
agg_stats[count_cols] = agg_stats[count_cols].fillna(0).astype(int)

# This is a nice way to print/view the results in terminal
print('\n\n' + agg_stats.head(22).to_markdown() + '\n\n')

Using GenAI to describe charts for reports

One of the ideas that has come up with the recent GenAI craze is to use these tools to conduct end-to-end data analysis. So you feed it a dataset + a question and out pops an analysis. Mike Zidar has notes on using a Google tool to do this:

I am not worried about these tools usurping crime analysts though. The reason is that the vast majority of data analysis (crime or business or whatever) is very superficial. The hardest part is not generating the chart, it is knowing what chart to generate, and how it will be used by real people to make real decisions. Often in crime analysis you get the ambiguous “well this will help allocate resources” – when in reality your chart can in no way help dictate any realistic decision a department is going to make.

If you cut out analysts, and just have front line individuals asking google “do crime analysis”, it will be hopelessly superficial, and the front line will just ignore it altogether.

I however do think that GenAI has the ability to become power-tools for super-users. That is, someone who does know what they want to calculate, but uses the computer to help them get that information faster. Not dissimilar to how auto-complete while texting helps you type faster. And here is one use case I have been thinking about – so analysts spend a ton of time automating different products, such as monthly CompStat reports. The reports should have tables/graphs in them, like this chart of gun crimes in DC for example:

Now, most reports you want to also have a plain text summary of what is going on in the chart. Currently when auto-generating reports, it is difficult to mix that plain text in and not make it very superficial using a rule based system. The newest round of many of the GenAI tools lets you upload an image and ask it questions about the image. So this is still very open ended, but has many more guiderails than simply telling the machine “go brr and generate analysis”. You have already decided the chart, you are just asking for a nice description of the chart to fill in your already pre-made metrics.

I did this with ChatGPT, Google’s Gemini, and Claude to see how it does with the gun crime chart, the below Raleigh weekly chart:

And in some cases a more tame monthly time series chart:

Because these models are changing all the time, keep in mind that when you read this the newest models may do even better than what I show here (and see the end section on how you may prompt engineer this to produce better results as well). That said lets check out the results!

ChatGPT

For ChatGPT I used the GPT-4o model, I first just asked about the DC chart “Describe the patterns in this chart”. ChatGPT as it is known, is quite verbose:

I then asked ChatGPT to keep it to two to three sentences, and I think it did very well.

Ok, now to the Raleigh chart with error bars:

So this is OK, it spotted the recent increasing trend. “Frequently exceeded the average of the prior 8 weeks (gray shaded area)” is wrong, there are only 2! But I think the last sentence about notable recent spikes is good.

I then gave it the Durham chart that had the anomalies in early 2019/2020:

And I again think this is quite reasonable. I mean an analyst should probably say “these must be reporting idiosyncrasies”, but I don’t think this is not so bad a description as to be misleading.

All in all very happy with the results for ChatGPT here – these charts are not typical line and bar charts, and ChatGPT interpreted each quite well. At least the description is not so bad that if I did these directly in an automated report it would be embarrassing.

Google

For all of these examples I am using the free tools (that typically have limits that I run out by just these two queries). I did this on 5/22 for Google (which I think is Gemini-1.5, I am not 100% sure). So here is the DC gun crime seasonal chart for Google with the prompt Describe the trends in this chart:

This is very wrong in multiple places. It did not do any better with the Raleigh chart:

There is not strong seasonality here, and it includes some filler “important to note” that is not actually important to note. After this I gave it a more tame monthly crime counts of Robberies in Houston chart to see if Google could redeem itself:

And it flopped pretty hard again. Maybe most charts are increasing, so the model is biased to say increasing (I don’t know).

So again this is Google’s free version, and so the paid may be better (or recent updates may be better). But this isn’t even close to make me want to prompt engineer further.

Claude

I did the tests with Sonnet 3.5 (so around a month after the tests with Google/ChatGPT). I used the shorter 2/3 sentences prompt.

I like this description even slightly better than ChatGPT. How about the Raleigh MV thefts chart:

Similar to ChatGPT it is not quite right but in very subtle ways. It does catch the upward trend. It is wrong in terms of data ends before 2024, and the gray area indicating greater variability is technically true but I would not describe it as noticeable. So again not embarrassingly wrong (like Google), but not quite right either.

How about the anomaly Durham chart:

Very similar to ChatGPT, which I think is again OK.

Prompt Engineering Ideas

So the idea behind prompt engineering is you can ask “Describe this chart” or you can ask “Describe this chart in two to three sentences” and it changes the results (in any of these tools). Subsequently a big part of this is figuring out the prompts that give the most reasonable results. Prompts in these GenAI tools when submitting images have two parts, the image and the text. Do not take me as an expert by any means, but for other analysts here are my guesses as to how to prompt engineer this to maybe return better results.

For the text part, one thing you may do for auto reports is to give examples of the text you want. So for example, you could have a prompt that is “Please fill in the blanks: The chart shows _____ trends over time.”. So provide more guidance as to the structure of how you want to the response to look. Or you could do: “Here is example description 1: …., description 2: ….”. This is how RAG applications work, but with a static report you can give static exemplars, they don’t really need to be dynamically looked up from prior reports.

For the image part of the prompt, in an auto-application you may submit a different image than is actually shown in the report. So for example I may have the X axis for monthly crimes be labeled with the actual months (instead of numbers). Putting all the months in smaller font I bet the GenAI tools will still read it just fine, even if I don’t want it to look like that in the final report.

And I probably shouldn’t include logos (since they are immaterial and just cause extra info that may distract the description), and the text footers. I also think making my legends more descriptive may help guide the tools interpretation. I may remove the title text all together and place the relevant info in the prompt “Here is a chart of Robberies in Houston from …. to … Please describe the chart, including any long term trends, or anomalous spikes (high or low) in any month.” The text prompt may keep the tools on track a bit more with the specific details, but still allow them leeway to interpret the chart without being too rigid.

For the error bar chart, you could insert into the prompt explicit dates they were outside, e.g. “weeks a and b were high, make sure to mention that”. So you could have a mix of explicit anomaly detection, insert those anomalies into the prompt, to just keep the results on track.

It would still be a lot of work to automate a report with such plain text language, but I think it could be a quite reasonable iterative workflow. So you generate the report in a format you can edit, like Word, review it. And then in subsequent reports try to tweak the parameters a wee bit to produce better outputs.

Wake LP talk on LPRs and javascript hacks in WooCommerce

For some Crime De-Coder news, I will be giving a tech talk on automated license plate readers for the Wake County Libertarian Party on Wednesday July 17th in Raleigh.

See my slides on the CRIME De-Coder website to get a preview of what I will be talking about.

This post will just be a quick one that is hopefully helpful to others. So I use WooCommerce + LuLu to distribute shipping for my print book. For those who have purchased a copy thank you! Many of the paperback purchases will be arriving at your homes very soon. There have been two hiccups with my store website for individuals.

Problem 1 is an error nonce is invalid popping up after trying to add the book to the cart. It is difficult to replicate and is an underlying cache error with WordPress as best I can tell. My advice to fix is go to the individual book page directly to add the book to your cart (paperback, ebook). It seems to mostly happen to me when I am on the main shop page and add directly to cart. If the problem persists for you let me know.

The second problem is that for the print book, to do shipping on LuLu’s end you need to input a phone number. As far as I can tell, most website advice in WooCommerce suggest to pay for an additional plug in to edit this option. I was able to use javascript (and the WPCode free plugin) though to force the phone number to be filled in for free though. Sharing here as I hope it will be helpful to others.

Check out the onload function first, that sets the attribute for either billing-phone or shipping-phone to be required. If you are fast and are looking at the exact right spot of the checkout page, you would be able to see this change from “Phone (Optional)” to “Phone”.

The change_label event listener is to modify the error message when you don’t put in a phone number (by default it confusingly says “Please enter a valid phone number (optional)”. So that part is a bit hacky with attaching the event listener to the entire webpage, but unless you are trying to purchase from a Commodore 64 it should be fine.

<script>
// This script forces the billing/shipping
// phone number to be filled in and not optional
// Andy Wheeler, done via WPCode Extension in Footer
function change_label(){
    var xl = document.getElementById("billing-phone");
    if (xl){
        var ll = xl.nextSibling;
        var nd = ll.nextSibling;
        if (nd) {
            if (nd.getAttribute('role') == 'alert') {
                nd.firstChild.innerText = "Please enter a valid phone number"
            };
        };
    };
    var xs = document.getElementById("shipping-phone");
    if (xs){
        var ls = xs.nextSibling;
        var ns = ls.nextSibling;
        if (ns) {
            if (ns.getAttribute('role') == 'alert') {
                ns.firstChild.innerText = "Please enter a valid phone number"
            };
        };
    };
};

// So click is not working when people
// just use tabs/keyboard select
// not sure how to fix that, but just results in a
// bad red note that says "optional" (but still need
// to fill in
document.addEventListener('click',change_label);

window.onload = function() {
    var x = document.getElementById("billing-phone");
    if (x) {
        var lab = x.nextSibling;
        lab.innerText = "Phone";
        x.setAttribute('aria-label','Phone')
        x.setAttribute('required','')
        // These don't seem to work unfortunately!
        //x.addEventListener("change",change_label);
        //x.setAttribute("onChange","change_label();")
    };
    var x2 = document.getElementById("shipping-phone");
    if (x2) {
        var lab2 = x2.nextSibling;
        lab2.innerText = "Phone";
        x2.setAttribute('aria-label','Phone')
        x2.setAttribute('required','')
    };
};
</script>

Because there is no phone verification, you could technically put in a fake number for these FYI and no one would know. (I have a google voice number I use for instances in which I don’t really want to give out personal.)

Thanks again for those who have purchased a copy – appreciate the support.

Some notes on self-publishing a tech book

So my book, Data Science for Crime Analysis with Python, is finally out for purchase on my Crime De-Coder website. Folks anywhere in the world can purchase a paperback or epub copy of the book. You can see this post on Crime De-Coder for a preview of the first two chapters, but I wanted to share some of my notes on self publishing. It was some work, but in retrospect it was worth it. Prior books I have been involved with (Wheeler 2017; Wheeler et al. 2021) plus my peer review experience I knew I did not need help copy-editing, so the notes are mostly about creating the physical book and logistics of selling it.

Academics may wish to go with a publisher for prestige reasons (I get it, I was once a professor as well). But it is quite nice once you have done the legwork to publish it yourself. You have control of pricing, and if you want to make money you can, or have it cheap/free for students.

Here I will detail some of the set up of compiling the book, and then the bit of work to distribute it.

Compiling the documents

So the way I compiled the book is via Quarto. I posted my config notes on how to get the book contents to look how I wanted on GitHub. Quarto is meant to run code at the same time (so works nicely for a learning to code book). But even if I just wanted a more typical science/tech book with text/images/equations, I would personally use Quarto since I am familiar with the set up at this point. (If you do not need to run dynamic code you could do it in Pandoc directly, not sure if there is a way to translate a Quarto yaml config to the equivalent Pandoc code it turns into.)

One thing that I think will interest many individuals – you write in plain text markdown. So my writing looks like:

# Chapter Heading

blah, blah blah

## Subheading

Cool stuff here ....

In a series of text files for each chapter of the book. And then I tell Quarto quarter render, and it turns my writing in those text files into both an Epub and a PDF (and other formats if you cared, such as word or html). You can set up the configuration for the book to be different for the different formats (for example I use different fonts in the PDF vs the epub, nice fonts in one look quite bad in the other). See the _quarto.yml file for the set up, in particular config options that are different for both PDF and Epub.

One thing is that ebooks are hard to format nicely – if I had a book I wanted to redo to be an epub, I would translate it to markdown. There are services online that will translate, they will do a bad job though with scientific texts with many figures (and surely will not help you choose nice fonts). So just learn markdown to translate. Folks who write in one format and save to the other (either Epub/HTML to PDF, or PDF to Epub/HTML) are doing it wrong and the translated format will look very bad. Most advice online is for people who have just books with just text, so science people with figures (and footnotes, citations, hyperlinks, equations, etc.) it is almost all bad advice.

So even for qualitative people, learning how to write in markdown to self-publish is a good skill to learn in my opinion.

Setting up the online store

For awhile I have been confused how SaaS companies offer payment plans. (Many websites just seem to copy from generic node templates.) Looking at the Stripe API it just seems over the top for me to script up all of my own solution to integrate Stripe directly. If I wanted to do a subscription I may need to figure that out, but it ended up being for my Hostinger website I can set up a sub-page that is WordPress (even though the entire website is not), and turn on WooCommerce for that sub-page.

WooCommerce ends up being easy, and you can set up the store to host web-assets to download on demand (so when you purchase it generates a unique URL that obfuscates where the digital asset is saved). No programming involved to set up my webstore, it was all just point and click to set things up one time and not that much work in the end.

I am not sure about setting up any DRM for the epub (so in reality people will purchase epub and share it illegally). I don’t know of a way to prevent this without using Amazon+Kindle to distribute the book. But the print book should be OK. (If there were a way for me to donate a single epub copy to all libraries in the US I would totally do that.)

I originally planned on having it on Amazon, but the low margins on both plus the formatting of their idiosyncratic kindle book format (as far as I can tell, I cannot really choose my fonts) made me decide against doing either the print or ebook on Amazon.

Print on Demand using LuLu

For print on demand, I use LuLu.com. They have a nice feature to integrate with WooCommerce, the only thing I wish shipping was dynamically calculated. (I need to make a flat shipping rate for different areas around the globe the way it is set up now, slightly annoying and will change the profit margins depending on area.)

LuLu is a few more dollars to print than Amazon, but it is worth it for my circumstance I believe. Now if I had a book I expected to get many “random Amazon search buys” I could see wanting it on Amazon. I expect more sales will be via personal advertising (like here on the blog, social media, or other crime analyst events). My Crime De-Coder site (and this blog) will likely be quite high in google searches for some of the keywords fairly quickly, so who knows, maybe just having on personal site is just as many sales.

LuLu does has an option to turn on distribution to other wholesalers (like Barnes & Noble and Amazon) – have not turned that on but maybe I will in the future.

LuLu has a pricing calculator to see how much to print on their website. Paperback and basically the cheapest color option for letter sized paper (which is quite large) is just over $17 for my 310 page book (Amazon was just over $15). For folks if you are less image heavy and more text, you could get away with a smaller size book (and maybe black/white) and I suspect will be much cheaper. LuLu’s printing of this book is higher quality compared to Amazon as well (better printing of the colors and nicer stock for the paperback cover).

Another nice thing about print on demand is I can go in and edit/update the book as I see fit. No need to worry about new versions. Not sure what that exactly means for citing the work (I could always go and change it), you can’t have a static version of record and an easy way to update at the same time.

Other Random Book Stuff

I purchased ISBNs on Bowker, something like 10 ISBNs for $200. (You want a unique ISBN for each type of the book, so you may want three in the end if you have epub/paperback/hardback.) Amazon and LuLu though have options to have them give you an ISBN though, so that may have not been necessary. I set the imprint to be my LLC though in Bowker, so CRIME De-Coder is the publisher.

You don’t technically need an ISBN at all, but it is a simple thing, and there may be ways for me to donate to libraries in the future. (If a University picks it up as a class text, I have been at places you need at least one copy for rent at the Uni library.)

I have not created an index – I may have a go at feeding my book through LLMs and seeing if I can auto-generate a nice index. (I just need a list of key words, after that can just go and find-replace the relevent text in the book to fill in so it auto-compiles an index.) I am not sure that is really necessary though for a how-to book, you should just look at the table of contents to see the individual (fairly small) sections. For epub you can just doing a direct text search, so not sure if people use an index at all in epubs.

Personal Goals

So I debated on releasing the book open source, I do want to try and see if I can make some money though. I don’t have this expectation, but there is potential to get some “data science” spillover, and if that is the case sales could in theory be quite high. (I was surprised in searching the “data science python” market on Amazon, it is definitely not saturated.) Personally I will consider at least 100 sales to be my floor for success. That is if I can sell at least 100 copies, I will consider writing more books. If I can’t sell 100 copies I have a hard time justifying the effort – it would just be too few of people buying the book to have the types of positive spillovers I want.

To make back money relative to the amount of work I put in, I would need more than 1000 sales (which I think is unrealistic). I think 500 sales is about best case, guesstimating the size of the crime analyst community that may be interested plus some additional sales for grad students. 1000 sales it would need to be in the multiple professors using it for a class book over several years. (Which if you are a professor and interested in this for a class let me know, I will give your class a discount.)

Another common way for individuals to make money off of books is not for sales, but to have training’s oriented around the book. I am hoping to do more of that for crime analysts directly in the future, but those opportunities I presume will be correlated with total sales.

I do enjoy writing, but I am busy, so cannot just say “I am going to drop 200 hours writing a book”. I would like to write additional python topics oriented towards crime analysts/criminology grad students like:

  • GIS analysis in python
  • Regression
  • Machine Learning & Optimization
  • Statistics for Crime Analysis
  • More advanced project management in python

Having figured out much of this grunt work definitely makes me more motivated, but ultimately in the end need to have a certain level of sales to justify the effort. So please if you like the blog pick up a copy and tell a friend you like my work!

References

My word template for Quarto

I have posted on Github my notes on creating a word template to use with quarto. And since Quarto is just feeding into pandoc, those who are just using pandoc (so not doing intermediate computations), should maybe find that template worthwhile as well.

So first, why word? Quarto by default looks pretty nice for HTML. That is fine for them to prioritize that, but the majority of reports I want to use quarto for HTML is not the best format. Many times I want a report that can be emailed in PDF and/or printed. And sometimes I (or my clients) want a semi-automated report that can be edited after the fact. In those cases word is a good choice.

Editing LaTeX is too hard, and I am pretty happy with the this template for small reports. I will be sharing my notes on writing my python book in Quarto soonish, but for now wanted to share how I created a word template.

Note some of the items may seem gratuitous (why so many CRIME De-Coder logos?). Part of those are just notes though (like how to insert an image after your author name, I have done this to insert my signature in reports for example). The qmd file has most of the things I am interested in doing in documents, such as how to format markdown tables in python, doings sections/footnotes, references, table/figure captions, etc.

I do like my logo though in the header (it is hyperlinked even, so in subsequent PDFs if you click the logo it will go to my website), and the footer page numbers I commonly need in reports as well. And my title page and TOC do not look bad as well IMO. I am not one to discuss fonts, but I like small caps for titles and the Verdana font is nice to make it look somewhat different.

Creating the Template

So first, you can do from the command line:

quarto pandoc -o custom-reference-doc.docx --print-default-data-file reference.docx

From there, you should edit that reference.docx file to get what you want. So for example, if you want to change the font used for code snippets, in Word you can open up Styles, and on the right hand side select different elements and edit them:

Here for example to change the font for code snippets, you modify the HTML code style (I like Consolas):

There ended up being a ton of things I edited, I did not keep a list. Offhand you will want to modify the Title, Headings 1 & 2, First Paragraph, Body Text. And then you can edit things like the page numbers and header/footer.

So when rendering a document, you can sometimes click on the element in the rendered document and figure out what style it inherits from. Here for example you can see in the test.docx file that the quote section uses the “Block Text” style:

This does not always work though, and it can take some digging/experimentation in the original template file to get the right style modifier. (If you are having a real hard problem, convert the word document format to .zip, and dig into the XML documents. You can see the style formats in inherits from in the XML tree.) It doesn’t work for the code segments for example. Do not render a document and edit the style in that document, only edit the original --print-default-data-file reference.docx that was generated from the command line to update your template.

I have placed a few notes in the readme on Github, but one of my main things was making tables look nice. So this plays nicely with markdown tables, which I can use python to render directly. Here is an example of spreading tables across multiple pages.

One thing to note though is that this has limits – different styles are interrelated, so sometimes I would change one and it would propagate errors to different elements. (I can’t figure out how to change the default bullets to squares instead of circles for example without having bullets in places they should not be in tables – try to figure that one out. I also cannot figure out how to change the default font in tables, I would use monospace, without changing the font for other text elements in normal blocks.) So this template was the best I could figure without making other parts broken.

I have a few notes in the qmd file as well, showing how to use different aspects of markdown, as well as some sneaky things to do extra stuff (like formatting fourth level headings to produce a page break, I do not think I will need that deep of headings).

Even for those not using Quarto for computational workflows, writing in markdown is a really useful skill. You write in plain text, and can then have the output in different formats. Even for qualitative folks (or people in industry creating documents), I think many people would be well served by writing content in plain text markdown, and then rendering to whatever output they wanted.