Google Search Console tells you which searches lead readers to your site. Your CMS tells you what you published. What it doesn’t tell you is if those two things overlap — whether readers were searching for something your newsroom didn’t write about.

That question is the whole point of this project. I built a pipeline at the Star Tribune that detects rising search queries, checks whether we published a story on the topic, and measures the gap if we did.

Here’s how it works, what I learned building it, and how to run it yourself.

The problem

On any given day, thousands of Google searches send readers to startribune.com. Some of those queries spike because something happened in the news — a Timberwolves playoff run, a wildfire near Ely, a storm rolling into the Twin Cities. When impressions on a query triple their normal level for two days in a row, that’s a signal: readers want coverage.

The question is whether we provided it. And answering it required a pipeline like this one.

Trend detection

The first step is finding the spikes. For each search query in Google Search Console, I compute a 14-day rolling average of daily impressions — what “normal” looks like for that query. Then I compare each day’s actual impressions to that baseline.

If a query hits 3× its baseline for two consecutive days, it’s flagged as a trend onset. That consecutive-days requirement filters out noise: a single day spike is often a data artifact or a one-off news event. Two days in a row suggests real, sustained reader interest. There’s also an impression floor — queries below 50 daily impressions are excluded, so we’re not flagging “trends” where the count goes from 2 to 6.

# Rolling 14-day mean, shifted one day forward (no lookahead bias)
baseline   = pivot.rolling(window=14, min_periods=3).mean().shift(1)
spike_ratio = pivot / baseline.replace(0, float("nan"))
is_spiking  = (spike_ratio >= 3.0) & (pivot >= 50)

The 14-day window captures two full weekly cycles, which matters because search volume has weekly patterns — news queries tend to spike on weekdays and quiet on weekends. A shorter window would mistake Tuesday for trending.

Matching trends to articles

Once I have a list of trending queries, I need to check whether we published something. The naive approach — exact string matching — fails immediately. A reader searching “Wolves” won’t match an article with “Timberwolves” in the headline. “Ant” won’t match “Anthony Edwards.” Jaccard similarity on keywords gets you part of the way there. (Quick refresher: Jaccard scores how much two phrases overlap, from 0 to 1 — the words they share divided by the total unique words across both.) But it has a precision problem: lower the threshold to catch more matches and you start pulling in articles that share a word but not a topic.

I ended up with a two-stage approach.

Stage 1: Jaccard pre-filter. For each trending query, I tokenize the query and every article headline + URL slug from a three-week window around the trend onset (7 days before through 14 days after). Any article that scores at least 0.05 Jaccard similarity with the query makes the candidate list. This is loose by design — the point is to catch near-misses like “Wolves” articles for a “timberwolves” query, not to make a final decision.

Stage 2: Cortex confirmation. The candidate list goes to Snowflake Cortex COMPLETE (mistral-7b) in a single batched SQL call. The prompt template is parameterized on two env vars — PUBLICATION_NAME and PUBLICATION_LOCATION — so any paper can drop in its own brand:

The {publication_name} is a {publication_location} newspaper.
A reader searched for "{query}".
Did the {publication_name} publish an article specifically about this topic?
Headline: "{headline}"
Section: {section}
Answer yes or no.

Cortex says yes or no. Only yes candidates make it into the results. The whole thing costs under a tenth of a cent per pipeline run.

If Cortex is unavailable, the pipeline falls back to a stricter Jaccard threshold (0.15) so it never crashes — it just becomes slightly less generous.

A case study in false positives

The first version of the Cortex prompt didn’t include the article’s section or anything about the publication. It just asked: “Did this paper publish an article about this topic?”

That broke on “anthony edwards.” A reader searching for Anthony Edwards is almost certainly looking for the Timberwolves point guard. But a News-section piece with “Anthony Fauci” in the headline shares the token “anthony” with the query — Jaccard flagged it as a candidate, and the underspecified prompt confirmed it as a match.

The fix was two lines: add the publication’s geography and add the article’s section. With that context, Cortex correctly rejects a News article about a public health official for a query that almost certainly means a basketball player.

prompt = (
    f"The {PUBLICATION_NAME} is a {PUBLICATION_LOCATION} newspaper. "
    f"A reader searched for \"{query}\". "
    f"Did the {PUBLICATION_NAME} publish an article specifically about this topic?\n"
    f"Headline: \"{headline}\"\n"
    f"Section: {section}\n"
    f"Answer yes or no."
)

The lesson: LLMs can do good binary classification on short prompts, but they need enough context to resolve ambiguity. “Did we write about X?” is underspecified. “Did this regional paper’s Sports section cover the search query X?” is not.

What the output looks like

Each pipeline run writes one row per trending query × matching article into a table called TREND_PUBLISH_GAP. The repo’s synthetic example has 12 trending queries for a fictional local newsroom — 7 covered, 5 not. Here’s a slice:

Each row is a finding the pipeline is asking the newsroom to react to.

• Negative gap hours mean the story published before readers started searching — coverage was ahead of the trend. The reporting was already there when readers came looking.
• Small positive gap hours mean the newsroom moved quickly once the trend started.
• no_coverage means the trend fired but nothing matched, even after Cortex. These are the gaps the pipeline exists to surface.

The categorical GAP_STATUS values are bucketed from GAP_HOURS:

The point of these buckets isn’t to grade a newsroom — ahead_of_trend isn’t always praise (the coverage may have driven the interest, or demand was already building below the spike threshold), and no_coverage isn’t always failure (some trends shouldn’t be covered, and the matcher misses things even with the LLM pass). The buckets are there so dashboards can filter, and so editors can ask the right follow-up question for each row instead of treating all “gaps” the same.

The pipeline also publishes a TRENDING_NOT_COVERED view that filters to just the last fourteen days of uncovered trends — that’s the one you’d look at before an editorial meeting.

Cost

Cortex COMPLETE with mistral-7b runs at fractions of a cent per pipeline run. Cortex is only called for candidates that clear the Jaccard pre-filter, so the call volume stays small. At roughly thirty trending queries per run with a handful of candidates each — daily cadence — the annual Cortex bill lands under $0.40. Warehouse compute is shared with whatever Snowflake usage your team already has, so it adds little to the bill.

Running it yourself

The full code is on GitHub. It ships in two flavors:

• Snowflake — pipeline.py, gsc_pull.py, load_gsc_csv.py. The production-tested path. Uses Snowflake Cortex COMPLETE for the LLM step.
• BigQuery — the same scripts with _bigquery suffixes. A mirror that has not been run end-to-end; useful as a starting point if BQ is your warehouse, with ML.GENERATE_TEXT against a remote Gemini model standing in for Cortex. Expect to debug.

The Snowflake setup, in brief:

git clone https://github.com/julietkelson/trend-tracker
cd trend-tracker
pip install -r requirements.txt
cp .env.example .env
# fill in your Snowflake + GSC credentials and your publication context
python gsc_pull.py --backfill   # or load_gsc_csv.py for the CSV path
python pipeline.py

A few pieces worth calling out:

Publication context is parameterized. PUBLICATION_NAME and PUBLICATION_LOCATION in .env get spliced into the Cortex prompt. That’s the same fix from the false-positives section earlier — instead of hardcoding “Minnesota newspaper,” the matcher learns your paper’s geography and brand at runtime. Generic prompts confuse Anthony Edwards for Anthony Fauci; brand-aware ones don’t.

Your CMS table. Article matching reads from a Snowflake table named in ARC_TABLE. The shipped load_arc() assumes Arc Publishing columns; if your CMS exports look different, that one function is where to adjust. Required fields are documented in the README.

Evergreen sections like Obituaries, Games, and Weather are excluded from the gap analysis by default, since they aren’t news-driven. Override the list in pipeline.py for your sections.

Tests. A pytest suite runs against mocked warehouse connections, so trend detection and matching logic can be verified without credentials.

What’s next

The repo ships both ingestion paths: gsc_pull.py uses a GCP service account against the Search Console API, and load_gsc_csv.py works off the GSC UI export for one-off retrospectives. The piece still missing is scheduling. A production deployment would need key-pair auth (SNOWFLAKE_AUTHENTICATOR=snowflake_jwt) instead of SSO, and a real scheduler — Airflow, cron, whatever your org runs — to fire daily before an editorial meeting.

If you build something similar for your newsroom, reach out and let me know how it goes!

Most newsrooms measure content by clicks. A story does well, it gets a lot of pageviews; it does poorly, it doesn’t. That’s a useful signal for reach. It’s a terrible signal for value, because the stories people read and the stories people pay for are not the same set.

I co-created New Subscriber Paths at the Star Tribune to measure the second one. The goal was a metric the newsroom could trust to answer a question dashboards weren’t really answering: what kinds of journalism actually drive subscriptions?

This is how it works, and how it changed the way News and Analytics talk to each other.

How it works

A path to subscription includes every article a reader viewed in the 30 days before they subscribed. Each new subscriber has one. The metric counts how often each article shows up across all of those paths.

In other words: a story’s path count = the number of users who purchased a subscription in the 30 days after reading the story. That turns out to have a much tighter relationship with subscription conversion than session volume alone.

Once you have paths per story, articles can be tagged with levers — story archetype, topic, location, framing, discovery strategy, photos, video, galleries, content vertical — to look for patterns. The output of that analysis gives a view of what kinds of journalism people are actually willing to pay for, not just what they click on, and how different elements of that journalism affect acquisition. Something few other newsrooms are measuring.

Paths aren’t a whole-picture metric. They don’t measure retention, reach, engagement, or readers who can’t afford to subscribe. They’re a conversion signal, full stop. Discounted paywall offers also create noise unrelated to content, so those periods, among other features, should be controlled for when reporting and modeling expected paths.

How we built it

GA and Piano data give us readership and subscription activity tied to an individual user, which is what makes the path possible: every subscriber’s last 30 days of reading can be reconstructed and counted.

Pipelines running in GitHub Actions derive paths each day. A separate model running in the same workflow predicts a probability curve for how many paths a story should have, given its days since publish, content section, and the paywall intro offer that was in market at the time. The output gets bucketed based on how it aligns to expectations, and that bucketed view is what the newsroom sees. Buckets keep the conversation about whether a story performed against prediction, not whether it had a big number.

The relationship part

This wasn’t only an analytics project — it reshaped how News and Analytics work together. The process was collaborative from the start: editors helped define what they wanted to know, and we built the infrastructure and analysis to answer it. The result was a shared language for talking about content value that both teams actually trusted.

That last word is the one that mattered. Newsrooms are skeptical of analytics for good reasons — most of the metrics they see don’t measure the journalism itself, and they’ve watched data get used as a stick instead of a carrot. Paths gave editors a number that lined up with their instincts about what was working, which meant they used it to inform their content strategy.

Trust didn’t appear on its own. To introduce Paths, we sat down with every department in the newsroom and walked through the metric, what it measured, what it didn’t, where the buckets came from. We trained editors and reporters on the dashboards directly, and held regular office hours so people could come in with a question, or with a pattern they were seeing, and ask whether it was real.

How we iterated

Once Paths was live, the question shifted from does it work? to what is it telling us? The answer kept changing depending on which lever you looked at. So we ran deep dives, one question at a time, whenever curiosity from the newsroom or our team made one worth chasing.

When there was interest in multimedia, we pulled apart how photos, video, and galleries affect paths. When the question turned to graphics, we looked at data visualization and embedded charts the same way. We added topic and archetype modeling so editors could ask sharper questions: when writing about a given topic, do potential subscribers convert more when the story is written in feature style or inverted pyramid?

Each deep dive ended in a presentation and discussion with team leads and editors. The agenda was largely shaped by their questions. Paths wasn’t a one-time finding the newsroom got a deck on and moved past. It was something they kept asking new questions of, and we kept answering them.

Impact

The reporting metric and the way it informs content strategy contributed to:

• ~$1M in annual subscriber revenue added above predictions each month
• +3.1K new subscribers per month above predictions since Paths began informing content strategy
• A News × Analytics working relationship that didn’t exist in the same form before

The numbers are proprietary so the chart below runs without axes, but the shape is the point. How often an article appears across subscription paths is tightly correlated with the number of new subscribers who started reading it within 30 days.

What to read next

The fuller version of this work — including the modeling approach, the lever taxonomy, and the dashboards the newsroom now uses — is in the SRCCON 2025 deck.

If you work at a newsroom and want to compare notes on subscription metrics, reach out.

Before you read this post, understand this – mine is not the voice you should prioritize. While I am outraged by the murder of George Floyd, and I hope you are too, the voices we need to listen to, follow, and promote right now are the voices of the black communities in this country.

I am not writing to recount the events of George Floyd’s murder. As the Washington Post put it:

“If you’re white, you can view the footage of Daniel Shaver’s death, feel some anger and move on. If you’re black, viewing the video of George Floyd’s death is a reminder that the throat under Officer Derek Chauvin’s knee could easily have been your own.”

The reason I am writing is because as a data analyst, it is clear to me not only from anecdote, but from the numbers, that the United States has a problem with police using force against people of color. I’ll show you the data briefly for Minneapolis. Then I’ll offer ways to help that have been promoted by those organizing and involved in the current protests.

White people, don’t just sit on this information. Donate money and supplies, engage with local organizations to find out how to help, educate yourselves. Utilize the privilege that George Floyd didn’t have to prevent this from happening again and to hold those responsible accountable.

The data for the following graphs comes from Open Minneapolis and spans 2008 through 2020. At the time of use, the most recent update was May 29, 2020).

Let’s start with the basic racial breakdown of incidents of police use of force. To be explicitly clear, Race here is the race of the person upon whom the force was inflicted.

Since 2008, 17,933 (62%) incidents of police use of force have been recorded against black people in Minneapolis. White people come in with 7,033 (24%) – almost 11,000 fewer. Black people make up 19% of the Minneapolis population.

For black people, the most common “problem” that brings police and results in these violent incidents is recorded as “Suspicious Person.” The number of “Suspicious Person” reports for white people is less than half of those listed for black people.

The neighborhood with the most records of “suspicious people” that ended in use of force against black people: Downtown West. A neighborhood that is 60% white and only 22% black.

Next, I want you to look at the type of force police use with people of color (POC):

George Floyd is one of the 14,000+ people of color to be counted in the “bodily force” category.

People of color outnumber white people by a long shot. And of all people of color, Minneapolis police seem to use the most force against black people. Specifically black men.

Now that you’re armed with the evidence, here are some things you can do:

• VOTE.
• Call your legislators and demand police accountability
• Protest (safely. COVID is still out there)
• Support the POC in your life and ask them what is helpful. Don’t assume that you know.
• Donate supplies to protesters
• Educate yourself

Where to donate:

• George Floyd Memorial Fund
• Black Visions Collective
• The Bail Project
• Reclaim the Block
• North Star Health Collective
• MIGIZI

This is the tenth in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is not one of those posts.

In order to not go completely insane, I’ve been taking a lot of walks. I try to be careful – I wear a mask and I stay 6 feet from others whenever possible – but there is one problem that I keep running into… (it will be a pun, trust me.)

Usually I can sense it approaching. I hear the shoes hitting the ground and the heavy breathing. And then I turn my head and a runner is breathing into my face with no mask (see, there’s the pun. Running.). Maybe it’s just me but I think they try to get as close to my face as possible so they can push their heavy exhales into my lungs. I know. Gross.

Anyway, budding data analyst that I am, I decided to test my hypothesis. For what felt like hours but was probably 30 minutes, I stood in my pajamas, notebook in hand, spying on the passersby, and tracking the following variables for 101 people:

• Mask (on the mouth and nose)
• Distancing (6ft or it don’t count. Based on my best judgment of if groups were a household, a couple, friends, etc.. Imperfect but most people were not in groups)
• Activity (walking, biking, or running)
• Age (kid, adult, senior. Everyone was an adult though so I didn’t use this)
• Sex (this is based on my perception and may not be accurate in some cases. Gotta note my biases)
• Compliance (mask OR distancing based on CDC guidelines)

Once I had gathered all my data from creeping it was time for the fun part. Yes, I do find this stuff fun.

I started off with some clustering. What does that mean? It means I taught my computer all of the information I gathered. I told it to look at all the cases (1 person = 1 case) and group the similar cases into clusters based on the distance between them. In actuality, it’s more complex and if it makes no sense at all to you, don’t worry. The most important thing to understand here is that we’re looking for patterns in the data.

Based on the data, I found that people broke up into these 5 groups:

• compliant male runners
• compliant walkers
• non compliant walkers
• mostly compliant male bikers
• non compliant female runners

These groups are not perfectly clear cut by variable, but they are pretty close. If you want the numbers for yourself, shoot me an email!

This was a bit surprising to me. I had assumed that men would be less compliant because I have more faith in my gender, but the data called me out on my biases and shows predominantly male groups to be categorized as compliant. But this does not tell me anything about how many people fall into groups and how the numbers themselves play out. So I made some graphs because I love a good visualization.

First and foremost, Male vs Female… Not directly related to my runner hypothesis but still interesting.

Not only is everyone doing poorly with mask wearing and social distancing, but it appears that women are doing worse than men. Not great. Work on it.

But back to my main point – am I crazy, or are runners being selfish, non-compliant disease vectors?

Let’s look at mask usage first:

Unsurprisingly, we have terrible numbers across the board, but the worst is for runners and bikers. This makes sense because it’s harder to breathe wearing a mask if you’re exercising. Plus if you’re far from other people, you can still follow CDC guidelines. Let’s see if people are doing that.

No not really. Bikers in Chicago are in the street where it is easier to be distant so this makes sense. The non-distancing bikers were people trying to pass each other, people biking side by side, and one full grown adult man biking on the sidewalk.

Here’s what it looks like when we put it all together, looking at overall compliance with CDC guidelines:

So I guess I was right. Runners are the worst. Pull it together. Runners, if you’re out there reading this, just move over. If people walking can do it, then you can do it too.

This is the ninth in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is not one of those posts.

In the dystopian fashion of 2020, I logged onto Zoom Saturday afternoon to watch my virtual graduation. Four years of college bookended by Trump’s election and a pandemic. Different people will have differing opinions on that first one, but if you know me then you know my opinion. If you don’t know me, I think that Trump is as bad for my health as the pandemic – and I’m immunocompromised.

To add another pinch of pessimism to this post, I decided to document where I was this weekend to compare it to where I would have been if there were no pandemic (unfortunately, Trump is still president in this scenario). Here’s how the weekend was supposed to go:

My family, grandparents and all, planned to come up for the weekend. There were brunches and dinners planned and, of course, my graduation ceremony. My roommates and I were excited to have our families all meeting each other, there would be graduation parties and Senior Week, and it would be a time of happiness and celebration. Not exactly the way things turned out. At the end of the weekend, my family would drive some of my stuff home for me, and later in the month I would drive the rest of it back, and hopefully go on to start my first job out of college.

Here’s how things actually went:

On Friday, I was in my room most of the day while everyone in my house either worked or was in class. I had dinner with my family, so I suppose that’s similar.

On Saturday, we sold my car to Carmax. We watched both my graduation and CNN’s star-studded, impersonal, umbrella graduation for high school and college graduates where everyone said “sorry this sucks. Here’s some promotional content.” Obama’s speech was nice. I FaceTimed with my roommates. I had dinner with my family. I ate too much ice cream.

On Sunday I had a stomach ache from the ice cream. I hung out with my mom. I napped a lot. I made dinner and ate it with my family.

Sprinkle in some Netflix/Hulu in there and you’ve got a pretty clear picture of my weekend. Which is not unlike my week days.

I spent roughly 95% of my time inside the house. I hate pie charts (even though I love pie), but I made a pie chart that I consider slightly less offensive than most pie charts because it’s meant not to give you real numbers, but instead a depression-inducing glimpse into the amount of time I spent in my bedroom this weekend.

A quick note about the data: minutes were rounded to multiples of 5, I didn’t count the bathroom, and I didn’t count time asleep.

Pretty disturbing, right? I thought so too. I was even shocked about how much of my time I spent in my room. So I broke it down to try to see if there was any way to make it less sad.

Here’s what I found:

Please excuse the Barney colors. I couldn’t find any pens. If you can see beyond that, you’ll notice most of the time comes from Friday and Sunday. I can rationalize some of that. On weekdays (Friday), every room in my house is occupied by people who are working or in class. This means that if I’m not in my room, I can’t make noise. My only excuse for Sunday is that I took a nap. Still, that’s a lot of time to be in my room. Especially on a weekend where I was meant to be celebrating and having fun with my family and friends.

It’s not all tragic. My uncle just brought us pies and breads that my aunt made (buy some here if you’re in/near Chicago). In about a week I’ll be back in Minnesota with my girlfriend. Hopefully I’ll be walking around outside more and writing more positive posts. I’ll let you know.

This is the eighth in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is one not of those posts.

This post is being posted in the middle of week nine because of the following excuses that I find valid but you may not:

• Finals
• Six hour drive from MN to IL
• My mom’s wedding (in quarantine)

Accept it or judge me. It’s the internet so I’ve prepared myself for both.

Anyway, this week I analyzed my music habits. I’ve been thinking a lot about music recently because my band, Geyser is finishing up an album. I know – shameless self promotion. I wanted to see if my music habits have changed since starting quarantine. I’ll take you through what I found.

I started off by requesting my music listening data from Apple because they don’t have a nice public API like Spotify does. That took about a week to get and they gave me way more data than I needed, but better too much than too little. My first question was this: How have my top artists changed?

I started off by looking at my all time artists overall (all time being since October, 2015).

I wasn’t really surprised by this. I go through phases where I find an artist I like and listen nonstop. That covers Brandi Carlile, Chris Stapleton, and Jade Bird. Wilco and Gillian Welch are another story. And I wanted to show you that story so check out the next graph.

You can see what I was talking about with the obsessive listening: Each of the three artists I mentioned begins with a steep incline when I start listening to them and then either continues up with each new album they release, or flattens out.

Now let’s talk about Gillian Welch. She’s a long time favorite artist and you see her climbing steadily over time in the graph, but then there’s a sharp uptick in 2020. That’s because I played her version of Radiohead’s Black Star in a concert and I was trying to learn Dave Rawlings’ killer guitar solo note for note.

And finally Wilco. Wilco is the dark blue line. You can see the same pattern that Gillian Welch took where there is a relatively flat but gradual climb and then a quick increase in plays. This one’s slightly embarrassing but what is the internet for if not sharing embarrassing things with the world. I really like Wilco and their music. They were one of my first concerts as a kid. But for the longest time, I felt weird listening to their music because I was friends with the singer’s son and we played music together. But according to this graph, I realized mid-2018 that this was stupid and I listen to their music a lot now because it’s great music and no one cares.

Okay. So I know who my top artists are overall, but who are my top artists in quarantine? Take a look:

Pretty different. Brandi Carlile and Wilco still make the cut, but my most listened to artist since quarantining is Sarah Jarosz. I didn’t expect this, but I have been listening to her music a lot. It’s on the calmer side and the instrumentation is really nice.

Next new addition – John Prine. I was pretty sad to hear that John Prine died from the coronavirus. If you haven’t heard his music, I suggest giving it a listen. He was a hero to songwriters. As I write this, I’m in an apartment across the street from the Old Town School of Folk Music where he used to take music lessons and where I grew up learning music. So, I’ve been listening to a lot of his music because his lyrics are the epitome of good songrwiting and his chords are simple and easy to follow. He once said if you play more than three chords you’re showing off.

Finally The Lone Bellow. I haven’t listened to their music in a long time, but when their new album came out this year, I was reminded of why I used to like them so much. Their three part harmonies and strong voices are soulful and fun to sing along to. I had ticket’s to one of their concerts in April. Hopefully I can see them play the new album in the future.

Okay. Artists are covered. What about genres? The other question I had about my music was how the genres I listened to morphed from before quarantine to now.

I started by looking at seven weeks of music before quarantine, and seven weeks during quarantine so I could compare them responsibly. My music taste has changed some over time so in order to see if there was a shift. Here’s that information:

Not really much of a change here in raw numbers, but where there is a change is proportions. It seems I’m listening to a lot less Alternative music, and I’m listening to a lot more pop, blues, folk-rock, and jazz. To be honest, I’m not quite sure why that is. It’s something to think about.

I thought that perhaps if I looked at genres over time I might gain some new insights:

It appears that my increase in alternative music started this year. I think I can attribute this to listening to reference tracks for Geyser’s new singles/album (another shameless plug) a lot as we finished up mixing. I also see that rock overtook pop. I’m not sure why, but I’m happy about it.

So, I suppose I’ve learned very little from this. You’ve probably learned more than me but little of value unless you’re very curious about my taste in music. If you’ve gleaned anything from this post, I hope it’s that there are a lot of reasons to listen to different artists during this time. While there may not be many new albums coming out, we can find ourselves being drawn to our old favorites for new reasons. That, and maybe you’ve found some new music to listen to right now.

This is the seventh in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is not one of those posts.

I hate clutter. I hate clutter but I like stuff. To make matters worse, I am bad at getting rid of stuff. I also love organizing and you can’t organize if you don’t have stuff.

So just to recap there:

• I hate clutter
• I like stuff
• I am bad at getting rid of stuff
• I like to organize said stuff

Now that that’s out of the way, we can get to the juicy… “stuff”. As you may have guessed, this week I have been tasked with contemplating organization. I find organizing things to be therapeutic. The feeling of completing a task that is not only visually appealing but incredibly useful does wonders for my Virgo brain. I organize my shirts by warmth and then by color, I keep my wool socks and my cotton socks in separate drawers, and I always put my pens back in chromatic order. I even do things with organizational order – I always brush my teeth before washing my face because sometimes you get that toothpaste foam in the corners of your mouth and then you have to wash it off.

So, because organization is a part of my day to day life, reflecting on it overall seemed like too broad a brush to paint this post with. I decided instead to focus on my most recent and largest organizational event: packing.

Even though I don’t go home for a week and a half, I decided on Sunday that the best use of my time would be productive procrastination which led me to pack nearly all of my clothes. I packed three bags – one with winter clothes, one with summer clothes, and one with my dog’s sweater and his bumble bee Halloween costume that I didn’t think he would need while he stays in Nashville with my sister. If you’re wondering, yes I did vomit a bit at the fact that I purchased a Halloween costume for my dog, but he looked very cute in it.

Every time I have packed up all of my belongings to move during college, I have gotten rid of clothes. It’s less of a thing that I want to do and more of a task that I force myself to do so that I don’t end up with six tee yellow shirts, all of which I “need” because they have slight variations in shade and neckline. I always try to donate my clothes to Goodwill, but in writing this post, I realized that I really have no idea what happens to those clothes once I drive away. So, I decided to do a little digging.

Here are some things I found:

• Americans throw 10.5 million tons of clothing into landfills each year
• Only 7% of the population buys second hand clothes
• Only 28% of the population donates used goods

This means that only a quarter of the number of people who donate clothes buy second hand clothing. Additionally, charities like Goodwill and the Salvation Army sell less than 20% of the clothing donated to them. The rest ends up in the hands of for-profit textile recyclers.

So let’s break that down. If we divide our disgraceful waste evenly across US citizens we get roughly 64lbs of clothing waste per person annually. This waste in reality would not be equally divided person to person, but for simplicity’s sake let’s assume it is.

That’s a lot of clothing!

According to The Atlantic, Americans donate 15% of their used clothes. This means that if we distribute the used clothes evenly like we did above, each American gets rid of about 75lbs of used clothes annually and donates only 11.25 of those pounds.

Here are some ways to think about those weights:

So the next time you think about throwing away your old clothes, try giving them away to your friends, recycling or donating them. They might not get bought but it’s better than throwing away more bars of gold.

This is the sixth in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is one of those posts.

This week, we’re focusing on text analysis. Aisling has asked us to take some of the reflections we’ve done since the start of remote classes and put them into the website Voyant to see what insights we could gain. I tried this with all of my blog posts and found that my top 3 words were “data”, “quarantine”, and “group”. Not surprising to me, but still, it is interesting to see that the words that follow those top 3 are the names of the people I’ve been with throughout this quarantine process.

I find text analysis to be a very interesting topic in data analysis. Although it’s an area I don’t have much experience with, it’s something I’ve been experimenting with on my own time and want to learn more about. I used this week’s assignment as an opportunity to do that. To start, I did a sentiment analysis of my blog posts using the R package syuzhet. This package is very easy to use and because I just wanted to do a simple analysis, I didn’t get too deep into the “behind the scenes” aspects of sentiment analysis for this project.

The first thing I did was get a basic overall understanding of the different sentiments found in my blog posts:

Okay… the highest value that comes back is positive but that is closely followed by negative. What does this mean? How does it work?

What’s happening here is that there is a dictionary of words for each category. For example, words like “good”, “happy”, and “smile” might be in the positive dictionary whereas words like “bad”, “unhappy”, and “frown” might be in the negative dictionary. The text is read one word at a time and if that word is found in one of the dictionaries, it gets counted. Here’s an example:

Sentence: “I smile when I am happy. I frown when I am sad. Today I am feeling good.”

Our dictionaries might look like this:

positive = 
[
  happy: 1,
  smile: 1,
  laugh: 0,
  good: 1
]

negative = 
[
  sad: 1,
  frown: 1,
  cry: 0,
  bad: 0
]

That puts our total count of positive words at 3 and negative words at 2. We can classify this sentence as slightly positive.

So this is good. My blog posts have a lot of words that likely signify positivity, joy, and trust. They also, however, have a lot of words that signify negativity and sadness. I wanted to see how this trends over time so I used syuzhet’s functions to plot how the emotional valence changed. On this graph, I’m only looking at positivity (above 0) and negativity (below 0) trends. This does not reflect every sentence of every blog post, but instead looks at overall trends on a scale that has my most negative sentiment at -1 and most positive sentiment at 1. We can think of it as a scale relative only to my blog posts – not positivity and negativity in general.

Hmmm… I don’t love that, but it makes sense. My motivation and general positivity have decreased since I began quarantining and this is likely reflected through my activities and thus what I write about. Again, this does not mean that my most recent posts are extremely negative. It just means that compared to my personal baseline in these posts, the more recent posts are more negative than the inital posts. I anticipated this but I was surprised by the severity of the drop and how strictly downward it moves.

Finally, I wanted to do something completely separate: I wanted to use Markov chains to generate text based on the blog posts I’ve already written. If you want to understand Markov chains for text generation, this post does a fairly good job of explaining without getting too mathematical. I will admit, I am no expert in Markov chains, but I find them amusing for text generation and they’re something that I want to learn more about. From a very, very, very high level perspective, Markov chains use probabilities to move from state to state. In this case, each state is a word and we are looking at the probability that a word will follow the previous word. We keep choosing words this way until we reach the desired amount of words.

I used the R package Markovify for this quick text generation. I gave the Markov chain model the following rules:

• only replicate either up to 5 words in a row of the original text (my combined blog posts) OR
• up to 40% of a generated sentence can overlap the original text
• the maximum sentence length is 150 characters
• only give me distinct sentences

Because my corpus is fairly small, the model did not have a lot of training material. This means that the output is very limited in possibility and it probably won’t look like something I wrote. If we gave it the entire Shakespeare corpus, it might be able to replicate his style (to a point), but my ~4,200 words don’t provide much material for the model to learn my writing style. I did it anyway because it’s funny. Here’s an excerpt from the generated text:

Let’s say I’m looking at your phone in the same office every day. For now, I think most young people have a safe place to isolate for two weeks. I think most young people have a safe place to isolate for two weeks. Let’s say I’m looking at your door while you spy from the graph, Rosh Squad exists – to appease Shane. That’s the first week I am in the family. With the quantified self, we’re trying to make GPS tracking more accurate, I don’t have time for that. This is not one but two weeks out of my cute dog, Roshi. This group will soon be moved from the start of quarantine. Mostly, the things that changed around me and I were supposed to go to a party?
So, all of the members are already in a group without Sam? He doesn’t want to be safer with your co-worker. Unsurprisingly, I have an obnoxiously unnecessary number of edges going into or out of hot spots, etc.. Finally, at the first problem. Being as I am in. Two people came in from out of my texting is to my walking pre-COVID.
Samuca and the horizontal line shows the start of week 6. This graph is that I had to.
Unsurprisingly, I have to be in a group to dis her in person.

Pretty nonsensical, but entertaining nonetheless. I wonder how generated text compares sentiment-wise to its training corpus. A project for another time.

This is the fifth in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is one of those posts.

This week, my class has been asked to contemplate data privacy. If you have not read it already, take a look at the New York Times’ The Privacy Project. This project describes very well why data privacy matters, the scope of personal data collected on individuals, how it can be used, and how to be safer with your own technology.

I think most young people have a good understanding that when you buy a smart phone and carry it around, it’s collecting data on you. I looked at this a bit in my post on the quantified self. Our phones track information that they share with us (steps, screen time, etc.), but they also collect data that they keep for themselves – or in some cases, sell. Most notably, GPS data.

In reading this New York Times series, I began to think about how this data might be used to track the Coronavirus and what that would mean for Americans. This is happening in other countries and is in talks for the US, although we don’t know much about the efficacy of using phone data to track social distancing and contamination. The Washington Post claims that “the U.S. government is in talks with Facebook, Google, and other tech companies about using anonymous location data,” but anonymous location data is easily un-anonymized. The New York Times Privacy Project describes this in detail, but I’ll explain it quickly here too in case you don’t have time for that.

Let’s say I’m looking at your phone data. I might pull all of the records that are labeled with your anonymized ID and be able to see your day to day movement. In a simple case, you sleep in the same house every night and work in the same office every day. A few Google searches and I could find your name. This data is not really anonymous.

That’s the first problem.

Now, here’s why I don’t think cell phone data should be used to track things like social distancing or possible contamination.

GPS data on the average smart phone is only accurate up to about 16 feet. This is why your Uber parks down the block to pick you up. It’s not their fault. Cut them some slack. While this is remarkable for using softwares like Google Maps, it’s not so good when your social distancing radius is six feet. If you’re within six feet of another person, a GPS does not know if you’re two feet away or ten. It’s also hard to tell if you’re on the same floor as someone else or not. GPS is even worse with vertical placement than with horizontal. It might appear that you’re giving a piggyback ride to your boss, when in fact she’s two floors above you. And God forbid you forget your phone in the bathroom. Now the government thinks you are sharing a stall with your co-worker.

But Juliet, you might say, every American is quarantining at home. No one is with their boss and their co-workers.

To which I would reply, you are wrong. Just ask my sister who FaceTimes me every day to show me the congregations of people drinking coffee below her Nashville apartment.

But let’s pretend that you’re right and all Americans are practicing strict social distancing and only going to stores when essential. Still, the 16 foot radius causes us an issue. And with regards to spreading infection, this could cause unnecessary panic. What if two people are next to each other at a stoplight in separate cars? Would we count them as potentially infected? Or maybe your Instacart delivery person drops your groceries at your door while you spy from the window. Are they sent some sort of message telling them to get tested for the virus? Not to mention the issue of noncompliance – what if you leave your phone at home and go to a party?

Sure, there are potential upsides to using cell phone location tracking. We can look at patterns of movement, see where people congregate, observe migrations in and out of hot spots, etc.. But the government could also attempt to use anonymous data to do things like enforce curfews, track social distancing, or even use identifiable metadata about your texts and phone calls to predict who you’re contacting and find out who you have been with. And not to sound too patriotic, but would this not be a violation of our fourth amendment rights?

In the Supreme Court case Carpenter v. United States, the court decided it was, calling cell phone location data a “detailed chronicle of a person’s physical presence compiled every day, every moment over years.” Police need a warrant to get these records. I suppose this could be deemed within the interest of national security, but I am no constitutional expert and I’m quickly veering out of my domain.

My point is, while there are ways to make GPS tracking more accurate, I don’t think this would work particularly well and I don’t think it’s worth the potential panic it could cause.

If you’re concerned about your data privacy, take a look again at the New York Times Privacy Project (for which this blog post is quickly becoming an unpaid advertisement).

This is the fourth in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is one of those posts.

Thinking about social networks made me realize that my typical social networks are all out of whack right now. Normally I’d be living at home – either in Chicago with my family, or in St Paul, as I am finally doing now, with my roommates. When I’m with these people I talk to them in person and don’t interact with them as much online. We communicate mainly with our voices, not our phones or computers. Since I started quarantining, the people I communicate with has changed a lot. When I realized this I went right to my phone to look at my texts and had the unsettling realization that I have an obnoxiously unnecessary number of group texts consisting of mainly the same people.

First, here’s what that looks like. Next, I’ll go through the upsetting reasons they’re all apparently necessary.

Let’s start by taking a look at some basic and disturbing facts about this graph…

I’m going to use some terminology here for networks:

• Nodes: each person here is a “node” or, a thing that can be connected to other things to show some sort of meaningful relationship.
• Edges: each line is an edge. These edges show how nodes are connected to each other. In this case, each edge shows two people who are connected by being in the same group chat. The color tells which group chat connects them.
• Degree: degree measures the number of edges going into or out of a node. This helps us understand how important or how connected one node is in relation to others in the graph. For example, the ME node has the highest degree because this is a narcissistic graph about me and I only care about the group chats I am in.

Now some facts:

1. There are 9 nodes in this graph.
2. There are 7 distinct edge colors.
3. Unsurprisingly, I have the highest degree. I have more than my sister, Maddie, so this makes me happy.
4. My mom is the least connected one in the family. Sorry, mom.
5. Sam is the least connected overall, but this will soon change.

So why are these all necessary and why will Sam soon be moved from the bottom? Let’s go through the chats one by one.

Samuca and the Newts: Me, Coleen, Sam, Mathea

This group must exist because it provides a communication channel among my roommates and me.

nothing is important here: Me, Coleen, Mathea

This group might seem redundant at first glance. Three of the members are already in a different group together. Why would they need to be in a group without Sam? Because we’re telling ~secrets~, that’s why. Sam’s birthday is April 9th, and this graph was created April 7th. This group will soon be redundant, but not yet.

Stay Yo Ass At Home: Me, Maddie, Shane, Mom, Victoria

The most juvenile yet semi-appropriately capitalized and most effective name calls upon us to “stay our asses at home” during quarantine. Inspired by Lori Lightfoot memes, this group is a way for the family (and my sister’s girlfriend (but not my mom’s fiance?)) to stay connected and in the loop about the infrequent excitement during the coronavirus. This group is necessary for family communication.

Rosh Squad: Me, Maddie, Victoria

This group seems unnecessary too, right? While, it’s not. First reason – it is dedicated exclusively to pictures of my cute dog, Roshi. Now, you’re probably wondering why we can’t send those in the Stay Yo Ass At Home chat. This is because my brother, Shane, gets bothered by texts that he deems “unnecessary.” If we remove Shane from the graph, Rosh Squad becomes unnecessary and cute dog pics can stay in Stay Yo Ass At Home.

Shay Shay & Maddie: Me, Shane, Maddie

This group is for the siblings to talk about our mom without her knowing. It’s necessary. Next.

???: Me, Maddie, Shane, Secret People

Why would Maddie, Shane, and I need another group to communicate in? We already have a group to dis our mother in secret and another to dis her in person. You might notice that this group includes other people who are not named on the graph. This group can be revealed later, but for now it has to remain a secret. I promise, it’s necessary.

Bitch Squadron: Me, Mom, Maddie

Bitch Squadron is the oldest group among the groups. The OG group chat between a mother and her daughters. This group exists for the same reason Rosh Squad exists – to appease Shane. He doesn’t want to be bothered.

So, all of these edges are necessary in the current setup. As a person who loves efficiency and optimization, I hate this and have found a solution: remove Shane.

This might seem drastic and barbaric, but hear me out. Removing Shane eliminates three groups – Rosh Squad, Bitch Squadron, Shay Shay & Maddie – because they no longer need to be distinct to avoid disrupting Shane’s day. Communication that existed in them previously can now flow through Stay Yo Ass At Home. Removing Shane also removes 14 edges.

• The 8 edges that connect to Shane through SYAAH, ???, and Shay Shay & Maddie
• The 3 edges that connect Rosh Squad
• The 3 edges that connect Bitch Squadron

Considering that nothing important here will also disappear in two days, removing Shane would:

• bring the group count to 3 (SYAAH, Samuca and the Newts, ???)
• bring the edge count to 14
• bring the node count to 8
• tie my mom and Sam for lowest degree (excluding secret)

Sometimes it’s hard to face the facts of science.

This is the third in a series of posts From Quarantine. Most posts from quarantine are prompted by Aisling Quigley’s Data Storytelling class at Macalester College. This is one of those posts.