Educated Guesswork

As promised, herewith begins the data analysis of my first 110,000 songs heard through iTunes.

Top Level

Saving the caveats for later, we dive in with the statement that I have listened at least once to 110,000 tracks in iTunes. Crossing this milestone occurred just before 5PM on April 17, 2019. Remaining to be heard at that point were 81,831 audio tracks, meaning 57.34% of my listenable* collection has been heard. The heard files occupy 705.21 GB of data, and span 442 days, 10 hours, 37 minutes, and 36 seconds in length. The unheard tracks take up 567.17 GB of disk space, and would take 326 days, 13 hours, 38 minutes, and 42 seconds of uninterrupted time to hear in total. Dividing the total time to hear the 110,000 tracks (a little over a year and two-and-a-half months) by the number of tracks gives us an average track length of 5 minutes and 48 seconds, thought that summary average is a trifle deceptive, as we shall see.

Average Track Length: 5:48

The average file size is 6.57 MB, though these have an even wider variance than the length of the files in question.

The average song rating for those tracks which I have rated is 3-2/3 stars, using iTunes much-deprecated rating system.

Average Track Rating: 3.67

Not all songs have been rated: I do not rate tracks designed to impart instruction in a foreign language (including Old English), and I have only begun consistently rating songs in more recent years (see caveats for issues with this datapoint). Only 142 tracks fall into the Language genre, leaving 32,830 songs I’ve heard without rating them.

Total Songs Rated: 77,028

The vast majority of the eleven myriad† songs here were listened to only one time. 91,172 tracks were heard only once. Of the remaining 18,828 songs, the majority (10,881) were heard twice. The numbers drop off drastically after that as the play count increases, with only 760 songs being heard more than 12 times. (There are some issues with the higher play counts, explained in caveats below.)

With all due caveats about the earliest play dates, it took 5,935 days to listen to these 110K tracks, that is to say, 16 years and 3 months (plus 3 days as change). Thus, I listened to, on average, eighteen-and-a-half songs per day. As will be seen, this listening rate fluctuated drastically over the course of the past sixteen years, though the overall rate is remarkably similar to the rate for the most recent 10,000 songs heard.

18.53 New Songs Heard per Day

Conversely, although only 442 days worth of music were listened to during that 5,935 day period, when we summarize based on the number of plays, we find that I’ve listened to just under 600 days of sound files of one species or another. (The actual figure is 599 days, 2 hours, 39 minutes, and 5 seconds, but who’s counting?) This means that I listened to iTunes (on average) just over two hour and thirty-five minutes each day.

2 hours, 35.33 minutes per day Listening to Stuff

Top Level Genre Information

Though I tried to scrub this data as effectively as possible (see caveats below), I found that my Genre information contained much garbage, and then found as I tried to correct that problem that this was and is an endless task, so I decided to end the task, and present you with the information as I have it now. You may feel free to object all you wish to the stuff presented here, and I look forward to your vicious attacks upon this and any other useless data.

That said, the songs or other audio already heard fall into 89 different genres. Fully one quarter of all tracks are classified as Rock (27,639 files). The next two most populated genres are Radio Show and the bipolar and not quite useful Alternative & Punk categories, each making up approximately 6% of all tracks already heard (6,773 and 6,567 examples, respectively). I am endeavoring to separate ‘Alternative’ from ‘Punk’, and hope to complete that before another ten thousand songs are heard; currently 4,421 files are classed as simply Punk, while another 3,009 are labeled as Alternative. If those were added to the almost pointless ‘Apples & Oranges’ category mentioned before, that would make the combined weight of these tracks equal to over 12.5% of all heard files. Also of note are 3,109 files for which no genre information is attached. Looking at those files for which at least 1,000 examples are found, the breakdown by genre is as follows:

And of course there’s a picture for those visually-minded among you:

Top Level Popular Artists

One way of determining the most popular artists is by looking at the number of songs played during the eleven myriad tracks I’ve listened to. Doing so give us the following breakdown:

There are, however, other ways of looking at this data. We can, for example, look at the total number of plays for each artist to garner a different view of my biggest faves, for there are bound to be some I love but for whom I don’t have lots of tracks, or simply for whom there were never that many songs released to begin with, but which I listen to over and over again. When I looked upon this data from this perspective, malhéreusement, I found that the issues surfaced above and noted in the caveats regarding higher play counts washed away any useful information. I next attempted to use ratings as a guide (despite the caveats), but that merely rearranged the data already given in the table above without surfacing any really new insights. I also tried using these same factors with an additional weighting by length of each track, but that merely promoted the radio shows and other artists with longer songs (or at least longer noodling; I’m looking at you, Jerry).

There were a few more revelations to come when we looked at the popular artists over time, which data are presented in the next section.

The View Over Time

The revelations of these 110,000 datapoints of sound attain sharper perspective when viewed over time. Here we learn some surprises and also see how drastically my listening habits have changed over the past 20K tracks. Three main factors appear: consistent average song length until the past few years, increasing file size, and widely varying listening rates until recently. The following chart gives a high-level overview (broken out by successive groups of ten thousand songs heard) of these and other aggregate data.

The primary change to my listening habits, which is apparent in the chart above, has been the introduction of large swathes of radio shows to my listening budget. These old shows are freely available on Archive.org and many other sites, and you can find them easily by searching on the term “OTR” (for “Old Time Radio” — although all radio narrative content is now ‘Old Time’, since ‘New Time Radio’ is now called “podcasts”). Several factors impelled the addition of this material to my frequent listening, the primary one being the material changes to my conditions of existence on this vale of tears some few years back. Naturally, these tracks have added substantially to the Average Track Length of my songs heard, as most shows are at least a half hour in length, with some exceptional cases lasting over two-and-a-half hours.‡ Although a very small number of radio shows were heard in earlier years, this category of audio file entered heavy rotation at the end of August in 2015, and has been a consistent part of my iTunes diet since that time.

One other change in — not my listening, but — my ripping habits has been to convert songs into higher bitrate mp3 files. I’ve done this because of the larger hard disks available, as well as the supposed benefits which accrue from the bigger files. I have to confess, however, that my hearing is possibly too poor to notice the difference, though even I can hear the ‘tin can’ effect of a few files I’ve grabbed which were ripped at 32 or even 16 (horrors!) kbps. I generally use 320 kbps, and I do continue to rip as mp3, primarily for ease of portability and future-proofing (to the extent that that is possible).

Lastly, I see that though the aggregate numbers show a remarkable consistency (indeed, the overall value for songs per day matches almost preternaturally well with the same datapoint for the past two tranches of 10K songs), my listening shows long lulls as well as frantic listening. The difference between listening to eight songs a day versus over fifty-five a day, particularly over such a long stretch as 10,000 songs, is quite substantial. A quick back-of-the-envelope (have you considered getting your statements online?) calculation reveals that the last figure, for the 81st-90th thousand songs, multiplied by the average song length during that period of four and a third minutes, means I was listening to four hours of iTunes each and every day. Perhaps not that remarkable to some, but it impresses me, though with what I am still unclear.

Listening Rates

Obviously not only the underlying material but also the environmental factors have changed over this 4/5ths of a score of years. That is, at times I had little ability to listen to my own music in the car, or I was just not driving that much, while at other times I did nothing but listen to my tunes. Or, I listened to iTunes at work, or at times the work was so intricate and involved that music would have been only a distraction. These and many other factors lie behind the differing song rates shown in the table above.

Looking more closely at the data by slicing not at the 10k but at each 1,000 songs presents a more nuanced picture. We see a long stretch where my listening dropped to almost single digits per day. Before and after that lengthy period of time (which lasted from approximately April 2006 through January 2013) the listening rate jumps up and down higgledy piggledy. So says the following chart:

Or, we can look at the inverse of this chart, and view how many days were required to reach each set of 1,000 songs:

(click on either chart for more detail)

Popular Artists Over Time

As mentioned in the discussion of popular artists, the popularity of the various artists changed over the over sixteen year span covered by this data. Though so many disparate artists were heard that any attempt to catalogue the ‘top’ artists seems futile, yet I shall assay that futility, and here below present the Top 25 Artists heard during each tranche of 10,000 songs. I use the same methodology as the Popular Artists in the previous section of general overview, with all the caveats, problems, and sighs inherent to same. For comparison’s sake, the data is presented in a scrollable table so that you can view how the top artists changed over time.

Most Popular Artists by Number of Songs Played per each 10K Songs

(scroll to right to view)

Unsurprisingly, Bob Dylan features prominently in all eleven (11) slices of these 110,000 songs, never falling any lower than the 3rd position. The last two myriads show the rise of Radio Show in my listening diet. There may also be evidence of increased data capture, as the ‘{unknown}’ artist entry falls below 10th place for the first time in the most recent set of 10,000 songs heard. Looking closely at each tranche reveals my interests over time; for example, the set ending with 40k songs heard includes FDR, Winston Churchill, and CBS among the Top 25, indicating this was when I was listening to old news broadcasts from World War II (‘CBS’ here refers to the progenitor of the World News Today program).There are a few other surprises and oddities in the data — 112 De La Soul tracks!?! — but I’ll leave most of those as an exercise for the reader.

Sample Tracks Over Time

Flying over sixteen years of data means that no meaningful detail can really be seen at the most granular level, so permit me to provide that meaningless detail. Without (much) further ado, I present 0.1% of all the songs heard in this set of 110,000 tracks, randomly chosen by grabbing the 1000th, 2000th, etc.

Concluding Remarks

Given the fact that the underlying data analysis terms have changed (see my earlier post on this subject), I am presenting this information more as a baseline for future reports rather than as continuing commentary on my listening habits. Of course, the underlying dataset presents all manner of wonders for the enthralled searcher, but I confess that I am looking forward at this point to just shoving this turkey out the door and getting back to listening to iTunes. (I cannot listen to audio while typing anymore, another resentment I have against my teenaged self.) The plethora of data I have is, as I have said before, pointless — doubly so because I am not selling anything based upon it, which seems to be all data may be used for in our New New World.

I wish all of you well, and will report back when I have listened to another 1,000 tracks (I am about halfway there since I began compiling this information on April 17th [UPDATE:Now less than 50 tracks away as I finally hit the ‘Publish’ button]). I am primarily going to use the information I have to clean up some of the cruddier parts of my data, which is harmless enough I suppose, though pointless. For now I look forward to closing out the stupidly large files I’ve been messing with to gather this information for your perusal.

Good Day

Technical Notes

All data generated using Excel for Mac 2011, based on iTunes library and playlist export text files. For certain calculations I used the i41CX+ app for the iPhone as well as a Pickett N803-ES Log Log Speed Rule Dual Base slide rule. All audio files managed through iTunes, now on version 12.8.2.3, with additional file manipulation with Audacity as well as brute force tweaking of filetypes to generate ring tones, etc. The iTunes Library is maintained on an external hard drive, with two other hard drives for backup using rsync. Most (though not all) files are also kept separately in physical formats that will likely become obsolete along with so much else.

Caveats

Song Ratings

Some problems exist in the data available to me, as some glitch between my iPhone (used to listen to most tracks) and iTunes causes intermittent ratings to be applied to whole albums, which I never do. Those ratings get translated as individual song ratings for songs which have no explicit rating, and it is not possible to distinguish between the two (explicit vs. induced) in the data export file I used for this analysis. It is an annoying problem, and one could add it to the heap of complaints that people seem to have about iTunes as a piece of software. I say, in contrast, however, that I know of no other program that could give me data about what I had listened to for the past 15 years, unless I wrote it myself, which I am not capable of doing. Of course, it may be objected that who would want to do so? I can only submit myself as the proof of the rule you would seek to impose.

I should also point out that I do not use 1-star ratings for anything save as a placeholder for possibly corrupt files. This is because I use the actual description associated with the star ratings in iTunes, and 1 star supposedly means in this system that “I hate it” — and I have not hated any track I have listened to. I have come close (looking at you, U-God), but thus far I have found that “I don’t like it” expresses my feelings well enough. Thus the ratings curve is skewed, but isn’t that true of all modern grade curves?

Play Counts

The highest play counts are somewhat suspect, as these are almost all songs which were tracked during a period in which my iTunes was being shared for my daughter’s iPod usage as well as my own. Thus certain Green Day and My Chemical Romance tracks (which I love) have plays which I cannot swear are all mine. And the most played track is Florence + the Machine’s “Howl”, which I can aver has not been heard by me 157 times, though that is what the data says. Perhaps the only track having more than 70 plays for which I can claim all those plays is the iTunes special version of John Cage’s 4’33” (I do not quote the song title for clarity), which I have heard 75 times — whatever that statement means.

Early Dates

The ability to track information about large datasets always comes with a cost. In general this cost is to be seen in the sheer difficulty of maintaining internally consistent datapoints across the entire set, a difficulty which can only grow as the number of entities tracked becomes greater. While the overall makeup, trends, and detail of the aggregate information will only become more precise as the number of points becomes larger, there will always be database inconsistencies which threaten to hobble complete understanding of the full dataset. This is due to three primary factors: mistaken, incomplete, or corrupt information in the original data capture; inconsistent data entry, especially with multiple sources; and changes to noted data points, schema, or methodology over time. There will always be certain outliers in any sufficiently large set of data that have missing, incorrect, or otherwise inconsistent information stored for particular datapoints. These outliers will surface to plague analysis once a ‘deep dive’ into the data is begun, and how these are handled determines much of what is possible in a complete analysis. Bottom line: You can never know everything in your data universe, unless the number of points in your data is so small as to be worthless for statistical purposes.

With that ridiculously overstated preface, I note that the set of 110,000 songs heard has a small number of songs which are missing one vital statistic: Date Played. The set was generated by looking for a value for ‘Number of Plays’ greater than one, and for 127 tracks the database contains no data for the ‘Date Last Played’ datapoint. Since this last bit of information is used to generate the view of when I listened to these files over time, we have a small (just over 0.115%) set of files for which I can’t tell you when I heard them, although iTunes assures me that I did. A very small number of these files simply do not exist anymore, lost in the great 20GB hard drive crash back in the earliest days of my iPod usage. (I’ll always miss you, variants of Blondie’s “Rapture” from that long lost EP.) It appears from a cursory examination of the two other datetime datapoints (‘Last Modified’ and ‘Date Added’) that none of these files was messed with or was created before the earliest date seen for plays: January 13, 2003. Thus I use this date for the earliest information given in the time sequence analysis. However, petulant perfectionists should note that there may be something wrong with the earliest dates given for song plays, as most of the 127 items missing this factoid seem to be from the beginnings of time — at least as concerns my iTunes tracking.

Also note that I don’t have any information about song usage — or even existence — before the cataclysmic hard drive failure mentioned above, where I lost the entirety (at that time) of my iTunes collection when my 20GB external hard disk failed utterly. Since that tragedy I have, of course, instituted a rigorous backup program, and since that time, also of course, no such failure has recurred.

I also note that the inconsistency between the datapoints ‘Number of Plays’ and ‘Date Last Played’ means that a different view of my data could give a different value for the total number of files heard, as should be obvious. What is not as obvious is that this particular inconsistency seems to to preponderate over the opposite; that is, the number of entries in the database which have a ‘Date Last Played’ value but no value for the ‘Number of Plays’ field is minuscule, with only eight (8) instances found. It seems likely that the original issue is or was caused by problematic data capture between iPods and iTunes through the various OSes and app versions used. Emerson!

Data Scrubbing

Besides the issues with date datapoints mentioned above, many other inconsistencies and outright errors exist in the full dataset. One of the main issues noted immediately at the beginning of my analysis was genre information, which was often either missing or so specific as to be useless. (It is not clear, for example, how useful such putative ‘genres’ as “Dylanesque”, “Meditative”, or “The Camera As Pen” actually are.) Though I have attempted to modify this and other entries in the underlying data, it soon became clear that to wait until all 110,000 song files had been completely reviewed and updated with so-called ‘correct’ information would both be pointless (see note about inherent inconsistency above) and take such a long time as to obviate any information I might care to impart about this ‘milestone’, as I could keep massaging the data long past the point where I have heard 120, 130, or even 150,000 tracks. I have decided to call a halt to the massive effort to impose some order upon all my iTunes files, though I have greatly modified the genre information, and hope to continue to do so in the future.

Thus whereas I originally had over 98 entries which were assigned a Genre for which they were the single example, I now only have 89 Genres all told in the set of music already played. I still have three categories containing only a single exemplar, but feel those Genres are reflective of my own musical taste and hope to add other songs to them as I continue my ad hoc data scrubbing. Thus I cannot promise that the underlying data will not be changed in the backend before the next full-scale analysis; I can only promise that I will attempt to play fair with you and let you know just how I’ve munged the information I have.

Footnotes

* The term “listenable” refers only to the type of file, and does not imply that the sounds contained therein are worthy of being heard.
† Taking “myriad” to mean “ten thousand”, as its original Greek root word does in most cases.
‡ The longest radio show tracks are actually composites of separate daily shows concatenated into single files for ease of listening. Thus, for example, the heyday of the great radio show Yours Truly, Johnny Dollar consisted of daily 15-minute shows which (generally) told a single story over the course of a single week. I took the individual shows and generated a single mp3 file of the complete story arc. (This led to certain needed tweaks to such parameters as Track Number etc., which you almost certainly don’t care about, assuming you’ve even read this far to find these words in this throw-away section of this meaningless report.)
§ If you see an connection between this self-indulgent data analysis and the known association of this Cyndi Lauper tune with masturbation, you are much more clever than I.