2023.08.22

VERSION

@@ -1,3 +1,2 @@
-February 08 2023
-
+August 22 2023
 

doc/CHANGES

@@ -15079,3 +15079,28 @@ February 08 2023
 midi2abc, mftext, midistats: modified midifile.c so that it indicates where
 it encounters an unexpected byte in the input midi file.
 
+
+March 24 2023
+More functions for the percussion track was added to midistats.c to
+support some future analysis.
+
+
+June 25 2023
+Midistats runs incorrectly if the midi file has no program channel
+commands. When a program channel command is missing, the program should
+be the acoutic piano (0) by default.
+
+If more than one channel with the same program color, they should be
+added together.
+
+
+August 22 2023
+
+Midistats improved the calculation of the number of rhythm patterns
+in a channel (barChn[chan].rhythmPattern) by allowing for dithering
+of note onset. Introduced dithermargin variable in function
+stats_noteon(). In function, output_track_summary(), suppressed
+notemeanpitch for percussion channel.
+
+
+

doc/drums.txt

@@ -0,0 +1,85 @@
+Advamced Percussion Analysis
+in the Midistats Program
+
+This is an addendum to the midistats.1 file.
+
+The MIDI file devotes channel 9 to the percussion instruments
+and over 60 percussion instruments are defined in the MIDI
+standard. Though there is a lot of diversity in the percussion
+track, for most MIDI files only the first 10 or so percussion
+instruments are important in defining the character of the track. The
+program Midiexplorer has various tools for exposing the percussion
+channel which are described in the documentation. The goal
+here is to find the essential characteristics of the percussion
+track which distinguishes the MIDI files. This is attempted
+in the program midistats.  Here is a short description.
+
+-corestats
+Produces a line with 3 numbers separated by tabs. eg
+384	8349	448
+It returns the number of divisions per quarter note beat (ppqn),
+the number of note onsets in the midi file, and the maximum
+number of quarter note beats in midi file.
+
+-pulseanalysis
+Counts the number of note onsets as a function of its onset time
+relative to a beat, grouping them into 12 intervals and returns
+the result as a discrete probability density function. Generally,
+the distribution consists of a couple of peaks corresponding
+to quarter notes or eigth notes. If the distribution is flat,
+it indicates that the times of the note occurrences have not been
+quantized into beats and fractions. Here is a sample output.
+0.3496,0.0000,0.0000,0.1602,0.0000,0.0002,0.2983,0.0000,0.0000,0.1914,0.0002,0.0001
+
+-panal
+Counts the number of note onsets for each percussion instrument. The first
+number is the code (pitch) of the instrument, the second number is the
+number of occurrences. eg.
+35 337	37 16	38 432	39 208	40 231	42 1088	46 384	49 42	54 1104	57 5	70 1040	85 16	
+
+-ppatfor n
+where n is the code number of the percussion instrument. Each beat
+is represented by a 4 bit number where the position of the on-bit
+indicates the time in the beat when the drum onset occurs. The bits
+are ordered from left to right (higher order bits to lower order
+bits). This is the order of bits that you would expect in a
+time series.
+Thus 0 indicates that there was no note onset in that beat, 1 indicates
+a note onset at the end of the beat, 4 indicates a note onset
+in the middle of the beat, and etc. The function returns a string
+of numbers ranging from 0 to 7 indicating the presence of note onsets
+for the selected percussion instrument for the sequence of beats
+in the midi file. Here is a truncated sample of the output.
+
+0 0 0 0 0 0 0 0 1 0 0 4 1 0 0 4 1 0 0 4 1 0 0 4 1 0 0 4 1 0 0 4 1 4 4 0
+1 0 0 0 1 0 5 0 1 0 5 0 1 0 5 0 1 0 5 0 1 0 5 0 1 0 5 0 1 0 5 0 1 0 0 0
+1 0 5 0 1 0 5 0 1 etc.
+
+One can see a repeating 4 beat pattern.
+
+-ppat
+midistats attempts to find two percussion instruments in the midi file
+which come closest to acting as the bass drum and snare drum.
+If it is unsuccessful, it returns a message of its failue. Otherwise,
+encodes the position of these drum onsets in a 8 bit byte for each
+quarter note beat in the midi file. The lower (right) 4 bits encode the
+bass drum and the higher (left) 4 bits encode the snare drum in the
+same manner as described above for -ppatfor.
+0 0 0 0 0 0 0 0 0 0 33 145 33 145 33 145 33 145 33 145 33 145 33 145 33 145
+33 145 33 145 33 145 33 145 33 145 33 145 33 145 33 145 33 145 33 145 33 145
+33 145 33 145 33 145 33 145 33 145 33 and etc.
+
+-ppathist
+computes and displays the histogram of the values that would appear
+when running the -ppat. eg.
+bass 35 337
+snare 38 432
+1 (0.1) 64  32 (2.0) 8  33 (2.1) 136  144 (9.0) 8  145 (9.1) 136 
+
+The bass percussion code, the number of onsets, and the snare
+percussion code and the number of onsets are given in the
+first two lines. In the next line the number of occurrences of
+each value in the -ppat listing is given. The number in parentheses
+splits the two 4-bit values with a period. Thus 33 = (2*16 + 1).
+
+

doc/midistats.1

@@ -80,6 +80,17 @@ all channels except the percussion channel.
 collisions. Midistats counts the bar rhythm patterns using a hashing
 function. Presently collisions are ignored so occasionally two
 distinct rhythm patterns are counted as one.
+.SH Advance Percussion Analysis Tools
+.PP
+A number of experimental tools for analyzing the percussion channel
+(track) were introduced into midistats and are accessible through
+the runtime arguments. When these tools are used in a script which
+runs through a collection of midi files, you can build a database
+of percussion descriptors. Some more details are given in the
+file drums.txt which comes with this documentation.
+
+
+
 .SH AUTHOR
 Seymour Shlien <fy733@ncf.ca>
 

doc/readme.txt

@@ -6,20 +6,18 @@ abc2abc  version 2.20 February 07 2023
 yaps     version 1.92 January 06 2023
 abcmatch version 1.82 June 14 2022
 midicopy version 1.38 May 06 2022
-midistats version 0.59 February 08 2023
+midistats version 0.71 August 222023
 
 24th January 2002
-
 Copyright James Allwright
-J.R.Allwright@westminster.ac.uk
+jamesallwright@yahoo.co.uk
 University of Westminster,
 London, UK
 
-October 2021
-
-Seymour Shlien
-Ottawa, Canada
+August 2023
+Copyright Seymour Shlien
 fy733@ncf.ca
+Ottawa, Canada
 
 This is free software. You may copy and re-distribute it under the terms of 
 the GNU General Public License version 2 or later, which is available from

midistats.c

@@ -18,7 +18,7 @@
  * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA
  */
  
-#define VERSION "0.62 March 08 2023 midistats"
+#define VERSION "0.71 August 22 2023 midistats"
 
 #include <limits.h>
 /* Microsoft Visual C++ Version 6.0 or higher */
@@ -75,6 +75,9 @@ int debug;
 int pulseanalysis;
 int percanalysis;
 int percpattern;
+int percpatternfor;
+int percpatternhist;
+int pitchclassanalysis;
 int corestats;
 int chordthreshold; /* number of maximum number of pulses separating note */
 int beatsPerBar = 4; /* 4/4 time */
@@ -100,6 +103,9 @@ int stats = 0; /* flag - gather and print statistics            */
 int pulseanalysis = 0;
 int percanalysis = 0;
 int percpattern = 0;
+int percpatternfor = 0;
+int percpatternhist = 0;
+int pitchclassanalysis = 0;
 int corestats = 0;
 
 
@@ -111,6 +117,9 @@ int notechan[2048],notechanvol[2048]; /*for linking on and off midi
 int last_tick[17]; /* for getting last pulse number in MIDI file */
 int last_on_tick[17]; /* for detecting chords [SS] 2019-08-02 */
 
+int histogram[256];
+unsigned char drumpat[8000];
+int percnum;
 
 
 
@@ -411,7 +420,7 @@ void stats_header (int format, int ntrks, int ldivision)
     trkdata.pressure[i] = 0; /* [SS] 2022-03-04 */
     trkdata.quietTime[i] = 0; /* [SS] 2022-08-22 */
     progcolor[i] = 0;
-    channel2prog[i] = -1;
+    channel2prog[i] = 0; /* [SS] 2023-06-25-8/
     channel2nnotes[i] = 0;
     chnactivity[i] = 0; /* [SS] 2018-02-02 */
     }
@@ -474,7 +483,7 @@ if (nprogs > 0) output_progs_data();
 else {
   for (i=0;i<17;i++) 
     if(chnactivity[i] > 0) 
-       progactivity[channel2prog[i]] = chnactivity[i];
+       progactivity[channel2prog[i]] += chnactivity[i]; /* [SS] 2023-06-25 */
   output_progs_data();
   }
 
@@ -575,7 +584,10 @@ for (i=1;i<17;i++) {
    printf("trkinfo ");
    printf("%d %d ",i,trkdata.program[i]); /* channel number and program*/
    printf("%d %d ",trkdata.notecount[i],trkdata.chordcount[i]);
-   printf("%d %d ",trkdata.notemeanpitch[i], trkdata.notelength[i]);
+   /* [SS] 2023-08-22 */
+   if (i != 10) printf("%d %d ",trkdata.notemeanpitch[i], trkdata.notelength[i]);
+   else 
+     printf("-1  0 ");
    printf("%d %d ",trkdata.cntlparam[i],trkdata.pressure[i]); /* [SS] 2022-03-04 */
    printf("%d %d",trkdata.quietTime[i],trkdata.rhythmpatterns[i]);
    trkdata.quietTime[i] = 0;
@@ -619,7 +631,9 @@ int chan, pitch, vol;
  int delta;
  int barnum;
  int unit;
+ int dithermargin; /* [SS] 2023-08-22 */
 
+ dithermargin = unitDivision/2 - 1; 
  if (vol == 0) {
     /* treat as noteoff */
     stats_noteoff(chan,pitch,vol);
@@ -652,7 +666,7 @@ int chan, pitch, vol;
 	 barChn[chan].rhythmPattern = 0;
          barChn[chan].activeBarNumber = barnum;
     }
-  unit = (Mf_currtime % divisionsPerBar)/unitDivision;
+  unit = ((Mf_currtime+dithermargin) % divisionsPerBar)/unitDivision;
   //printf("unit = %d pattern = %d \n",unit,barChn[chan].rhythmPattern);
   barChn[chan].rhythmPattern = barChn[chan].rhythmPattern |= (1UL << unit);
   }
@@ -721,7 +735,7 @@ if (trkdata.program[chan+1] != 0) {
   printf("program %d %d\n",chan+1,program);
   trkdata.program[chan+1] = program;
   }
-  if (channel2prog[chan+1]== -1) channel2prog[chan+1] = program;
+  if (channel2prog[chan+1]== 0) channel2prog[chan+1] = program; /* [SS] 2023-06-25*/
 }
 
 
@@ -953,7 +967,6 @@ for (i = 0; i < lastEvent; i++) {
 }
 
 void drumpattern (int perc) {
-unsigned char drumpat[1000];
 int i;
 int channel;
 int pitch;
@@ -963,7 +976,7 @@ int quarter;
 int remainder;
 int part;
 quarter = division/4;
-for (i = 0; i<1000; i++) drumpat[i] = 0;
+for (i = 0; i<8000; i++) drumpat[i] = 0;
 for (i = 0; i <lastEvent; i++) {
   channel = midievents[i].channel;
   if (channel != 9) continue;
@@ -973,12 +986,92 @@ for (i = 0; i <lastEvent; i++) {
   index = onset/division;
   remainder = onset % division;
   part = remainder/quarter;
+  if (index >= 8000) {printf("index too large in drumpattern\n");
+	              break;
+                      }
+  part = 3 - part; /* order the bits going left to right */
   drumpat[index] = drumpat[index] |= 1 << part;
   }
+}
+
+void dualDrumPattern (int perc1, int perc2) {
+int i;
+int channel;
+int pitch;
+int onset;
+int index;
+int quarter;
+int remainder;
+int part;
+quarter = division/4;
+for (i = 0; i<8000; i++) drumpat[i] = 0;
+for (i = 0; i <lastEvent; i++) {
+  channel = midievents[i].channel;
+  if (channel != 9) continue;
+  pitch = midievents[i].pitch;
+  if (pitch != perc1  && pitch != perc2) continue;
+  onset = midievents[i].onsetTime;
+  index = onset/division;
+  if (index >= 8000) {printf("index too large in drumpattern\n");
+	              break;
+                      }
+  remainder = onset % division;
+  part = remainder/quarter;
+  part = 3 - part; /* order the bits from left to right */
+  if (pitch == perc1) drumpat[index] = drumpat[index] |= 1 << part;
+  else drumpat[index] = drumpat[index] |= 16 * (1 << part);
+  }
+}
+
+void pitchClassAnalysis () {
+int i;
+int channel;
+int pitch;
+for (i=0;i<12;i++) pitchclass_activity[i] = 0;
+for (i = 0; i < lastEvent; i++) {
+  channel = midievents[i].channel;
+  if (channel == 9) continue;
+  pitch = midievents[i].pitch;
+  pitchclass_activity[pitch % 12]++;
+  }
+}
+
+void output_perc_pattern (int i) {
+int left,right;
+left = i/16;
+right = i % 16;
+printf(" (%d.%d)",left,right);
+}
+
+
+void drumPatternHistogram () {
+int i;
+for (i=0;i<256;i++) {
+ histogram[i] = 0;
+}
+for (i=0;i<lastBeat;i++) {
+  histogram[drumpat[i]]++;
+  }
+for (i=1;i<256;i++) {
+  if (histogram[i] > 0) {
+	  printf(" %d",i);
+	  output_perc_pattern(i);
+	  printf(" %d ", histogram[i]);
+         }
+    }
+  printf("\n");
+}
+
+
+void output_drumpat () {
+int i;
 for (i=0;i<lastBeat;i++) printf("%d ",drumpat[i]);
+/*for (i=0;i<lastBeat;i++) output_perc_pattern(drumpat[i]);*/
 printf("\n");
 }
 
+
+
 void percsummary () {
 int i;
 int bassindex,snareindex;
@@ -1002,10 +1095,22 @@ if (bassmax && snaremax) {
 	printf("snare %d %d\n",snareindex,snaremax);
    } else {
         printf("missing bass or snare\n");
+	return;
    }
-			
+dualDrumPattern(bassindex,snareindex);
 }
 
+void outputPitchClassHistogram() {
+int i;
+float activity;
+activity = 0;
+for (i=0;i<12;i++) activity += (float) pitchclass_activity[i];
+for (i=0;i<11;i++) printf("%5.3f,",pitchclass_activity[i]/activity);
+printf("%5.3f",pitchclass_activity[11]/activity);
+printf("\n");
+}
+
+
   
 void corestatsOutput() {
 printf("%d\t%d\t%d\n", division,lastEvent,lastBeat);
@@ -1129,18 +1234,41 @@ int argc;
 	  stats = 0;
           }
 
-  arg = getarg("-percanalysis",argc,argv);
+  arg = getarg("-panal",argc,argv);
   if (arg != -1) {
 	  percanalysis = 1;
 	  stats = 0;
           }
 
-  arg = getarg("-percpattern",argc,argv);
+  arg = getarg("-ppat",argc,argv);
   if (arg != -1) {
 	  percpattern = 1;
 	  stats = 0;
           }
 
+  arg = getarg("-ppatfor",argc,argv);
+  if (arg != -1) {
+	  percpatternfor = 1;
+	  stats = 0;
+	  if (arg != -1 && arg <argc) {
+		  percnum = readnum(argv[arg]);
+	          printf("percnum = %d\n",percnum);
+                  }
+          }
+
+  arg = getarg("-ppathist",argc,argv);
+  if (arg != -1) {
+	  percpatternhist = 1;
+	  stats = 0;
+          }
+
+  arg = getarg("-pitchclass",argc,argv);
+  if (arg != -1) {
+	  pitchclassanalysis = 1;
+	  stats = 0;
+          }
+
+
   arg = getarg("-o",argc,argv);
   if ((arg != -1) && (arg < argc))  {
     outhandle = efopen(argv[arg],"w");  /* open output abc file */
@@ -1163,12 +1291,15 @@ int argc;
     printf("midistats filename <options>\n");
     printf("         -corestats\n");
     printf("         -pulseanalysis\n");
-    printf("         -percanalysis\n");
-    printf("         -percpattern\n");
+    printf("         -panal\n");
+    printf("         -ppat\n");
+    printf("         -ppatfor\n");
+    printf("         -ppathist\n");
+    printf("         -pitchclass\n");
     printf("         -ver version number\n");
     printf("         -d <number> debug parameter\n");
     printf(" The input filename is assumed to be any string not\n");
-    printf(" beginning with a - (hyphen). It may be placed anywhere.\n");
+    printf(" beginning with a - (hyphen).\n");
     exit(0);
   };
   return arg;
@@ -1202,12 +1333,25 @@ if (percanalysis) {
         }
         printf("\n");
       }
-if (percanalysis) drumpattern(40);
 if (percpattern) {
     drumanalysis();
     percsummary();
+    output_drumpat();
+    }
+if (percpatternfor) {
+    drumpattern(percnum);
+    output_drumpat();
+    }
+if (percpatternhist) {
+    drumanalysis();
+    percsummary();
+    drumPatternHistogram();
     }
 if (corestats) corestatsOutput();
+if (pitchclassanalysis) {
+   pitchClassAnalysis();
+   outputPitchClassHistogram(); 
+   }
 }
 
 
@@ -1221,6 +1365,8 @@ int argc;
 
   arg = process_command_line_arguments(argc,argv);
   if(stats == 1)  midistats(argc,argv);
-  if(pulseanalysis || corestats || percanalysis || percpattern) loadEvents();
+  if(pulseanalysis || corestats || percanalysis ||\
+    percpatternfor || percpattern || percpatternhist ||\
+    pitchclassanalysis) loadEvents();
   return 0;
 }