Let the paintings play

Paola Gervasio, Alfio Quarteroni, Daniele Cassani

Abstract


In this paper we introduce a mathematical method to extract similarities between paintings and music tracks. Our approach is based on the digitalization of both paintings and music tracks by means of finite expansions in terms of orthogonal basis functions (with both Fourier and wavelet bases). The best fit between a specific painting and a sample of music tracks from a given composer is achieved via an $L^2$ projection upon a finite dimensional subspace.
Several examples are provided for the analysis of a collection of works of art by the Italian artist Marcello Morandini. Finally we have developed an original applet that implements the process above and which can be freely downloaded from the site https://github.com/pgerva/playing-paintings.git

Full Text:

PDF

References


F. Alías, J. Socoró, d X. Sevillano, A review of physical and perceptual feature extraction techniques for speech, music and environmental sounds, Applied Sciences 6 (2016), no. 5.

J. Alm, J. Walker, Time-frequency analysis of musical instruments, SIAM Review 44 (2002), no. 3, 457-476.

A. T. Cemgil, B. Kappen, D. Barber, Generative model based polyphonic music transcription, In: Proceedings of the 2003 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (2003), 181-184.

J. W. Cooley, J. W. Tukey, An algorithm for the machine calculation of complex Fourier series, Mathematics of Computation 19 (1965), 297-301.

I. Daubechies, Ten Lectures on Wavelets, SIAM, Philadelphia, PA, 61, 1992. Retrieved from https://doi.org/10.1137/1.9781611970104

M. Fitzpatrick, Create GUI Applications with Python & Qt5 (PySide2 Edition), 5th ed., 2022. Retrieved from https://www.pythonguis.com/pyside2-book/

Fondazione Marcello Morandini, Varese, Italy, 2021. Retrieved from https://www.fondazionemarcellomorandini.com

P. Gervasio, A Python App for Playing Paintings, 2022. Retrieved from https://github.com/pgerva/playing-paintings.git

R. C. Gonzalez, R. E. Woods, Digital Image Processing, 3rd ed., Pearson Prentice Hall, Upper Saddle River, NJ, 2008.

C. Ishi, O. Chatot, H. Ishiguro, N. Hagita, "Evaluation of a music-based real-time sound localization of multiple sound sources in real noisy environments," in Proceedings of the 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (2009), 2027-2032.

R. Kronland-Martinet, Wavelet transform for analysis, synthesis, and processing of speech and music sounds, Computer Music Journal 12 (1988), no. 4, 11-20.

W. Lang, K. Forinash, Time-frequency analysis with the continuous wavelet transform, American Journal of Physics 66 (1998), no. 9, 794-797.

T. Li, M. Ogihara, "Content-based music similarity search and emotion detection," in Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 5 (2004), V-705-V-708.

S. Mallat, A Wavelet Tour of Signal Processing, Academic Press, San Diego, CA, 1998.

M. Mannone, Dalla musica all'immagine, dall'immagine alla musica, relazioni matematiche fra composizione musicale e arte figurative, Compostampa, Palermo, (2011).

M. Mannone, Networks of music and images, Gli Spazi della Musica 2 (2017). Retrieved from http://www.ojs.unito.it/index.php/spazidellamusica

M. Mannone, Introduction to gestural similarity in music: An application of category theory to the orchestra, Journal of Mathematics and Music 12 (2018), no. 2, 63-87.

M. Mannone, A musical reading of a contemporary installation and back: Mathematical investigations of patterns in Qwalala, Journal of Mathematics and Music 16 (2022), no. 1, 80-96.

M. Mannone, F. Favali, B. Di Donato, L. Turchet, Quantum GestART: Identifying and applying correlations between mathematics, art, and perceptual organization, Journal of Mathematics and Music 15 (2021), no. 1, 62-94.

G. Mazzola, Synthesis, SToA 1001.90, Zurich, 1990.

G. Mazzola, The Topos of Music: Geometric Logic of Concepts, Theory, and Performance, Birkhäuser, Basel, 2002.

G. Mazzola, M. Mannone, Y. Pang, Cool Math for Hot Music: A First Introduction to Mathematics for Music Theorists, Springer International Publishing, 2016.

M. Meneguzzo (ed.), Marcello Morandini. Catalogo Ragionato, Skira, Milan, 2020.

D. Payling, Visual Music Composition with Electronic Sound and Video, unpublished doctoral dissertation, Sta_ordshire University, 2014.

G. Peeters, B. Giordano, P. Susini, N. Misdariis, S. McAdams, The Timbre Toolbox: Extracting audio descriptors from musical signals, Journal of the Acoustical Society of America 130 (2011), no. 5, 2902-2916.

R. Polikar, The Wavelet Tutorial, 300 (2003), no. 561.

Qt for Python Project (ed.), PySide2 5.15.2.1, 2022. Retrieved from https://pypi.org/project/PySide2/

G. Santini, Synesthesizer: Physical modelling and machine learning for a color-based synthesizer in virtual reality, In: M. Montiel, F. Gomez-Martin, and O. A. Agustín-Aquino (eds.), Mathematics and Computation in Music, Springer International Publishing, (2019), 229-235.

G. Sharma, K. Umapathy, S. Krishnan, "Trends in audio signal feature extraction methods," Applied Acoustics 158 (2020).

B. Su, S.-K. Jeng, Multi-timbre chord classification using wavelet transform and self-organized map neural networks, In: Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP) 5 (2001), 3377-3380.

C. Tsiourti, A. Weiss, K. Wac, and M. Vincze, Multimodal integration of emotional signals from voice, body, and context: Effects of (in)congruence on emotion recognition and attitudes towards robots, International Journal of Social Robotics 11 (2019), 555-573.

Wikipedia, Marcello Morandini, (2021). Retrieved from https://en.wikipedia.org/wiki/Marcello Morandini(Last modifed, 13 June 2022, at 6:30 (UTC))

I. Xenakis, Musiques Formelles: Nouveaux Principes Formels de Composition Musicale, Richard-Masse, Paris, 1963.

S. Zhao, Y. Li, X. Yao, W. Nie, P. Xu, J. Yang, K. Keutzer, Emotion-based end-to-end matching between image and music in valence-arousal space, arXiv (2020), arXiv:2009.05103. Retrieved from https://arxiv.org/abs/2009.05103

Y. Zhou, Z. Wang, C. Fang, T. Bui, T. L. Berg, Visual to sound: Generating natural sound for videos in the wild, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018), 3550-3558.




DOI: https://doi.org/10.52846/ami.v53i1.2412