ISRG Logo

interactive systems

LIT Logo.

Research

Machine Listening & Perception

At ISRG, our machine-listening research involves the modelling of human auditory perception, and codifying these models using machine-learning approaches. This research is relevant in applications dependent on acquiring detailed, acoustically-informed analyses of complex sound-signals. One of the primary outputs of this research will be its deployment on handheld devices for mechanical-fault detection (acoustic diagnostics). Future Industry 4.0 implementations will target the development of intelligent, in-situ machine interfaces that will possess capabilities to accurately resolve auditory cues presented by human-users in acoustically-chaotic and reverberant environments.


Autonomous Sound-Scene Classification

When implemented on a large-scale network of sensors, our machine-listening research enables detailed sound-scene analysis and classification of urban and rural soundscapes. This technology functions as a backbone for developers and local authorities to design mobile/wearable apps that would benefit from detailed, accurate descriptors of objects and events based on acoustic-signature emissions.


Assistive Listening & Hearable Design

Our perceptual models contribute to an emerging technological trend - hearables. These devices, which focus on providing auditory services and enhancements in a wearable form-factor, require a level of acoustic precision that can only be provided by in-depth computational auditory-scene analysis. Our research in this area also involves evaluating EEG data of users presented with auditory stimuli. Our future goals in this regard include collaboration with colleagues in neuroscience to link brain electrical activity with hearable control parameters.


Immersive Multimodal Computing

One of our primary research activities is determining perceptual and cognitive constraints when users interact with immersive, multimodal computers. This work involves understanding cross-modal interactions between visual and auditory stimuli; working memory limitations; and the characteristics inherent in attention mechanisms and other cognitive processes. The primary output of this research is the development of frameworks that guide interface developers through the process of presenting multidimensional data to users utilising both visual and auditory sensory streams.

Team

Background Image. Lightbulb in dark backdrop.

ResearchSTAFF

Dr Flaithri Neff (Director & PI)
Dr Malachy Ronan (PI)
Dr Eoghan Neff (PI)
Dr John Greenwood (PI)

Background Image. Hand holding compass app on smart phone.

PostGRADS

Ronan O'Dea
Rokaia Jedir
Joe Fitzpatrick
Tom Brophy
Mike Gavin
Ricard Pinsenschaum

Background Image. Stacks of academic books.

PartnerSHIPS

Dr Dónal Fitzpatrick (DCU)
Mr Dave Murphy (UCC)
Dr Ian Pitt (UCC)
Dr Linda Butler (Hibernia)
Dr Patrick Felicia (WIT)
Dr Tracey Mehigan (UCC)
NSAI TC02/SC03/WG01
AES - Ireland Section
Limerick City & County Council

Background Image. Camera on tripod.

IndustryLINKS

Dell Technologies
Emdalo Technologies
Logitech Inc
Ierne Ltd
Inplicity Ltd

Output

CONFERENCE PROCEEDINGS

Stream Segregation: Utilizing Harmonic Variance in Auditory Graphs. Fitzpatrick, J. and Neff, F. (2018 ). In Proceedings of SMC. Limassol, Cyprus, 4-7 July 2018.

Auditory Stream Disruption in Human Computer Interaction, O'Dea, R., Jedir, R., and Neff, F. (2018). In Proceedings of the 32nd British Human- Computer Interaction Conference, Belfast, Northern Ireland.


The Data-Driven Algorithmic Composer - Fitzpatrick, J and Neff, F. (2017). In Proceedings of Audio Mostly 17, London, United Kingdom, August 23- 26, 2017.

Sonification playback rates during matching tasks of visualised and sonified EEG data - Gavin, M., Jedir, R. and Neff, F. (2016). Proceedings of the Interactive Audio Systems Symposium, September 23rd 2016, University of York, United Kingdom.

Evaluating Gesture Characteristics When Using a Bluetooth Handheld Mu- sic Controller - Pinsenschaum, R. and Neff, F. (2016). Proceedings of the Audio Mostly 2016, Interactive Institute Sweden. ACM Publishing, NY.

Evaluating the cueing efficiency of sound icon designs in a multimodal system during acoustically chaotic events - Brophy, T., Neff F., Fitzpatrick, D., Pitt, I., Noonan, C., Carroll, T. (2015). The International Conference on the Multimodal Experience of Music, 2015.

Accelerometer and Spatial Audio Technology: Making Touch-Screen Mobile Devices Accessible - Neff, F., Mehigan, T., and Pitt, I.J. (2010), 12th Inter- national Conference on Computers Helping People with Special Needs, July 14-16, 2010, Vienna University of Technology, Austria.

Evaluation of Pause Intervals between Haptic/Audio Cues and Subsequent Speech Information - Kehoe, A., Neff, F., Pitt, I., (2008), 10th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI). 2-5 September 2008 Amsterdam, the Netherlands.

Recommendations for pause intervals between haptic/audio cues and sub- sequent speech information - Kehoe, A., Neff, F., Pitt, I., (2008), Irish Human-Computer Interaction Conference (I-HCI), Cork, Ireland, September 19-20th 2008.

Augmenting auditory interface design with human echolocation - Neff, F., Kehoe, A., Pitt, I., (2008), Irish Human-Computer Interaction Conference (I-HCI), Cork, Ireland, September 19-20th 2008.

A Study toward the Development of a Spatial, Non-Speech Auditory Interface for Trigonometric Problem Solving - Neff, F., Pitt, I., (2008), Proceedings of the 14th International Conference on Auditory Display, Paris, France, June 24-27, 2008.

Using Spatial Non-Speech Sound to Relay Mathematical Problems to Visually Disabled Students - Neff, F., Pitt, I., (2008), International Technology, Education and Development Conference (INTED), Valencia 2008.

A Cognitive Model of Irish Traditional Fiddle Playing in the Development of a Teaching Aid for Improvisation - Neff, F., Neff, E., Pitt, I., (2007), 3rd International Conference on Automated Production of Cross Media Content for Multi-Channel Distribution (AXMEDIS), i-maestro workshops, Barcelona 2007.

A Consideration of Perceptual Interaction in an Auditory Prolific Mobile Device, Spatial Audio for Mobile Devices - Neff, F., Pitt, I., Keheo, A., (2007), Spatial Audio for Mobile Devices, Mobile HCI 2007, 9th International Conference on Human Interaction with Mobile Devices and Services, Singapore 2007.

Improving Accessibility of User Assistance Material through use of Speech Technology and Auxiliary Display Devices - Kehoe, A., Neff, F. and Pitt, I.J. (2007), The Accessible Design in the Digital World Conference, York, 2007.

Modifications to a Speech-Enabled User Assistance System Based on Pilot Study Results - Kehoe, A., Neff, F. and Pitt, I.J. (2007), ACM International Conference on Design of Communication (SIGDOC), El Paso, Texas, USA, October 22-24th 2007.

User Modelling to Support the Development of an Auditory Help System - Neff, F., Kehoe, A. and Pitt, I.J. (2007), International Conference on Text, Speech and Dialogue, Pilsen, Czech Republic, 3-7th September 2007.

Extending traditional user assistance systems to support an auditory interface - Kehoe, A., Neff, F. and Pitt, I.J. (2007), Artificial Intelligence and Applications (AIA 2007), Innsbruck, Austria, 12-14 February 2007.

Using Speech Technology to provide Context-Specific Help in Games - Ke- hoe, A., Neff, F. and Pitt, I.J. (2006), Computer Games (CGames 2006), Dublin, Ireland, 22-24th November 2006.

Design and Implementation of an Auditory Help System for Computer Games - Kehoe, A., Neff, F. and Pitt, I.J. (2006), Game Design and Technology Workshop, Liverpool, November 2006.

Close

JOURNAL ARTICLES

A System to Assist Visually Disabled Musicians to Participate in Orchestras - Brophy, T., Neff, F., Noonan, C., Pitt, I., and Fitzpatrick, D. (2015). The Journal on Technology and Persons with Disabilities, 160.

Use of Voice Input to Enhance Cursor Control in Mainstream Gaming Ap- plications - Kehoe, A., Neff, F.,and Pitt, I.J. (2009), in Universal Access in the Information Society (UAIS), special issue on ”Vocal Interaction: Beyond Traditional Automatic Speech Recognition”, Harada, S., Kurniawan, S. and Sporka, A.J. (Eds), ISSN 1615-5289.

Close

BOOK CHAPTERS

Using Spatial Audio in Game Technology for Expressing Mathematical Problems to Blind Students - Neff, F., and Pitt, I.J. (2011), in the Handbook of Research on Improving Learning and Motivation through Educational Games: Multidisciplinary Approaches, Felicia, P. (Ed.), IGI-Global, USA.

Spatial Sound for Computer Games and Virtual Reality - Murphy, D. and Neff, F. (2011), in Game Sound Technology and Player Interaction: Con- cepts and Developments, Grimshaw, M (Ed.), IGI- Global, USA.

Multimodal Cues: Exploring Pause Intervals between Haptic/Audio Cues and Subsequent Speech Information - Kehoe, A., Neff, F., and Pitt, I.J. (2010), in Multimodality in Mobile Computing and Mobile Devices: Meth- ods for Adaptable Usability, Kurkovsky, S. (Ed.), IGI-Global, USA.

Considering the Perceptual Implications of Auditory Rich Content on the Web - Neff, F., Pitt, I., (2009/2010), in Integrating Usability Engineer- ing For Designing The Web Experience: Methodologies And Principles, Spiliotopoulus, T., Papadopoulou, P., Martakos, D., and Kouroupetroglou, G. (eds.), IGI- Global, USA.

Close

POSTERS & PRESENTATIONS

Synergising the science of computing with pedagogy through a process interaction matrix approach - Butler, L and Neff. F (2019). Presented at the Educational Studies Association of Ireland Conference, Sligo, Ireland. 11th-13th April 2019.

A Review of the Impact of Auditory Content on Reward Systems in Game- Based Learning. Fitzpatrick, J. and Neff, F. (2019). Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th-28th June, 2019.

Integrating pedagogical and HCI principles in the design of game-based learning environments. Butler, L and Neff, F. (2019). Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th-28th June, 2019.

A Framework for the Effective Design of Auditory Notifications in Game-Based Learning Environments. O'Dea, R and Neff, F. (2019). Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th-28th June, 2019.

User-Centred Game Based Learning: The Role of Working Memory Performance during Multimodal Interaction. Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th- 28th June, 2019.

The Effects of Music on Episodic Memory - Jedir, R and Neff, F. (2018). Poster at 15th International Conference on Music Perception and Cognition 10th triennial conference of the European Society for the Cognitive Sciences of Music, Graz, Austria. 23-28 July 2018.

Associations Between Episodic Memory and Auditory Perceptual Process- ing due to Cognitive Interactions in Integrative Emotional Centres, Jedir, R. and Neff, F. (2018). Presented at 15th International Conference on Music Perception and Cognition, University of Graz, Austria.

Problematizing Second Language (L2) Learning using Emerging VR Systems - Butler, L and Neff, F. (2015), The Ireland-Canada International Symposium - Higher Education in Transformation 2015, Dublin Castle, Ireland.

Investigating the Potential of Human Echolocation in Virtual Sonic Trigonometry - Neff, F., Pitt, I., (2008), Acoustics08, Paris 2008.

Close

Projects

  • Acoustically-Informed Smart Hearable Technology. Image comprises a word-cloud with relevant terminolgy. These are: AISHT; Acoustically-Informed; Noise-Mapping; Smart-Hearables; soundscapes wearable mobile edge-gateways cloud realtime acoustic-analysis signal-processing; smart-audio-adjustments noise-control acoustic-infrastructure sensors signal-transcoding privacy-layer security-layer audio-networks scene-classification.
    AISHT: Acoustically-Informed Smart Hearable Technology
  • Intelligent Sonic Signature Analysis. Image comprises a word-cloud with relevant terminology. These are: ISSA; Sonic-Signature Analysis Intelligent; machine-faults diagnostics acoustic-sensors active-monitoring; acoustic-signature acoustic-profile sonic-schematics reverb-filters response-curves acoustic-coupling AI ML machine-listening early-fault-finding fault-predictions acoustic-warnings.
    ISSA: Intelligent Sonic Signature Analysis.
  •  Heritage Auralisation and Acoustic Archiving. Image comprises a word-cloud with relevant terminology. These are: HA-A-A-A; Acoustic-Archiving Auralisation Heritage; mixed-reality augmented-reality virtual-reality 3D-audio acoustic-analysis binaural MR AR VR spatialisation; convolution-reverb waterfall-plots reference-microphones dodecahedron-speaker impulse-response acoustic-profiling psychoacoustics spatial-cues reverb-characteristics.
    Ha-a-a-a: Heritage Auralisation and Acoustic Archiving
  •  Artificial Intelligence for Independence. Image comprises a word-cloud with relevant terminology. These are: AI4I; artificial intelligence independence; accessibility equality transparency independent-living smart-communities; technology-engagement technology-enhanced user-centred automation flexibility adaptable multimodal.
    AI4I: Artificial Intelligence for Independence
  • Future Education Augmented with Smart Apps. Image comprises a word-cloud with relevant terminology. These are: FEASA; learning-augmentation smart-apps future-education knowledge; personalisation adaptable-systems assistive-learning edu-apps web-apps; perception cognition HCI learning-models learner-centred multimodal-interfaces artificial-intelligence blended-learning online-learning.
    FEASA: Future Education Augmented with Smart Apps

Contact

If you are interested in our research, or would like to talk about collaborating with us, please contact Dr Flaithri Neff by ObscureMyEmail.

Acknowledgements

Site developed using Rapidweaver: https://www.realmacsoftware.com/rapidweaver/

ISRG icon adapted from an SVG file created by Rapidweaver stacks developer, Big White Duck: https://www.bigwhiteduck.com

Machine Listening & Perception Background Image: photo by Felix Berger on Unsplash
Autonomous Sound-Scene Classification Background Image: photo by Jakob Owens on Unsplash
Active Listening & Hearable Design Background Image: photo by Mateo Abrahan on Unsplash
Immersive Multimodal Computing Background Image: photo by Samuel Zeller on Unsplash

Research Staff Background Image: photo by DXL on Unsplash
Postgrads Background Image: photo by Robert Penaloza on Unsplash
Partnerships Background Image: photo by Jaredd Craig on Unsplash
Industry Links Background Image: photo by Vincent Delegge on Unsplash

Unsplash images: https://unsplash.com