ISRG Logo

    interactive systems

    LIT Logo.

    Research

    Machine Listening & Perception

    At ISRG, our machine-listening research involves the modelling of human auditory perception, and codifying these models using machine-learning approaches. This research is relevant in applications dependent on acquiring detailed, acoustically-informed analyses of complex sound-signals. One of the primary outputs of this research will be its deployment on handheld devices for mechanical-fault detection (acoustic diagnostics). Future Industry 4.0 implementations will target the development of intelligent, in-situ machine interfaces that will possess capabilities to accurately resolve auditory cues presented by human-users in acoustically-chaotic and reverberant environments.

    Autonomous Sound-Scene Classification

    When implemented on a large-scale network of sensors, our machine-listening research enables detailed sound-scene analysis and classification of urban and rural soundscapes. This technology functions as a backbone for developers and local authorities to design mobile/wearable apps that would benefit from detailed, accurate descriptors of objects and events based on acoustic-signature emissions.

    Assistive Listening & Hearable Design

    Our perceptual models contribute to an emerging technological trend - hearables. These devices, which focus on providing auditory services and enhancements in a wearable form-factor, require a level of acoustic precision that can only be provided by in-depth computational auditory-scene analysis. Our research in this area also involves evaluating EEG data of users presented with auditory stimuli. Our future goals in this regard include collaboration with colleagues in neuroscience to link brain electrical activity with hearable control parameters.

    Immersive Multimodal Computing

    One of our primary research activities is determining perceptual and cognitive constraints when users interact with immersive, multimodal computers. This work involves understanding cross-modal interactions between visual and auditory stimuli; working memory limitations; and the characteristics inherent in attention mechanisms and other cognitive processes. The primary output of this research is the development of frameworks that guide interface developers through the process of presenting multidimensional data to users utilising both visual and auditory sensory streams.

    Team

    Background Image. Lightbulb in dark backdrop.

    ResearchSTAFF

    Dr Flaithri Neff (Director & PI)
    Dr Malachy Ronan (PI)
    Dr Eoghan Neff (PI)
    Dr John Greenwood (PI)

    Background Image. Hand holding compass app on smart phone.

    PostGRADS

    Ronan O'Dea
    Rokaia Jedir
    Joe Fitzpatrick
    Tom Brophy
    Mike Gavin
    Ricard Pinsenschaum

    Background Image. Stacks of academic books.

    PartnerSHIPS

    Dr Dónal Fitzpatrick (DCU)
    Mr Dave Murphy (UCC)
    Dr Ian Pitt (UCC)
    Dr Linda Butler (Hibernia)
    Dr Patrick Felicia (WIT)
    Dr Tracey Mehigan (UCC)
    NSAI TC02/SC03/WG01
    AES - Ireland Section
    Limerick City & County Council

    Background Image. Camera on tripod.

    IndustryLINKS

    Dell Technologies
    Emdalo Technologies
    Logitech Inc
    Ierne Ltd
    Inplicity Ltd

    Output

    Conference Proceedings

    CONFERENCE PROCEEDINGS

    Stream Segregation: Utilizing Harmonic Variance in Auditory Graphs. Fitzpatrick, J. and Neff, F. (2018 ). In Proceedings of SMC. Limassol, Cyprus, 4-7 July 2018.

    Auditory Stream Disruption in Human Computer Interaction, O'Dea, R., Jedir, R., and Neff, F. (2018). In Proceedings of the 32nd British Human- Computer Interaction Conference, Belfast, Northern Ireland.

    The Data-Driven Algorithmic Composer - Fitzpatrick, J and Neff, F. (2017). In Proceedings of Audio Mostly 17, London, United Kingdom, August 23- 26, 2017.

    Sonification playback rates during matching tasks of visualised and sonified EEG data - Gavin, M., Jedir, R. and Neff, F. (2016). Proceedings of the Interactive Audio Systems Symposium, September 23rd 2016, University of York, United Kingdom.

    Evaluating Gesture Characteristics When Using a Bluetooth Handheld Mu- sic Controller - Pinsenschaum, R. and Neff, F. (2016). Proceedings of the Audio Mostly 2016, Interactive Institute Sweden. ACM Publishing, NY.

    Evaluating the cueing efficiency of sound icon designs in a multimodal system during acoustically chaotic events - Brophy, T., Neff F., Fitzpatrick, D., Pitt, I., Noonan, C., Carroll, T. (2015). The International Conference on the Multimodal Experience of Music, 2015.

    Accelerometer and Spatial Audio Technology: Making Touch-Screen Mobile Devices Accessible - Neff, F., Mehigan, T., and Pitt, I.J. (2010), 12th Inter- national Conference on Computers Helping People with Special Needs, July 14-16, 2010, Vienna University of Technology, Austria.

    Evaluation of Pause Intervals between Haptic/Audio Cues and Subsequent Speech Information - Kehoe, A., Neff, F., Pitt, I., (2008), 10th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI). 2-5 September 2008 Amsterdam, the Netherlands.

    Recommendations for pause intervals between haptic/audio cues and sub- sequent speech information - Kehoe, A., Neff, F., Pitt, I., (2008), Irish Human-Computer Interaction Conference (I-HCI), Cork, Ireland, September 19-20th 2008.

    Augmenting auditory interface design with human echolocation - Neff, F., Kehoe, A., Pitt, I., (2008), Irish Human-Computer Interaction Conference (I-HCI), Cork, Ireland, September 19-20th 2008.

    A Study toward the Development of a Spatial, Non-Speech Auditory Interface for Trigonometric Problem Solving - Neff, F., Pitt, I., (2008), Proceedings of the 14th International Conference on Auditory Display, Paris, France, June 24-27, 2008.

    Using Spatial Non-Speech Sound to Relay Mathematical Problems to Visually Disabled Students - Neff, F., Pitt, I., (2008), International Technology, Education and Development Conference (INTED), Valencia 2008.

    A Cognitive Model of Irish Traditional Fiddle Playing in the Development of a Teaching Aid for Improvisation - Neff, F., Neff, E., Pitt, I., (2007), 3rd International Conference on Automated Production of Cross Media Content for Multi-Channel Distribution (AXMEDIS), i-maestro workshops, Barcelona 2007.

    A Consideration of Perceptual Interaction in an Auditory Prolific Mobile Device, Spatial Audio for Mobile Devices - Neff, F., Pitt, I., Keheo, A., (2007), Spatial Audio for Mobile Devices, Mobile HCI 2007, 9th International Conference on Human Interaction with Mobile Devices and Services, Singapore 2007.

    Improving Accessibility of User Assistance Material through use of Speech Technology and Auxiliary Display Devices - Kehoe, A., Neff, F. and Pitt, I.J. (2007), The Accessible Design in the Digital World Conference, York, 2007.

    Modifications to a Speech-Enabled User Assistance System Based on Pilot Study Results - Kehoe, A., Neff, F. and Pitt, I.J. (2007), ACM International Conference on Design of Communication (SIGDOC), El Paso, Texas, USA, October 22-24th 2007.

    User Modelling to Support the Development of an Auditory Help System - Neff, F., Kehoe, A. and Pitt, I.J. (2007), International Conference on Text, Speech and Dialogue, Pilsen, Czech Republic, 3-7th September 2007.

    Extending traditional user assistance systems to support an auditory interface - Kehoe, A., Neff, F. and Pitt, I.J. (2007), Artificial Intelligence and Applications (AIA 2007), Innsbruck, Austria, 12-14 February 2007.

    Using Speech Technology to provide Context-Specific Help in Games - Ke- hoe, A., Neff, F. and Pitt, I.J. (2006), Computer Games (CGames 2006), Dublin, Ireland, 22-24th November 2006.

    Design and Implementation of an Auditory Help System for Computer Games - Kehoe, A., Neff, F. and Pitt, I.J. (2006), Game Design and Technology Workshop, Liverpool, November 2006.

    Close
    Journal Articles

    JOURNAL ARTICLES

    A System to Assist Visually Disabled Musicians to Participate in Orchestras - Brophy, T., Neff, F., Noonan, C., Pitt, I., and Fitzpatrick, D. (2015). The Journal on Technology and Persons with Disabilities, 160.

    Use of Voice Input to Enhance Cursor Control in Mainstream Gaming Ap- plications - Kehoe, A., Neff, F.,and Pitt, I.J. (2009), in Universal Access in the Information Society (UAIS), special issue on ”Vocal Interaction: Beyond Traditional Automatic Speech Recognition”, Harada, S., Kurniawan, S. and Sporka, A.J. (Eds), ISSN 1615-5289.

    Close
    Book Chapters

    BOOK CHAPTERS

    Using Spatial Audio in Game Technology for Expressing Mathematical Problems to Blind Students - Neff, F., and Pitt, I.J. (2011), in the Handbook of Research on Improving Learning and Motivation through Educational Games: Multidisciplinary Approaches, Felicia, P. (Ed.), IGI-Global, USA.

    Spatial Sound for Computer Games and Virtual Reality - Murphy, D. and Neff, F. (2011), in Game Sound Technology and Player Interaction: Con- cepts and Developments, Grimshaw, M (Ed.), IGI- Global, USA.

    Multimodal Cues: Exploring Pause Intervals between Haptic/Audio Cues and Subsequent Speech Information - Kehoe, A., Neff, F., and Pitt, I.J. (2010), in Multimodality in Mobile Computing and Mobile Devices: Meth- ods for Adaptable Usability, Kurkovsky, S. (Ed.), IGI-Global, USA.

    Considering the Perceptual Implications of Auditory Rich Content on the Web - Neff, F., Pitt, I., (2009/2010), in Integrating Usability Engineer- ing For Designing The Web Experience: Methodologies And Principles, Spiliotopoulus, T., Papadopoulou, P., Martakos, D., and Kouroupetroglou, G. (eds.), IGI- Global, USA.

    Close
    Posters & Presentations

    POSTERS & PRESENTATIONS

    Synergising the science of computing with pedagogy through a process interaction matrix approach - Butler, L and Neff. F (2019). Presented at the Educational Studies Association of Ireland Conference, Sligo, Ireland. 11th-13th April 2019.

    A Review of the Impact of Auditory Content on Reward Systems in Game- Based Learning. Fitzpatrick, J. and Neff, F. (2019). Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th-28th June, 2019.

    Integrating pedagogical and HCI principles in the design of game-based learning environments. Butler, L and Neff, F. (2019). Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th-28th June, 2019.

    A Framework for the Effective Design of Auditory Notifications in Game-Based Learning Environments. O'Dea, R and Neff, F. (2019). Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th-28th June, 2019.

    User-Centred Game Based Learning: The Role of Working Memory Performance during Multimodal Interaction. Presented at the Irish Game-Based Learning Conference (iGBL), Cork City, Ireland. 26th- 28th June, 2019.

    The Effects of Music on Episodic Memory - Jedir, R and Neff, F. (2018). Poster at 15th International Conference on Music Perception and Cognition 10th triennial conference of the European Society for the Cognitive Sciences of Music, Graz, Austria. 23-28 July 2018.

    Associations Between Episodic Memory and Auditory Perceptual Process- ing due to Cognitive Interactions in Integrative Emotional Centres, Jedir, R. and Neff, F. (2018). Presented at 15th International Conference on Music Perception and Cognition, University of Graz, Austria.

    Problematizing Second Language (L2) Learning using Emerging VR Systems - Butler, L and Neff, F. (2015), The Ireland-Canada International Symposium - Higher Education in Transformation 2015, Dublin Castle, Ireland.

    Investigating the Potential of Human Echolocation in Virtual Sonic Trigonometry - Neff, F., Pitt, I., (2008), Acoustics08, Paris 2008.

    Close

    Projects

    • Acoustically-Informed Smart Hearable Technology. Image comprises a word-cloud with relevant terminolgy. These are: AISHT; Acoustically-Informed; Noise-Mapping; Smart-Hearables; soundscapes wearable mobile edge-gateways cloud realtime acoustic-analysis signal-processing; smart-audio-adjustments noise-control acoustic-infrastructure sensors signal-transcoding privacy-layer security-layer audio-networks scene-classification.
      AISHT: Acoustically-Informed Smart Hearable Technology
    • Intelligent Sonic Signature Analysis. Image comprises a word-cloud with relevant terminology. These are: ISSA; Sonic-Signature Analysis Intelligent; machine-faults diagnostics acoustic-sensors active-monitoring; acoustic-signature acoustic-profile sonic-schematics reverb-filters response-curves acoustic-coupling AI ML machine-listening early-fault-finding fault-predictions acoustic-warnings.
      ISSA: Intelligent Sonic Signature Analysis.
    • Heritage Auralisation and Acoustic Archiving. Image comprises a word-cloud with relevant terminology. These are: HA-A-A-A; Acoustic-Archiving Auralisation Heritage; mixed-reality augmented-reality virtual-reality 3D-audio acoustic-analysis binaural MR AR VR spatialisation; convolution-reverb waterfall-plots reference-microphones dodecahedron-speaker impulse-response acoustic-profiling psychoacoustics spatial-cues reverb-characteristics.
      Ha-a-a-a: Heritage Auralisation and Acoustic Archiving
    • Artificial Intelligence for Independence. Image comprises a word-cloud with relevant terminology. These are: AI4I; artificial intelligence independence; accessibility equality transparency independent-living smart-communities; technology-engagement technology-enhanced user-centred automation flexibility adaptable multimodal.
      AI4I: Artificial Intelligence for Independence
    • Future Education Augmented with Smart Apps. Image comprises a word-cloud with relevant terminology. These are: FEASA; learning-augmentation smart-apps future-education knowledge; personalisation adaptable-systems assistive-learning edu-apps web-apps; perception cognition HCI learning-models learner-centred multimodal-interfaces artificial-intelligence blended-learning online-learning.
      FEASA: Future Education Augmented with Smart Apps

    Contact

    If you are interested in our research, or would like to talk about collaborating with us, please contact Dr Flaithri Neff by ObscureMyEmail.

    Acknowledgements

    Site developed using Rapidweaver: https://www.realmacsoftware.com/rapidweaver/

    ISRG icon adapted from an SVG file created by Rapidweaver stacks developer, Big White Duck: https://www.bigwhiteduck.com

    Machine Listening & Perception Background Image: photo by Felix Berger on Unsplash
    Autonomous Sound-Scene Classification Background Image: photo by Jakob Owens on Unsplash
    Active Listening & Hearable Design Background Image: photo by Mateo Abrahan on Unsplash
    Immersive Multimodal Computing Background Image: photo by Samuel Zeller on Unsplash

    Research Staff Background Image: photo by DXL on Unsplash
    Postgrads Background Image: photo by Robert Penaloza on Unsplash
    Partnerships Background Image: photo by Jaredd Craig on Unsplash
    Industry Links Background Image: photo by Vincent Delegge on Unsplash

    Unsplash images: https://unsplash.com

    © 2019 Flaithri Neff | ISRG, LIT.