An intelligent multimedia information system for multimodal content extraction and querying

Adnan Yazici, Murat Koyuncu, Turgay Yilmaz, Saeid Sattari, Mustafa Sert, Elvan Gulen

Research output: Contribution to journalArticle

2 Citations (Scopus)

Abstract

This paper introduces an intelligent multimedia information system, which exploits machine learning and database technologies. The system extracts semantic contents of videos automatically by using the visual, auditory and textual modalities, then, stores the extracted contents in an appropriate format to retrieve them efficiently in subsequent requests for information. The semantic contents are extracted from these three modalities of data separately. Afterwards, the outputs from these modalities are fused to increase the accuracy of the object extraction process. The semantic contents that are extracted using the information fusion are stored in an intelligent and fuzzy object-oriented database system. In order to answer user queries efficiently, a multidimensional indexing mechanism that combines the extracted high-level semantic information with the low-level video features is developed. The proposed multimedia information system is implemented as a prototype and its performance is evaluated using news video datasets for answering content and concept-based queries considering all these modalities and their fused data. The performance results show that the developed multimedia information system is robust and scalable for large scale multimedia applications.

Original languageEnglish
Pages (from-to)2225-2260
Number of pages36
JournalMultimedia Tools and Applications
Volume77
Issue number2
DOIs
Publication statusPublished - Jan 1 2018

Fingerprint

Information systems
Semantics
Information fusion
Learning systems

Keywords

  • Multimedia applications
  • Multimedia data fusion
  • Multimedia databases
  • Multimedia information retrieval
  • Multimedia platform
  • Multimedia querying
  • Prototype multimedia system
  • Semantic content extraction
  • Video data

ASJC Scopus subject areas

  • Software
  • Media Technology
  • Hardware and Architecture
  • Computer Networks and Communications

Cite this

An intelligent multimedia information system for multimodal content extraction and querying. / Yazici, Adnan; Koyuncu, Murat; Yilmaz, Turgay; Sattari, Saeid; Sert, Mustafa; Gulen, Elvan.

In: Multimedia Tools and Applications, Vol. 77, No. 2, 01.01.2018, p. 2225-2260.

Research output: Contribution to journalArticle

Yazici, Adnan ; Koyuncu, Murat ; Yilmaz, Turgay ; Sattari, Saeid ; Sert, Mustafa ; Gulen, Elvan. / An intelligent multimedia information system for multimodal content extraction and querying. In: Multimedia Tools and Applications. 2018 ; Vol. 77, No. 2. pp. 2225-2260.
@article{8c585a7a74564200a28ef91545352117,
title = "An intelligent multimedia information system for multimodal content extraction and querying",
abstract = "This paper introduces an intelligent multimedia information system, which exploits machine learning and database technologies. The system extracts semantic contents of videos automatically by using the visual, auditory and textual modalities, then, stores the extracted contents in an appropriate format to retrieve them efficiently in subsequent requests for information. The semantic contents are extracted from these three modalities of data separately. Afterwards, the outputs from these modalities are fused to increase the accuracy of the object extraction process. The semantic contents that are extracted using the information fusion are stored in an intelligent and fuzzy object-oriented database system. In order to answer user queries efficiently, a multidimensional indexing mechanism that combines the extracted high-level semantic information with the low-level video features is developed. The proposed multimedia information system is implemented as a prototype and its performance is evaluated using news video datasets for answering content and concept-based queries considering all these modalities and their fused data. The performance results show that the developed multimedia information system is robust and scalable for large scale multimedia applications.",
keywords = "Multimedia applications, Multimedia data fusion, Multimedia databases, Multimedia information retrieval, Multimedia platform, Multimedia querying, Prototype multimedia system, Semantic content extraction, Video data",
author = "Adnan Yazici and Murat Koyuncu and Turgay Yilmaz and Saeid Sattari and Mustafa Sert and Elvan Gulen",
year = "2018",
month = "1",
day = "1",
doi = "10.1007/s11042-017-4378-6",
language = "English",
volume = "77",
pages = "2225--2260",
journal = "Multimedia Tools and Applications",
issn = "1380-7501",
publisher = "Springer Netherlands",
number = "2",

}

TY - JOUR

T1 - An intelligent multimedia information system for multimodal content extraction and querying

AU - Yazici, Adnan

AU - Koyuncu, Murat

AU - Yilmaz, Turgay

AU - Sattari, Saeid

AU - Sert, Mustafa

AU - Gulen, Elvan

PY - 2018/1/1

Y1 - 2018/1/1

N2 - This paper introduces an intelligent multimedia information system, which exploits machine learning and database technologies. The system extracts semantic contents of videos automatically by using the visual, auditory and textual modalities, then, stores the extracted contents in an appropriate format to retrieve them efficiently in subsequent requests for information. The semantic contents are extracted from these three modalities of data separately. Afterwards, the outputs from these modalities are fused to increase the accuracy of the object extraction process. The semantic contents that are extracted using the information fusion are stored in an intelligent and fuzzy object-oriented database system. In order to answer user queries efficiently, a multidimensional indexing mechanism that combines the extracted high-level semantic information with the low-level video features is developed. The proposed multimedia information system is implemented as a prototype and its performance is evaluated using news video datasets for answering content and concept-based queries considering all these modalities and their fused data. The performance results show that the developed multimedia information system is robust and scalable for large scale multimedia applications.

AB - This paper introduces an intelligent multimedia information system, which exploits machine learning and database technologies. The system extracts semantic contents of videos automatically by using the visual, auditory and textual modalities, then, stores the extracted contents in an appropriate format to retrieve them efficiently in subsequent requests for information. The semantic contents are extracted from these three modalities of data separately. Afterwards, the outputs from these modalities are fused to increase the accuracy of the object extraction process. The semantic contents that are extracted using the information fusion are stored in an intelligent and fuzzy object-oriented database system. In order to answer user queries efficiently, a multidimensional indexing mechanism that combines the extracted high-level semantic information with the low-level video features is developed. The proposed multimedia information system is implemented as a prototype and its performance is evaluated using news video datasets for answering content and concept-based queries considering all these modalities and their fused data. The performance results show that the developed multimedia information system is robust and scalable for large scale multimedia applications.

KW - Multimedia applications

KW - Multimedia data fusion

KW - Multimedia databases

KW - Multimedia information retrieval

KW - Multimedia platform

KW - Multimedia querying

KW - Prototype multimedia system

KW - Semantic content extraction

KW - Video data

UR - http://www.scopus.com/inward/record.url?scp=85011317368&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85011317368&partnerID=8YFLogxK

U2 - 10.1007/s11042-017-4378-6

DO - 10.1007/s11042-017-4378-6

M3 - Article

VL - 77

SP - 2225

EP - 2260

JO - Multimedia Tools and Applications

JF - Multimedia Tools and Applications

SN - 1380-7501

IS - 2

ER -