TY - JOUR
T1 - An intelligent multimedia information system for multimodal content extraction and querying
AU - Yazici, Adnan
AU - Koyuncu, Murat
AU - Yilmaz, Turgay
AU - Sattari, Saeid
AU - Sert, Mustafa
AU - Gulen, Elvan
N1 - Funding Information:
This work is supported by the research grants from TUBITAK with the grant numbers “MFAG-114R082”. We thank to all of previous researchers of Multimedia DB Lab. at METU and Ahmet Cosar, who have contributed to this research.
Publisher Copyright:
© 2017, Springer Science+Business Media New York.
PY - 2018/1/1
Y1 - 2018/1/1
N2 - This paper introduces an intelligent multimedia information system, which exploits machine learning and database technologies. The system extracts semantic contents of videos automatically by using the visual, auditory and textual modalities, then, stores the extracted contents in an appropriate format to retrieve them efficiently in subsequent requests for information. The semantic contents are extracted from these three modalities of data separately. Afterwards, the outputs from these modalities are fused to increase the accuracy of the object extraction process. The semantic contents that are extracted using the information fusion are stored in an intelligent and fuzzy object-oriented database system. In order to answer user queries efficiently, a multidimensional indexing mechanism that combines the extracted high-level semantic information with the low-level video features is developed. The proposed multimedia information system is implemented as a prototype and its performance is evaluated using news video datasets for answering content and concept-based queries considering all these modalities and their fused data. The performance results show that the developed multimedia information system is robust and scalable for large scale multimedia applications.
AB - This paper introduces an intelligent multimedia information system, which exploits machine learning and database technologies. The system extracts semantic contents of videos automatically by using the visual, auditory and textual modalities, then, stores the extracted contents in an appropriate format to retrieve them efficiently in subsequent requests for information. The semantic contents are extracted from these three modalities of data separately. Afterwards, the outputs from these modalities are fused to increase the accuracy of the object extraction process. The semantic contents that are extracted using the information fusion are stored in an intelligent and fuzzy object-oriented database system. In order to answer user queries efficiently, a multidimensional indexing mechanism that combines the extracted high-level semantic information with the low-level video features is developed. The proposed multimedia information system is implemented as a prototype and its performance is evaluated using news video datasets for answering content and concept-based queries considering all these modalities and their fused data. The performance results show that the developed multimedia information system is robust and scalable for large scale multimedia applications.
KW - Multimedia applications
KW - Multimedia data fusion
KW - Multimedia databases
KW - Multimedia information retrieval
KW - Multimedia platform
KW - Multimedia querying
KW - Prototype multimedia system
KW - Semantic content extraction
KW - Video data
UR - http://www.scopus.com/inward/record.url?scp=85011317368&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85011317368&partnerID=8YFLogxK
U2 - 10.1007/s11042-017-4378-6
DO - 10.1007/s11042-017-4378-6
M3 - Article
AN - SCOPUS:85011317368
SN - 1380-7501
VL - 77
SP - 2225
EP - 2260
JO - Multimedia Tools and Applications
JF - Multimedia Tools and Applications
IS - 2
ER -