Спросить
Войти
OZBEK STEMASINI YARATISH
Ismailov Alisher Shakirovich alisherismailov 1991 @gmail.com Andijon mashinasozlik instituti
Annotatsiya: Ushbu maqolada an&anaviy qoidalarga asoslangan tizimning simli o&xshashlik yondashuvidan iborat yangi kontekstsiz bepul stemmer taklif etiladi. Ushbu algoritmni gibrid algoritm deb atash mumkin. Bu tilga bog&liq algoritm. Kontekstdagi bepul stemmer degani, kontekstga asoslanmagan so&zni ishlatadigan stemmer degan ma&noni anglatadi, har bir kontekst uchun bunday qoida qo&llaniladi. An&anaviy kontekst bo&yicha erkin qoidaga asoslangan yondashuv yordamida so&zlarni olib tashlaganimizdan so&ng, u dinamik dasturlashning simli o&xshashlik funktsiyasini qo&llash orqali engib o&tilgan so&zlarning tagiga yoki tagiga tushishi mumkin. Satrlarning o&xshashligi funktsiyasini o&lchash uchun tahrirlash masofasidan foydalaniladi. O&chirilgan inflatsiya qilingan so&z matn bazasida mavjud bo&lgan so&zlar bilan taqqoslanadi. Minimal masofaga ega bo&lgan so&z, olib tashlangan inflatsiya qilingan so&zning o&rnini egallaydi, bu esa uning kelib chiqishiga olib keladi. Ushbu yondashuvda an&anaviy qoidalarga asoslangan tizim va korpusga asoslangan yondashuv tushunchalari keng qo&llaniladi. Ushbu algoritm o&zbek tili uchun sinovdan o&tkaziladi.
Kalit so&zlar: Stemming, Text-mining, algoritm, o&zbek tili.
DEVELOPMENT OF UZBEK STEMMING
Ismailov Alisher Shakirovich alisherismailov 1991 @gmail.com Andijon Machine Building Institute
Abstract: In this paper, a new context free stemmer is proposed which consists of the combination of traditional rule based system with string similarity approach. This algorithm can be called as hybrid algorithm. It is language dependent algorithm. Context free stemmer means that stemmer which stems the word that is not based on the context, for every context such rule is applied. After stripping the words using traditional context free rule based approach, it may over stem or under stem the inflected words which are overcome by applying string similarity function of dynamic programming. For measuring the string similarity function, edit distance will be used. The stripped inflected word is compared with the words stored in a text database available. That word having minimum distance is taken as the substitution
of the stripped inflected word which leads to the stem of it. The concept of traditional rule based system and corpus based approach is heavily used in this approach. This algorithm will be tested for Uzbek Language.
РАЗВИТИЕ УЗБЕКСКОГО СТЕММИНГА
Исмаилов Алишер Шакирович alisherismailov1991 @gmail.com Андижанский машиностроительный институт
Аннотация: Этот документ предлагает новый свободный от контекста стеммер, состоящий из подхода проводной аналогии системы, основанной на традиционных правилах. Этот алгоритм можно назвать гибридным алгоритмом. Это зависящий от языка алгоритм. Свободный терминатор в контексте означает терминатор, который использует неконтекстное слово, и такое правило применяется к каждому контексту. После того как мы удалили слова, используя свободный подход на основе правил в традиционном контексте, его можно отбросить или свести к словам, которые были преодолены с помощью функции подобия проводов динамического программирования. Расстояние редактирования используется для измерения функции сходства линий. Удаленное завышенное слово сравнивается со словами, присутствующими в текстовой базе данных. Слово с минимальным расстоянием заменяет удаленное завышенное слово, что приводит к его происхождению. В этом подходе широко используются понятия традиционной системы, основанной на правилах, и корпусного подхода. Этот алгоритм проверен на узбекском языке.
Misol uchun, "insonlar" so&zini tabiiy til jaroyoni(Natural Language processing) qidirsak, qidiruv tizimni ishlash jaroyonini sekinlashtirishi mumkin, agar so&zni o&zagini ajratib olib qidirsak tizimni ishlash jaroyoni tezlashadi. Stemming so&z jamlanmalari (text mining)da juda muhim o&rin tutadi. Stemming odatda kiritilgan so&zni o&zagani ajratib oladi va bu jarayon axborot qidiruvni ishlash jarayonini yaxshilaydi (Figure 1).
Oddiy So z
Algoritm Jarayoni Sterna (O zak)
a) so&zni o&zagani topishda natija kiritilgan so&zga bog&liq bo&lmagan bo&lishi mumkin(over-stemming).
b) so&zni prefiks va suffikslardan ajratib olishda so&z o&zagiga tegishli bo&lgan harfni ham o&chirib tashlanishi mumkin(under stemming).
Ushbu muammolar algoritmni yangi dasturlarda qo&llanganda qiyinchiliklar tu&gdirishi mumkin. O&zbek stema alrotimni rivonlantirishdaagi asosiy maqsad yuqorida keltirilgan muammolardan holi algotirm yaratishdir. Ilmiy tilda so&zni o&zagini topish lemmatizatsiya deb ataladi[25, 26, 27].
Stema mavzusida bir qancha ilmiy ishlar mavjud. Misol uchun, nemis, ispan, hind stemalarini keltirish mumkin. Hind stemasi bir nechta hind tillarida ishlaydi. Ular Tamil, Punjabi, Bengali, Gujarati, Hind va marati tillari. Shunga o&xshash arab stemalari ham mavjud. Har bir stema yaratuvchilarni o&zlarini tillarida ishlaydi.
[1] stemasi - arab tili stemasi hisoblanadi. Bu stema 5 usuldan foydalanib yaratilgan. Ulardan 4 tasi so&zdagi harflarni joylashishi usuli va beshinchi usul qoidalarga asoslangan tizimi (traditional rule based system). Arab stemasida qoidalarga asoslangan tuzatish algoritmi bilan birga ishlaganda yaxshi natija bergan.
[2] stema ham arab tili stemasi. Bu stema mashina o&rganish tizimidan foydalanilgan va juda yaxshi va aniq natija bergan. Bunday yaxshi natijaga erishish uchun ular Baye va ZeroR va algoritmlaridan foydalanganlar.
[3] - stema yengil stema va axborot qidiruv usullaridan foydalanib yaratilgan.
[4] - stema bir nechta stemming algoritmlari va solishtirish usullaridan foydalanilgan. Ushbu stemada Lovins va Poster stemalaridan foydalanilgan.
[5] - bu maqola stema algoritmini Ingliz tili uchun tayyorlangan va bu stema gibrid usulidan foydalanilgan. Gujarati (hind) va Arab algoritimlaridan foydalanilgan.
Taklif etilayaotgan modelda quyidagilarni hisobga oladi: Prefiks, suffiks va o&zak.
Suffiks deb so&zni o&zagidan keyingi qo&chimchalarga aytiladi.
Bu o&zbek so&zlarning oldingi qismidagi qo&shimchalar.
Ushbu algoritm uchun 700 ta o&zak so&zlardan foydalaniladi.
Bu algoritm klassik qoidalarga asoslangan tizim va solishtirish tizimlaridan foydalanib yaratiladi. 2-rasmda ko&rish mumkin.
So&zni Kiritsh
Qoidalarga Asoslangan Algoritm Yordamida So&zni O&zagini Ajratib Olish
Ajratib olingan so&z o&zagini ba&zadagi so&zlar bilan solishtirish
Natija (stem)
[24] - maqola 0.50 foiz aniqlikda ishlagani uchun biz ushbu maqolani eng yaxshi natija sifatida oldik. So&zni o&zagini ajratib olingandan so&ng uni ba& zadagi o&zak so&zlar bilan solishtirish yo&li bilan topiladi. So&zni o&zagini ajratib olishda ba&zida o&zakdan ortiqcha harf olinib ketishi yoki o&zakda ortiqcha harf qolib ketish
ehtimoli bor. Shuning uchun o&xshash so&zlar solishtirish(string similarity approach) usuli bilan algoritmda qoTlaniladi.
Foydalanilgan adabiyotlar
[1] Y. AI-Nashashibi, D. D. Neagu and Y. Ali, "Stemming Techniques for Arabic Words: A Comparative Study," 2nd International Conference on Computer Technology and Development (ICCTD), 2010, pp. 270-276.
[2] H. Mohammad, B. Zuhair, C. Keely and M. David, "An Arabic Stemming Approach Using Machine Learning with Arabic Dialogue System," ICGST AIML-11 Conference, Dubai, April 2011, pp. 9-16.
[3] L. S. Leah, B. Lisa and C. E. Margaret, "Improving Stem- ming for Arabic Information Retrieval: Light Stemming and Co-occurance Analysis," SIGIR, ACM, 11-15 August 2002.
[4] L. S. Leah, B. Lisa and C. E. Margaret, "Conservatice Stemming for Search and Indexing," ACM, August 2005, pp. 15-19.
[5] S. Jikitsha and P. C. Bankim, "Stemming Techniques and Naive Approach for Gujarati Stemmer," International Conference in Recent Trends in Information Technology and Computer Science, IJCA, 2012, pp. 9-11.
[6] A. F. Alajmi, E. M. Saad and M. H. Awadalla, "Hidden Makov Model Based Arabic Morphological Analyzer," International Journal of Computer Engineering Research, IJCER, Vol. 2, No. 2, 2011, pp. 28-33.
[7] M. Upendra and P. Chandra, "MAULIK: An Effective Stem- mer for Hindi Lanuage," International Journal of Computer Science and Engineering, IJCSE, Vol. 4, No. 5, 2012, pp. 711-717.
[8] R. Ananthakrishnana and R. D. Durgesh, "A Light Stem- mer for Hindi."
[9] K. Dinesh and R. Kumar, "Design and Development of Stemmer for Pujabi," International Journal of Computer Applications, IJCA, Vol. 11, No. 12, 2010, pp. 1823.
[10] S. Llia, "Overview of Stemming Algorithms," Depaul University.
[11] F. B. William and F. J. Christopher, "Strength and Simi- larity of Affix Removal Stemming Algorithms," James Maison University and Virginia Tech.
[12] O. H. M. Ali and L. Ma Shi, "Stemming Algorithm to Clas- sify Arabaic Documents," Symposium on Progress in Information Communication Technology, 2009, pp. 111- 115.
[13] A. James and K. Giridhar, "Stemming in the Language Modeling Framework," SIGIR, ACM, Toronto, 28 July-1 August 2003.
[14] A. Farag and N. Andreas, "N-Gram Conflation Approach for Arabic Text," SIGIR, ACM, Amsterdam, 7 July 2007.
[15] K. Dinesh and R. Prince, "Stemming of Punjabi Words by Using Brute Force Technique," International Journal of Engineering Science and Technology, IJEST, Vol. 3, No. 2, 2011.
[16] D. Sajib and N. Vincent, "Unsupervised Morphological Parsing of Bengali," Lang Resource Evaluation, Springer, 2007
[17] R. Monica, M. Scott and Y. Yiming, "Unsuperised Learn- ing of Arabic Stemming Using a Parallel Corpus," Proceeding of the 41st Annual Meeting of the Association for Computation Linguistics, July 2003, pp. 301-398.
[18] N. S. Giridhar, K. V. Prema and N. V. Subba Reddy, "A Prospective Study of Stemming Algorithms for Web Text Mining," Ganapt University Journal of Engineering Te- chnology, Vol. 1, 2011, pp. 28-34.
[19] K. Chouvalit and B. Veera, "Inverted Lists String Match- ing Algorithms," International Journal of Computer The- oryand Engineering, Vol. 2, No. 3, 2010, pp. 352-357.
[20] K. Koudas, S. Sunita and S. Divesh, "Record Linkage: Si- milarity Measures and Algorithms."
[21] J. Ms. Anjali, "A Comparative Study of Stemming Algo- rithms," IJCTA, Vol. 2, No. 6, 2011, pp. 1930-1938.
[22] B. Bal Krishna and S. Prajol, "A Morphological Analyzer and a Stemmer for Nepali," Madan Puraskar Pustakalaya, Working Papers 2004-2007.
[23] F. Cuna Ekmekcioglu, L. F. Michael and W. Peter, "Stemming and N-Gram Matching for Term Conflation in Turkish Texts," Information Research, Vol. 2, 1996.
[24] C. Sitaula, "Semantic Text Clustering Using Enhanced Vec- tor Space Model Using Nepali Language," GESJ, Vol. 36, No. 4, 2012, pp. 41-46.
[25] MANNING, C. D., RAGHAVAN, P., AND SCH&UTZE, H. 2008. Introduction to Information Retrieval. Cambridge University Press. Cambridge, U.K.
[26] AL-SHAMMARI, E. AND LIN, J. 2008a. A novel Arabic lemmatization algorithm. In Proceedings of Conference AND &08. 113-118.
[27] SNAJDER, J., BA^SIC, B. D., AND TADIC, M. 2008. Automatic acquisition of inflectional lexica for morphological normalization. Inform. Process. Manag. 44, 1720-1731.
