{"id":513,"date":"2009-07-24T11:33:44","date_gmt":"2009-07-24T18:33:44","guid":{"rendered":"http:\/\/wp.colliertech.org\/cj\/?p=513"},"modified":"2009-07-24T11:35:52","modified_gmt":"2009-07-24T18:35:52","slug":"parsing-english-language-if-you-consider-irc-logs-to-qualify","status":"publish","type":"post","link":"https:\/\/wp.c9h.org\/cj\/?p=513","title":{"rendered":"Parsing English Language (if you consider IRC logs to qualify)"},"content":{"rendered":"

Haha. I wrote this<\/a> on October 3rd, 2006. I started logging IRC conversations in 2005. I’ve got lots from #perl and #mono. I wonder if the n’ere-do-wells who frequent those channels will mind me using their public commentary to build an english language parser. I’ll make it “open source,” of course. Erhm, I mean Free Software.<\/p>\n

\nSo… I’ve been trying to parse english language (text) for a while. I haven’t quite done it yet. My new idea involves IRC bots. We have lots and lots of conversations happen in public IRC channels every day. The conversations usually involve transmission of ideas in textual format. These transmissions are quite often (but not always) understood by the receiving party. The transmissions are expected to have been intercepted by the general public, so having a bot intercept and attempt to interpret should not offend anybody :) It will anyway, of course, but we’ll pretend that this 0.00001% minority doesn’t mind being marginalized for the time being.<\/p>\n

So the idea is this: split the words on the standard word boundaries and do a dictionary lookup on each of them. Cache the results locally so a network hit isn’t incurred for each word we look up. The info we will take from the dictionary lookup is the “part of speech” that the word falls into. In the english language, this is not often a scalar value. Computers are getting fast, so I think that this shouldn’t be too much of a problem.<\/p>\n

So now we have a list of words and the parts of speech for each of these words. These words are assumed to combine in such a way that they form a coherent idea.<\/p>\n

At this point, we can create an XML document from these word elements, for instance:<\/p>\n

\r\n<sentence>\r\n  <article>the<\/article>                                                                                                                             \r\n  <noun plurality=\"plural\">cats<\/noun>                                                                                                               \r\n  <verb>are<\/verb>                                                                                                                                   \r\n  <adjective>cute<\/adjective>                                                                                                                        \r\n<\/sentence>\r\n<\/pre>\n
There would need to be an XSD or DTD against which to test this document in order to determine whether these parts of speech can be combined in this way to make a valid, gramatically correct, sentence.\n<\/p><\/blockquote>\n\n
Tweet<\/a><\/div>\n","protected":false},"excerpt":{"rendered":"
Haha. I wrote this on October 3rd, 2006. I started logging IRC conversations in 2005. I’ve got lots from #perl and #mono. I wonder if the n’ere-do-wells who frequent those channels will mind me using their public commentary to build an english language parser. I’ll make it “open source,” of course. Erhm, I mean Free […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":""},"categories":[165,181,116,7,18],"tags":[],"class_list":["post-513","post","type-post","status-publish","format-standard","hentry","category-databases","category-irc","category-language","category-mono","category-perl"],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p1YDIB-8h","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=\/wp\/v2\/posts\/513","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=513"}],"version-history":[{"count":2,"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=\/wp\/v2\/posts\/513\/revisions"}],"predecessor-version":[{"id":515,"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=\/wp\/v2\/posts\/513\/revisions\/515"}],"wp:attachment":[{"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=513"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=513"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wp.c9h.org\/cj\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=513"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}