Golden Time: Help Offers

[rpapo]

I correct myself as far as the dictionary size is concerned: running the program right now, the raw dictionary had 156,208 entries. With conjugations, the size ballooned to 5,511,101 entries. The conjugations currently performed are:

• Stem
  Plain Non-Past (positive and negative (2 forms))
  Plain Past (positive and negative)
  Plain Imperative (2 forms)
  Plain Presumptive (2 forms)
  Plain Conjunctive
  Plain Provisional (positive and negative)
  Plain Conditional (positive and negative)
  Polite Conditional (positive and negative)
  Plain Passive Non-Past
  Plain Passive Past
  Plain Wish
  Polite Non-Past (positive and negative)
  Polite Past (positive and negative)
  Polite Presumptive
  Potential (2 forms)
  Causative
  Negative Conjunctive (-zu)
  Plain Alternative (positive and negative)
  Polite Alternative (positive and negative)
  Plain Progressive
  Plain Progressive Casual
  Polite Progressive
  Plain Progressive Past (2 forms)
  Polite Progressive Past
  Plain Negative Presumptive
  Polite Negative Presumptive
  Noun (from adjective)
  Adverb (from adjective)
  Seems like (from adjective)

Some of the conjugations combine, so there's even more stuff generated. Only the current memory consumption is stopping me from adding even more variants . . . some of which I could really use.

[YoakeNoHikari]

Oh my Lord. I didn't even realize there were so many conjugations.

[rpapo]

Oh my Lord. I didn't even realize there were so many conjugations.

I haven't finished putting them all in, either...

See http://www.epochrypha.com/japanese/materials/verbs/ for a pretty good list.

[rpapo]

...what language is your Program in, Rpapo? would it be possible to get it? (I suppose language doesn't matter if it's an .exe. )

Come and get it. Download http://mywebpages.comcast.net/rpapo/Nihongo.zip.

The program to be run is at x64\release\nihongo.exe, and it requires two main input parameters: the name of the file to be read in (it must be Unicode TXT format), and the name of the file to be generated as output. Additionally, you can specify the line number (zero-based) where you want to start parsing, and the line number where you want to stop parsing. A complete command line might look like this:

Code: Select all

c:\Nihongo> x64\release\nihongo.exe GoldenTime_1.txt OutputText.txt 1623 1631

Beyond that, the dictionary extensions are not yet configurable. The program has a number of extended dictionary entries hard-coded in it (for like Banri, Taiga Aisaka, etc), but that extension list is not in a separate file at this point.

[hobogunner]

Awesome...I will download that when I get home.

Can you create custom output? Such as after it references the dictionaries, create a seperate entry that goes through and tells you any names? (the romaji converter that I use online writes Nemoto as Konpon....) so it would be nice to be able to explicitely tell it what it is. (In a different line is fine, but just to make sure I know.)

[rpapo]

Awesome...I will download that when I get home.

Can you create custom output? Such as after it references the dictionaries, create a seperate entry that goes through and tells you any names? (the romaji converter that I use online writes Nemoto as Konpon....) so it would be nice to be able to explicitely tell it what it is. (In a different line is fine, but just to make sure I know.)

That's what the dictionary extensions are all about. I don't have that externalized at this point: the extensions are registered in the code. Currently I use this for most of the character's names, plus for a few place names and spoken elisions like してしまった => しちゃった.

[hobogunner]

Hmmm....I will certainly look into figuring that out. Not too familiar with C++, but if you parse the text, it reads it line by line, how hard would it be to create a new sub routine to add a secondary 'personal dictionary' reference? As in have a third line aside from the other 2 dictionaries?

[rpapo]

Hmmm....I will certainly look into figuring that out. Not too familiar with C++, but if you parse the text, it reads it line by line, how hard would it be to create a new sub routine to add a secondary 'personal dictionary' reference? As in have a third line aside from the other 2 dictionaries?

The logic is more complicated that what you're thinking. For one thing, the basic unit for parsing is not a line, but rather a phrase, which is something delimited by punctuation. After all, Japanese words in general do not contain periods, commas, spaces, parentheses nor quotation marks. Since we are attempting to break the text into inindividual words, we can safely bet that punctuation divides words.

[hobogunner]

I understand that, however even without dividing marks, one could find a certain piece of text by simply running through it again, there is obviously a text scraper set up, to compare and get the longest phrase it can, therefore, it has to have some way of going through every individual level.....telling it to scrape again and specificaly find certain characters would't be too hard, would it?

Sorry if that idea makes no sense, I have yet to look at it since I'm on my iPad.

[rpapo]

I understand that, however even without dividing marks, one could find a certain piece of text by simply running through it again, there is obviously a text scraper set up, to compare and get the longest phrase it can, therefore, it has to have some way of going through every individual level.....telling it to scrape again and specificaly find certain characters would't be too hard, would it?

Sorry if that idea makes no sense, I have yet to look at it since I'm on my iPad.

If you want the parser to recognize names, you simply add a dictionary entry for the name. With the current code of my parser, you need to add to the stuff that starts in file Nihongo.cpp, line 977 and later. You will find entries like the following:

Code: Select all

AddWord ( L"竹宮", L"たけみや", L"(name) Takemiya" ) ;

All you have to do is add entries to suit yourself, and then recompile the application afterwards (of course).

[hobogunner]

That's awesome....now I also see why it takes so much memory.

[rpapo]

That's awesome....now I also see why it takes so much memory.

My intent is to externalize the list of additional words, so you don't have to be a programmer with Visual Studio 2008 handy to modify the list and recompile the program.

FWIW, as a side-effect of the parsing run, two special TXT files get created: Dictionary.txt and Index.txt. These files are a text representation of the fully elaborated dictionary with all the generated conjugations. They are huge...

[jonathanasdf]

Hey, could I add you on msn/gtalk or something? If you would like, I could work together with you on this. The ability to add your own words and phrases and stuff seem it might be pretty nice.

Edit: actually nevermind, it's a bit too long that I suddenly got lazy...

[hobogunner]

I created this topic: viewtopic.php?f=31&t=4691#p121114

as to help with us guys who are going to be adding individual components into it. Discussion of what we've done.

[rock96]

Is there need to merge posts from this topic?