A Sight Reading Approach to Using the DCC

One of the key features of the DCC site is that each text comes equipped with hand-made running vocabulary lists, containing the main definitions for each word, but also the particular one relevant to the context. Very common words are excluded. These take a lot of effort to prepare, of course, so I thought it would be good to explain why we do this.

The point is not just to make it easier for readers to find the correct lemma behind a given form (something automated tools are still very bad at). It also allows for a way of teaching that focuses students’ out of class efforts on vocabulary acquisition and comprehension, rather than the (much harder task of) translation. A vocabulary-focused sight reading approach can help fight the bane of Latin and Greek pedagogy: students writing down the “correct” translation in class, and giving it back on tests, which improves their ability to memorize English, but doesn’t do much for their Latin or Greek.

In essence this is what is now fashionably called a flipped classroom approach, where easier rote tasks are put outside class time, and the hardest tasks are done inside class, collaboratively. In my view the positive psychological effect of this are well worth the effort. Many classical teachers have used this kind of approach over the years. My own particular inspiration is Edwin Post, a professor at De Pauw around the turn of the 20th c., and author of the wonderful Latin at Sight (1895). I know many teachers out there are doing similar things, and would love to hear suggestions and refinements, especially things that DCC could do to better enable this kind of pedagogy.

The routine as I have worked it out in my own classes (one which of course admits of many variations) is as follows:

Students’ class preparation consist of a mix of
• vocabulary memorization for passages to be read at sight in class, and
• comprehension/grammar worksheets on other passages (ones not dealt with normally in class).
Class itself consists mainly of
• sight translation, and
• review and discussion of previously sight-read passages
• grammar review as needed
Testing consists of
• sight passages with comprehension and grammar questions (like the worksheets), and
• vocabulary quizzes.

Textual analysis is done orally in class, through more interpretive worksheets on previously read passages, and in paper assignments.

The rationale behind doing things this way is that:
• students become good at reading Latin or Greek ex tempore. They lose their fear of it. They start to recognize word groupings and syntactical relationships, rather than isolated vocabulary items.
• students learn to guess at unknown words based on context rather than becoming stuck on the first unfamiliar word, or relying too much on the dictionary
• students have no incentive to memorize English translations; the incentive is to master high frequency vocabulary that is likely to be seen again in a new context. These items are learned contextually.
• students get used to identifying grammatical features that actually occur in the text, rather than isolated grammar lessons that don’t always have a clear relationship to reading. Grammar is less a burdensome extra, but as a tool that allows the extracting sense out of a text.
• total quantity of text covered may be somewhat less in class, but worksheets allow at least as much reading total as in the traditional method, probably more

To implement this it is important to
• Have vocabulary lists made up ahead of time. If working toward a high frequency master list, separate the lists into high and non-high frequency portions. Otherwise, just have reasonably comprehensive lists made up. Put it all on a web site for them to study before class. Quiz these occasionally first thing in class. No need to do this every day. They have an incentive to learn vocab. so as not to look too clueless in class. Midterm and final involve comprehensive vocabulary review of words already seen.
• Have worksheets made up ahead of time. Comprehension questions can be written in Latin or Greek, and call for responses in Latin or Greek. This is very difficult at first, but helpful in the long run. Comprehension questions in English are somewhat easier, but make it possible at times for students to merely skim the text looking for key words. But one needs to be resigned to the fact that they will not glean every single nuance of these passages. This is ok. More exposure is better. For the grammar questions, have them spot several instances of a particular construction; or manipulate things, e.g., find several verbs in the imperfect and put in all six tenses and translate (this is a mini synopsis). Focus on pronouns, relative pronouns, reflexives, participles, transitive vs. intransitive verbs, finding word groupings like transitive verbs and their direct object. This kind of grammatical analysis powerfully reinforces sight reading skill.
• When sight reading in class it is essential to do “pre-reading.” Give a little talk about what the passage is about, point out proper names, unusual vocabulary, tricky constructions ahead of time. That way they go in knowing what it is basically about, and will not be phased by knotty bits.
• Make a point of reviewing everything. This gives lots of confidence, reading fluency, vocabulary reinforcement.
• Progress to more sophisticated worksheets that include interpretive tasks, like picking out the most significant or emphatic words, judging the tone, finding literary and rhetorical techniques, inferring what the author wants you to think about what it being said.
• Throughout it is important to communicate with the students what you are doing and why. The notions of high frequency vocabulary, guessing, getting the gist and not worrying so much about the details, these are things the students can get behind. With this good will you can do a lot of more detailed grammatical discussion and textual analysis.
• Grading should be low stakes on the worksheets, at least initially

The feedback from my students on this has been good. Certainly the relationship to grammar is transformed. They suddenly become rather curious about grammatical structures that will help them figure out what is going on. With the worksheets the assumption is that the text makes some kind of sense, rather than what used to be the default assumption, that it’s Latin (or Greek), so it’s not really supposed to make that much sense anyway, right?

–Chris Francese

Latin, visualized

I’m dreaming of an infographic. It shows the top 1000 most common Latin words, broken into groups by semantic category. In each group, you can see at a glance the relative frequency of the words–which ones are the most common. Maybe the extremely common ones are bigger, maybe they rise higher into the third dimension. This information has been culled from the data assembled painstakingly by hand by the Belgian LASLA group, and in Diederich’s Frequency of Latin Words and Their Endings (1939).

Better than that, one can also see at a glance which words are predominantly poetic, which are predominantly prosaic, and which are neither particularly prosaic of poetic. Maybe that’s a color thing–redder for more poetic, bluer for more prosaic. Or maybe the poetic words are higher, up among the clouds, while the prosaic words tramp on the ground. This too has been determined based on the excellent data of LASLA and Diederich, which enumerates occurrences by poetry and prose.

And since the words are grouped by semantic categories, you can see what the main topics of preserved Latin literature are, its main preoccupations. The body. The house. Violence. Writing. Knowledge. Speech. The elements.

Here’s the question: what should be the visual theme? Should it be a landscape? A library?    A Roman temple? Those of you who are visually inclined, help me out. You can probably tell that I have been reading EdwardTufte (Envisioning Information). What if we could make the awesome frequency data we have come alive in graphic form? How cool a pedagogical tool would that be?

–Chris Francese

Perseus and Classics DH

I am extremely grateful to the folks at Perseus, and to all the others who took the time to reply to my earlier, rather dyspeptic, posts about the Perseus Word Study Tool. A valuable blog post by Bridget Almas, the Senior Programmer at Perseus, is up on the front page of Perseus right now. And the conversation on Google+ was animated as well.

Bridget Almas’ central point is that resources are limited, and we need to prioritize. Is morphology per se really worth the investment, in comparison with other more urgent needs? And would a truly open, distributed editing environment for the annotation classical texts (like that already in place for papyri on Papyri.info, and planned by Perseus) attract enough people interested in that particular issue? Probably not, she suggests.

Helma Dik, heavily involved in improving and re-purposing the Perseus data at the University of Chicago, points out that with any tool, including the LWST, we need to teach students how to use it properly. She insists that the large-scale results produced by the Perseus parser are valuable, and that its accuracy can be substantially improved incrementally by means of hand parsing. She has made good progress on the Greek side with Perseus at Chicago. The Latin parser lags because no one has yet taken the time to improve it, and she suggests that that–rather than grousing about the current inadequacy of the tool–should be the focus of our efforts.

Laura Gibbs of the University of Oklahoma, a pioneer and dynamo in digital pedagogy in classics, makes the (to me) central point that what students most need is not full parsing of every word, but to know from what word a given form comes. From a pedagogical standpoint, the correct dictionary head word is the only crucial information. The process of intelligent glossing and annotation is greatly aided by having a core vocabulary, a list of very common items that will not be glossed at all. She argues that this process of pedagogically-driven glossing must be human-created, not machine-generated. She would like to see a collaborative digital environment for reading Latin and Greek together asynchronously, hopefully one that does not focus exclusively on translation as the goal, but on comprehension and reading.

Justin Schwamm, the driver behind Tres Columnae and another pioneer and expert in digital classical pedagogy, helpfully focuses the discussion on the pedagogical goal, which for him is getting people to read Latin, not just translate it. If the tools don’t contribute to reading fluency, we shouldn’t use them. He also points out that from a user’s perspective, the LWST provide too much information. Students’ eyes tend to glaze over when presented with a solid mass of new information, whether in a print textbook or on a web page.

I think that Bridget Almas has put her finger on the central problem we face right now. What should be the priorities, and (a closely related question) how are we to marshal the labor, and raise the level of interest among classicists in improving current digital tools and creating new ones? This profession is full of amazing, learned, selfless, phenomenally hard-working people. Why are so few of them putting energy into digital collaboration, teaching and dissemination, as opposed to traditional print monographs and articles? If we could get 20% of those PhD’s toiling away on university press monographs to work on digital editions, where would we be?

The value of DH for humanists lies in its collaborative nature and the transformation of scholarly communication it enables; in the innovative and effective pedagogy it facilitates; and in the vast increase in access to information and learning it makes possible. Why are more classicists not excited by this? Things are of course changing slowly, and we’re all working on this in our own ways, but to accelerate the acceptance of digital classics in the profession and bring in more labor to fix things like the LWST, a few things are especially important, I think. None of this is news, just trying to articulate it as clearly as I can:

First, it’s important to keep exploring modes of peer review. Classicists are very sensitive to what Dan Cohen has called the social contract in scholarly communication represented by presses, proof-reading, peer review, and also design aesthetics. Digital publishing has a severe, nay, crippling deficit in prestige. Second, and of course related, we have to keep focusing on the quality of the content. Classicists have a very low tolerance for error, and thus distrust the internet more than most. But quality is not enough, as the lackluster start of the Princeton/Stanford working papers shows. Next, and this is more relevant to the issue at hand, we should make tools like the LWST respond to current pedagogy and reading practices. The tools should be aimed laser-like at the real needs of users, and respond to their culture of reading. Pedagogy, rather than computational linguistics, must be central to the iterative design process. Finally, something that is not really in the debate as far as I know, we should focus on the scholarly voice. Classicists’ reading culture places a high value on the expert, and prizes the trained scholar above all. Most digital tools currently are either highly impersonal (as with the LWST) and thus would be viewed with suspicion even if they were more accurate; or they try to rely on crowd-sourcing, which goes rather against the grain of the classical mind. Reverence for expert opinion both inhibits ordinary readers from contributing to crowd-sourced annotation (compare the rather slow start of The Open Utopia), and prevents most readers from taking it seriously. The good news is that the digital environment allows closer contact with scholars through blogging and especially though audio recordings (check out the classical material on New Books Network).

Ok, I have strayed rather far afield from the Latin Word Study Tool. My grand scheme was to create a distributed editing environment for creating vocabulary lists, like the environment the papyrologists have at Papyri.info, but the minds at Perseus were way, way ahead of me, and have something like that in the works already. In a future post I will think about what kinds of features I might want to see in such a thing.

Do the Flaws in the Perseus Word Study Tool Matter?

In a recent post I tried to categorize the problems of the Perseus Word Study Tool, as tested on a section of Vergil. More surprising to me than the overall rate of error (about one in three words was misidentified in some way) was the fact that many of the errors were not subject to correction by means of Perseus’ “voting” system; and that even when voting was in operation, it often did not correct the error. Sometimes the correct choice was not an available option; other times, unanimous correct votes were ignored, and unanimous incorrect votes were accepted. At Aen. 5.17, to add another example to those mentioned the earlier post, the vocative magnanime was incorrectly called an adverb on the basis of six incorrect user votes.

The inadequacy of the LWST will not have been news to anyone who has used it. The question is, is the level of error pedagogically significant? Is the LWST good enough for the purposes of a typical Latin student? In other words, should the average Latinist care? It is not good enough, and the level of error and the specific types of errors in this flagship classical DH project are pedagogically significant and worthy of attention, I believe, for several reasons.

1. Words that give students the most trouble–relative pronouns, demonstratives, quam, ut, modo, Q-words in general–are exactly those least likely to be handled well by the LWST. The earlier post has some examples from my small sample, but I’ll add here that in Aen. 5.30 (magis . . . ) quam, when it comes to that quam, the LWST offered no fewer than seven possible quams to choose from (all numbered quam 1-7), none of which has the correct definition in the context (“than”).

2. The LWST is of course helpless when it comes to unusual or idiomatic expressions, of which there is a good example in my sample at 5.6, were notum must be translated “the knowledge that.”

3. The tool naturally can analyze only what is there. It cannot tell when something is left out or assumed.

4. A major structural problem is represented by bad short definitions of the type (to choose again from examples offered by my sample)  iubet = “imposed,” iam = “are you going so soon,” frustra = “in deception, in error,” or more subtly, the fact that the common meaning of tendere, “direct one’s course,” does not appear in the short def. for that word.This is important because, even though one can click on and read the full Lewis & Short dictoinary definition, intermediate students are very unlikely to click through and sift through long entries in search of the correct definition.

5. Moreover, the LWST obscures the relationships between words, which is key to learning to read Latin. This is why seemingly minor accidence mistakes are meaningful. Misled on a part of speech, or the gender of an adjective or the case of a noun, the student will likely not see the syntactical connection between words, and thus the tool reinforces the urge to produce the dreaded “word salad” translations.

6. More broadly, with its cryptic statistical data and jumbled pseudo-information, the LWST reinforces the the impression that many students have: that Latin isn’t really supposed to make sense anyway, that it’s all some kind of fiendish crossword puzzle.

Gregory Crane in an important article and apologia for Perseus, has said that the goal of the Perseus Project is to provide “machine-actionable knowledge.”

Reference materials, in particular, are structured to support automatic systems (e.g., the morphological analyzer learns Greek and Latin morphology from a machine actionable grammar) and to be decomposed into small chunks and then recombined to provide dynamic commentaries. If you retrieve a book in a language that you cannot read or on a topic that you cannot understand, the system can find translations where these already exist, machine translation and translation support systems, reference works, and general background information suited to the general background and immediate purposes of the reader. In knowledge bases, the boundaries between books begin to dissolve.

But clearly machines are spectacularly bad at understanding Latin at the moment. Crane thinks in terms of many decades, and is waiting for massive improvements in artificial intelligence, or teams of graduate students to encode correct grammatical analysis in texts. But such a prospect seems increasingly far off, and given the size of the Perseus Digital Library (10.5 million words at the moment), it seem unlikely that the millions of errors can be corrected any time soon, if ever. Indeed, would it be worth huge the investment of time and money? In the meantime, we need to create a collaborative tool for generating reasonably correct and reliable vocabulary lists for Latin (and Greek) authors that will be helpful for students and teachers around the world. Why we should do this, and what kind of tool I have in mind, will be the subjects of future posts.

–Chris Francese

 

Types of Error in the Perseus Latin Word Study Tool

The Perseus Latin Word Study Tool (LWST) is intended to provide dictionary definitions and grammatical analysis of all words in the Latin texts available in the Perseus Digital Library, currently 10.5 million words.

A check of the definitions and grammatical analysis of an arbitrarily chosen chunk of Vergil’s Aeneid (5.1-34, 223 words), found that it was incorrect in 79 instances, or 35.4% of the time (and correct 64.6% of the time). The most common type of error (21 instances,  26.6% of all errors, 9.4% of all words) was a mistake of accidence, for example duri (5.5) was taken as genitive singular instead of nominative plural. In 17 cases (21.5% of errors, 7.6% of all words) words were assigned to the wrong lemma, as when quoque (“and whither”) was derived from quoque (“also, too”), or venti (“winds,” 5.20) was assigned to the verb venio, “come,” as if it were the perfect participle. This particular mistake occurred three times in this passage, and the correct lemma was not listed as a possible option. In 14 instances (17.7% of errors, 6.3% of all words) the dictionary definitions provided were wildly wrong. This was true of some very common words. iam was glossed as “are you going so soon,” nec as “and not yet,” ab as “all the way from.” Elissae (5.3) was glossed as “Hannibal.” In every case this type of error was seen to come from the pulling, seemingly at random, of a word or phrase from the dictionary of Lewis & Short on which the LWRT is based. In 11 instances (13.9% of errors,  4.9% of all words), the relevant definition in the context at hand was not provided (though it could be found by clicking to and reading through the full Lewis & Short dictionary entry). For example, cerno was glossed as “separate, part, sift,” but not “perceive,” or infelicis (5.3) glossed as “unfruitful, not fertile barren,” rather than “unfortunate.” More seriously, all relative pronouns were glossed as interrogatives (“who? which? what? what kind of a?”), and described simply as “pron.” The word “relative” did not appear on the page. In 8 instances (10% of errors, 3.6% of all words) a word was assigned to the incorrect part of speech, as when medium (5.1) was called a noun rather than an adjective, or locutus (5.14) assigned to the rare 4th decl. noun “a speaking” rather than to loquor. In 4 cases (5% of errors, 1.8% of all words), there was no definition available. And in all cases deponent verbs were incorrectly labeled passive (4 instances in this particular section, or 5% of errors, 1.8% of all words).

Now, the makers of Perseus are perfectly aware of the flaws in LWST, and attempt to use the power of social media of help remedy the situation. Subjoined to the analysis of every ambiguous word, after an explanation of the methodology used, one finds a plea to help by voting.

The possible parses for this word have been evaluated by an experimental system that attempts to determine which parse is correct in this context. The system is composed of a number of “evaluators”–each of which uses different criteria to score the possibilities–whose votes are weighted to determine the best answer. The percentages in the table above show each evaluator’s score for each form, which are then combined to determine each form’s overall score.
This selection used the following evaluators:
• User-voting evaluator: Scores parses based on the number of votes each one has received from users. Weighted more heavily as more users vote for a given word in a text.
• Prior-form frequency evaluator: Evaluates forms based on the preceding word in the text; finds the most likely parse among this word’s possible morphological features and the preceding word’s possible features based on the frequency of each possible pair.
• Word-frequency evaluator: Scores parses based on how often the dictionary word appears in the Perseus corpus. Only used when a given form could be from more than one possible word.
• Tagger evaluator: Evaluator based on pre-computed automatic morphological tagging
• Form frequency evaluator: Scores parses based on how often their morphological features (first-person, indicative, plural, and so on) occur among all the words in the Perseus corpus.
User votes are weighted more heavily than the other methods, which are all treated equally.
Don’t agree with the results? Cast your vote for the correct form by clicking on the [vote] link to the right of the form above!

But here too, some problems arose in my sample. First of all, only a handful of doubtful words had any votes. Second, many of the error types identified above do not admit of voting. And third, those that did have votes did not always benefit from having them. Here is the entry on the word rates in ut pelagus tenuere rates (5.8), showing a preference for the (incorrect) accusative, despite nine user votes for the (correct) nominative.

 

On the word pater in Quidve, pater Neptune, paras? (5.14), ten incorrect user votes for the nominative win out over the (obviously correct) vocative.

More common, however, is the lack of any user votes at all, as in this very confusing jumble of information on the word hoc (5.18). Note that the correct lemmatization (> hic) has a nonsensical definition; that the morphological analysis states it can only be a pronoun (“pron.”) whereas here, as often, it is a demonstrative adjective; and finally that the LWST incorrectly concludes that the form derives from the lemma huc.

Another odd and thankfully rare genre of error occurs in the case of deinde (5.14), which is correctly analyzed, but put beside a fictional alternative, the present imperative of a verb *deindo.

I would like to know if the same level of error and types of errors occur when LWST is unleashed on a prose text. Perhaps there the idea of a “prior-form frequency evaluator” would make more sense.

It is not my intent to denigrate the huge achievements of Perseus in our field. It is certainly better to have the LWST than not to have it. My purpose here is just to investigate the nature and extent of its errors. If this sample is at all representative, something along the lines of 3.5 million errors exist in the current database. I would also like to ask, is it realistic to think that qualified people can be found to correct the mistakes of the LWST? What is the incentive for professional Latinists to do so?

I also have a proposal for a different kind of tool, which I will save for another post, since this one is already too long. Your thoughts?

–Chris Francese

Getting social

NITLE‘s Senior Fellow extraordinaire Bryan Alexander stopped by Dickinson on June 28 to help a working group of faculty staff and administrators crafting a digital humanities grant proposal. As part of his presentation he kindly gave us some expert feedback on the DCC site. He liked the design very much and appreciated the foregrounding of our editorial committee, but pointed out that out current design is not very mobile-friendly; that the search field is rather hidden and inadequate; and that the site itself is not interactive. And he asked what we were doing to promote awareness of the site, to which I didn’t have much of an answer.

Bryan’s main suggestion on that front was to create more of a presence in social media. I had been intending to do this, but hadn’t really focused on how important it is if the site is actually going to get used. There’s a lot going on in the interweb, and you can’t just sit back and assume the relevant audience will find you. I started up a new DCC Facebook page, and a Twitter feed (@DCComm) as well. I must say I am much more comfortable on FB than on Twitter. But the classical Twitterati are very energetic and supportive, and I’m starting to understand why people like the medium.

As for getting word out in a more traditional way, I will be speaking in several places in the fall on the DCC and the Digital Humanities principles that drive it: The University of Illinois Urbana-Champaign (Sept. 14), The University of Virginia (Sept. 22 and 23), The University of Pennsylvania (October 18), and in Amsterdam at the European Society for Textual Scholarship’s 2012 conference ‘Editing Fundamentals’ (November 22-24). I am very much looking forward to all the feedback and suggestions I will get, and am hoping to lure some more collaborators into the project as well.

–Chris Francese