Collaborative Semantic Analysis.

[swestrup]

I just had a random thought that I figured I should write down before I forget it. When _sps_ was here on Saturday, I mentioned to him a paper I had seen on the difficulty of storing and retrieving scientific papers that are relevant to a field of research. It has gotten so bad in the field of mathematics that it is now often easier to spend a year re-solving a tricky mathematical problem than it is to find an existing paper with the solution. There is a (woefully underfunded) institute that tries to produce a controlled-vocabulary description of the semantic elements in new papers, and record them. They keep falling further and further behind.

Anyway, _sps_ had some not-unreasonable ideas on how to encode useful indexes of these math papers so that relevant materials could be searched for. The big question is: how do you do the semantic analysis? For something like Math, you need a human, and one that understands the math as well. Plus, it would help if they just happened to know of all of the other bits of math that the paper overlapped, even if they are in other fields and use different nomenclature.

Anyway, it suddenly occurred to me that it might be possible (I'm not sure how) to design a mathematics-paper search-engine and browser which had the express purpose of eliciting from a mathematician information about the nature of the paper being studied, and how closely its contents matched that mathematicians current work. This would be done, not by asking questions, but by allowing the mathematician to categorize his searches by project, and to pay attention to how long he spent studying various sections of the paper. As well, if we provided various renaming and renomenclaturing systems, we might get further information by observing the transformations that were performed on the paper.

In the end, I would hope the gathered data from a large number of mathematicians could be used to build a fuzzy index of any given paper, and to let us build a map of which things seemed to be close to each other in a semantic space.  I don't know, ultimately, how well such a system would work, but I think it would be worth giving it a try.

Comments

[joenotcharles.livejournal.com]

Hmm. Seems to me that math, using a technical vocabulary, should be fairly amenable to straight keyword analysis. What am I missing?

[swestrup]

The problem is that math reuses the same word for many distinct concepts. The paper I read indicated that 'End' had 7 technical meanings, 'form' had 5 and 'set' had a bunch as well. These would have to be distinguished by context. As well, there is the problem that mathematicians are always discovering things like the fact that a 'Hobson's group' is just another way of describing a 'Hopf Algebra' and that there is a one-to-one mapping between them and 'Gerber Manifolds'. This is not the sort of thing which is too easily done just by keywords. Plus, the article pointed out that there are currently estimated to be 260,000 concepts in need of keywords in Mathematics, and only some 13,000 official keywords to use.

[swestrup]

Oh, and I found a pointer to the paper that I read that got me thinking about this:

Mathematical knowledge management is needed

[joenotcharles.livejournal.com]

For things like "end", "form" and "set", most of the distinct meanings would commonly be found with another set of keywords - the names are often a giveaway, for instance.

Equation of "Hobson's group" and "Gerber manifold" sounds like you'd need to have mathematicians constantly updating the keyword lists to indicate synonyms, but that's much easier than having them review the papers.

Those numbers are a little startling, though - I knew there was overloading, but that's a hell of a difference.

[joenotcharles.livejournal.com]

Hmm. I think he points out a problem that needs solving, yes, but I still think you can make big strides with a more intelligent search system before needing to go through and add metadata to every document by hand. He seems to be assuming that "full text search" can be discarded simply because it doesn't work all that well right now.

[swestrup]

Ah, but my point was it doesn't have to added 'by hand'. It may be possible to deduce the metadata just by observing browsing behavior. If so, it has applications to many types of search, not just for math.

Interesting problem of taxonomy

[ketherian.livejournal.com]

Sounds like a problem that's grown beyond the reasonable bounds of taxonomy and metadata Good primer here. It may even go beyond the abilities of Semiotics; in which case I could never see a human being able to create a Controlled vocabulary large enough and flexible enough to keep up with the field.

Studying eye movements across a browser and time spent on the subject while reading would provide some structure in which to base the metadata, but wouldn't you get as much studying the references (e.g.: bibliography, citations, etc) as well? I'm not sure examining the reader of the paper would be as worthwhile as studying the writer. Certainly having both as part of the generation of the metadata would provide interesting results - but don't you normally have to read an entire paper before deciding if it fits within the context of what you are researching? How do you discount those who read the paper and found it was not helpful? If these are a small number, I suppose they'd work themselves out of your index; but it could lead to some very quirky results.

Wee. That was fun. Thanks for the brain stretch.

Re: Interesting problem of taxonomy

[swestrup]

The problem with bibliographies and citations (useful as they are) is that they are a usually just a list of the articles that the writer remembers as being influenctial, after they have written the work. There are a number of selection effects here (padding if the list is too short, pruning if its too long) since many writers don't have good insight into the creative process and what affected it. As well, a writer may not know of other fields that his work is relevant in, and that's really what we want to discover.

As to the question of detecting perceived relevance, that shouldn't be too hard. My own experience of doing paper searches is that one first reads the abstract to see if the article might be relevant, then one skims the article to get the gist of what its really about, and if they even talk about the area you are interested in. Finally, if all looks good, you read the whole thing carefully. If you measure how long someone spends on each of these activities, you get a good idea of how relevant they think it is. As well, if they drop the item into a 'relevant citation' list, you note that. You also let them drop it into a 'checked but irrelvant' list so they know they've already looked at it, and that gives you added data. If done carefully, you can do it all with an instrumented browser and an eye tracker isn't even necessary (although it would help).

Re: Interesting problem of taxonomy

[sps.livejournal.com]

I can't see this. Bibliographies are political things: academics are promoted based on how often they are cited, so when you are considering what to put in a bibliography you are thinking, is this person my friend? Do I owe them a favour? Do I want them to owe me? Can I leave out this person because I hate them so much, or will they then vote against me on the committee? And even then you can only include things you can remember, which for a mathematician (who is trained to work things out over again) isn't usually much!

[sps.livejournal.com]

Mathematics uses very few concepts, which come up over and over again. Progress in maths is almost entirely about collapsing things together. In fact, since proofs are in principle automatically checkable once found, you could argue that finding the minimal set of keywords is the only content of mathematics (of course, the keywords would be structured into lambda-trees...).