This is as days pass by, by Stuart Langridge

Is there any chance I could see the code behind your implementation, Todd? I have something up and running, but everything that I search returns all docs with a relevance of 1 in each case; clearly not desired behavior. It also returns nothing on words that are in the actual $self->{ word_list }, which seems very wrong as well.

The perl.com article has a few issues in it, btw… most notably the line on the final script that uses a method not available in the class: $engine->set_threshold( 0.8 );

It’s a fascinating concept and I’m looking forward to seeing it return “real” results, but my non-math-genius head is having some trouble wrapping around the basic concepts.

From what I can tell, Autonomy uses Bayesian networks, which is a rather different model than the vector-space approach underlying LSI.Colin is right to say that the key step in LSI is the dimensionality reduction, which squishes things down and creates the expanded-recall magic.You might want to take a look at Search::ContextGraph for a different approach that gives similar behavior to LSI, without all the overhead of the vector stuff. And send in patches :-)

I’m using a search engine trivially built on top of VectorSpace.pm from the perl.com story (with one change[1] to avoid a warning; still not sure this is the right fix).I’ve been pretty happy with it after not quite 2 months use.[1]- my $offset = $self->{‘word_index‘}->{$w};- index( $vector, $offset ) .= $value;+ if (defined($w) && defined($self->{word_index}->{$w})) {+ my $offset = $self->{‘word_index‘}->{$w};+ index( $vector, $offset ) .= $value;+ }

I’m not an expert at all, so I may be confusing matters here, but I think your description misses the most important aspect of LSI, which is that the SVD stage finds similarities between terms, allowing searches to return results which do not contain the terms in the query. Your description sounds like a standard vector-space system without LSI.As far as Autonomy is concerned, I’ve always got the impression that the important thing about their systems is the user-interface and the degree of integration into whatever else it is that you’re doing, rather than the particular algorithms they use. (I also somewhat distrust them generally—though you probably shouldn’t base your buying decisions on the hunches of random readers of your site. :) )

I’m not an expert at all, so I may be confusing matters here, but I think your description misses the most important aspect of LSI, which is that the SVD stage finds similarities between terms, allowing searches to return results which do not contain the terms in the query. Your description sounds like a standard vector-space system without LSI.As far as Autonomy is concerned, I’ve always got the impression that the important thing about their systems is the user-interface and the degree of integration into whatever else it is that you’re doing, rather than the particular algorithms they use.

I find this particularly fascinating because I did a course on vectors last term at Uni and was given the impression that they were mainly used for graphics related stuff – seeing how they could be used for seach as well was really interesting.