Comments on: Latent Semantic Indexing http://www.kryogenix.org/days/2003/04/28/semantic scratched tallies on the prison wall Wed, 03 Dec 2008 23:58:36 +0000 http://wordpress.org/?v=2.6.5 By: Mark Guckeyson http://www.kryogenix.org/days/2003/04/28/semantic#comment-2454 Mark Guckeyson Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/04/28/semantic/#comment-2454 <p>Is there any chance I could see the code behind your implementation, Todd? I have something up and running, but everything that I search returns all docs with a relevance of 1 in each case; clearly not desired behavior. It also returns nothing on words that are in the actual $self->{ word_list }, which seems very wrong as well.</p><p>The perl.com article has a few issues in it, btw… most notably the line on the final script that uses a method not available in the class: <code>$engine->set_threshold( 0.8 );</code></p><p>It’s a fascinating concept and I’m looking forward to seeing it return “real” results, but my non-math-genius head is having some trouble wrapping around the basic concepts.</p> Is there any chance I could see the code behind your implementation, Todd? I have something up and running, but everything that I search returns all docs with a relevance of 1 in each case; clearly not desired behavior. It also returns nothing on words that are in the actual $self->{ word_list }, which seems very wrong as well.

The perl.com article has a few issues in it, btw… most notably the line on the final script that uses a method not available in the class: $engine->set_threshold( 0.8 );

It’s a fascinating concept and I’m looking forward to seeing it return “real” results, but my non-math-genius head is having some trouble wrapping around the basic concepts.

]]> By: Maciej Ceglowski http://www.kryogenix.org/days/2003/04/28/semantic#comment-2455 Maciej Ceglowski Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/04/28/semantic/#comment-2455 <p>From what I can tell, Autonomy uses Bayesian networks, which is a rather different model than the vector-space approach underlying <span class="caps">LSI</span>.Colin is right to say that the key step in <span class="caps">LSI</span> is the dimensionality reduction, which squishes things down and creates the expanded-recall magic.You might want to take a look at Search::ContextGraph for a different approach that gives similar behavior to <span class="caps">LSI</span>, without all the overhead of the vector stuff. And send in patches :-)</p> From what I can tell, Autonomy uses Bayesian networks, which is a rather different model than the vector-space approach underlying LSI.Colin is right to say that the key step in LSI is the dimensionality reduction, which squishes things down and creates the expanded-recall magic.You might want to take a look at Search::ContextGraph for a different approach that gives similar behavior to LSI, without all the overhead of the vector stuff. And send in patches :-)

]]> By: Todd Larason http://www.kryogenix.org/days/2003/04/28/semantic#comment-2456 Todd Larason Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/04/28/semantic/#comment-2456 <p>I’m using a search engine trivially built on top of VectorSpace.pm from the perl.com story (with one change[1] to avoid a warning; still not sure this is the right fix).I’ve been pretty happy with it after not quite 2 months use.[1]- my $offset = $self->{‘word_index‘}->{$w};- index( $vector, $offset ) .= $value;+ if (defined($w) && defined($self->{word_index}->{$w})) {+ my $offset = $self->{‘word_index‘}->{$w};+ index( $vector, $offset ) .= $value;+ }</p> I’m using a search engine trivially built on top of VectorSpace.pm from the perl.com story (with one change[1] to avoid a warning; still not sure this is the right fix).I’ve been pretty happy with it after not quite 2 months use.[1]- my $offset = $self->{‘word_index‘}->{$w};- index( $vector, $offset ) .= $value;+ if (defined($w) && defined($self->{word_index}->{$w})) {+ my $offset = $self->{‘word_index‘}->{$w};+ index( $vector, $offset ) .= $value;+ }

]]> By: colin_zr http://www.kryogenix.org/days/2003/04/28/semantic#comment-2457 colin_zr Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/04/28/semantic/#comment-2457 <p>I’m not an expert at all, so I may be confusing matters here, but I think your description misses the most important aspect of <span class="caps">LSI</span>, which is that the <span class="caps">SVD</span> stage finds similarities between terms, allowing searches to return results which do not contain the terms in the query. Your description sounds like a standard vector-space system without <span class="caps">LSI</span>.As far as Autonomy is concerned, I’ve always got the impression that the important thing about their systems is the user-interface and the degree of integration into whatever else it is that you’re doing, rather than the particular algorithms they use. (I also somewhat distrust them generally—though you probably shouldn’t base your buying decisions on the hunches of random readers of your site. :) )</p> I’m not an expert at all, so I may be confusing matters here, but I think your description misses the most important aspect of LSI, which is that the SVD stage finds similarities between terms, allowing searches to return results which do not contain the terms in the query. Your description sounds like a standard vector-space system without LSI.As far as Autonomy is concerned, I’ve always got the impression that the important thing about their systems is the user-interface and the degree of integration into whatever else it is that you’re doing, rather than the particular algorithms they use. (I also somewhat distrust them generally—though you probably shouldn’t base your buying decisions on the hunches of random readers of your site. :) )

]]> By: colin_zr http://www.kryogenix.org/days/2003/04/28/semantic#comment-2458 colin_zr Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/04/28/semantic/#comment-2458 <p>I’m not an expert at all, so I may be confusing matters here, but I think your description misses the most important aspect of <span class="caps">LSI</span>, which is that the <span class="caps">SVD</span> stage finds similarities between terms, allowing searches to return results which do not contain the terms in the query. Your description sounds like a standard vector-space system without <span class="caps">LSI</span>.As far as Autonomy is concerned, I’ve always got the impression that the important thing about their systems is the user-interface and the degree of integration into whatever else it is that you’re doing, rather than the particular algorithms they use.</p> I’m not an expert at all, so I may be confusing matters here, but I think your description misses the most important aspect of LSI, which is that the SVD stage finds similarities between terms, allowing searches to return results which do not contain the terms in the query. Your description sounds like a standard vector-space system without LSI.As far as Autonomy is concerned, I’ve always got the impression that the important thing about their systems is the user-interface and the degree of integration into whatever else it is that you’re doing, rather than the particular algorithms they use.

]]> By: Simon Willison http://www.kryogenix.org/days/2003/04/28/semantic#comment-2459 Simon Willison Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/04/28/semantic/#comment-2459 <p>I find this particularly fascinating because I did a course on vectors last term at Uni and was given the impression that they were mainly used for graphics related stuff – seeing how they could be used for seach as well was really interesting.</p> I find this particularly fascinating because I did a course on vectors last term at Uni and was given the impression that they were mainly used for graphics related stuff – seeing how they could be used for seach as well was really interesting.

]]>