Weblog


Weblog

Simon Willison: In praise of functional programming

30.04.2003, 12:54 · Martin · Permalink

Structure and Interpretation of Computer Programs (full text HTML)

On my reading list...

keyword similarity works (too)

30.04.2003, 11:14 · Martin · Permalink

Well the "Textarchiv" search vor "Ähnliche Artikel" for this German Newspaper shows that using keyword searches to find similar articles does actually work rather well. And given the state of the technology, this might actually be a very feasible approach, as is requires hardly any new tech to implement it...

I think the main difference - that is often overlooked - here is the fact that the semantic approaches offer one step further. They can find related text that share no keywords at all, but might share neighbours

[btw. the 'bayesian approach' I talk about in my essay is nothing less than a glorified keyword-occurence-comparison...]

Okay, das ist was für mich...

30.04.2003, 10:42 · Martin · Permalink

vom Schockwellenreiter geklaut:
[Numerik] Der Strassen-Algorithmus ist einer der schnellen Algorithmen zur Multiplikation von Matrizen. Der Trick ist im Grunde, die Matrix so in kleinere Matrizen zu zerlegen, dass man weniger multipliziert und dafür mehr addiert. Ist nett in der Werkstatt Multiplikation und in diesem Skript Effiziente Algorithmen und Datenstrukturen erklärt. Wer sich weiter mit Matrizenmultiplikation auseinander setzen möchte, dürfte auf dieser Seite gut aufgehoben sein: Fast Parallel Matrix Multiplication - Strategies for Practical Hybrid Algorithms.

Symbian publishes OPL language source code under LGPL

30.04.2003, 10:27 · Martin · Permalink

being an APL I should be enthusiastic about this... But I'm not sure if this also applies to the Psion 5, and not only to Symbian OS smartphones which I don't own and am not enthusiastic about... (SourceForge project page)

Whatis AudioBooksForFree.Com?

29.04.2003, 09:48 · Martin · Permalink

Semant-O-Matic

28.04.2003, 23:05 · Martin · Permalink

This site indexes the contents of eleven weblogs over the period March - June 2002 ( a total of 1440 posts ). The goal of this search engine is to demonstrate a fuzzy-search technique called latent semantic indexing.

LSI has been around for some time, remember?

my head spins

28.04.2003, 16:25 · Martin · Permalink

I just finished reading the PDF about 'Contextual Network Graphs' I linked earlier...
Way to go for my little experimentation, and those algebraic formular are way beyond my grasp. But the basic idea is rather simple, maybe comparable to the TGGoogleBrowser only not visually. One really strong point is the fact that adding AND removing terms/documents from the 'graph' (or the TDM) is trivially simple. (Not so for VectorSpace/LSI approaches)

All this will of course be handled in my essay, once it is finished, if ever

Maciej's Etech slides on LSI

26.04.2003, 23:04 · Martin · Permalink

Peer-to-Peer Semantic Search Engines: Building a Memex

via Kellan

Projects: Semantic Indexing

Contextual Network Search (CNS) is a graph-based alternative to LSI which NITLE began work on in 2003. In a 1981 dissertation at the University of Illinois, Scott Preece describes an almost identical technique under the name spreading activation search, but we have not been able to find further work in this direction. If you are aware of such work, please let us know.

PDF

New module Search::ContextGraph... that's from April 2002, but it doesn't seem to be in CPAN... ah author: Maciej Ceglowski, okay we know that one from his 'building a memex' slides... So where is that Perl code?

correction: the CPAN date is 27 April 2003, so this stuff is to new to be listed...

similar entries really work

26.04.2003, 02:25 · Martin · Permalink

look what I just found linked from my last post. That is instant gratification if every i've seen it. Usefull further pointers to stuuf that is related and that I linked earlier. And a good point in demonstrating that keyword based search might not have worked here so well. I would have had the intention to find that posts, and the recollection that it actually was there in the first place. I would then had to enter some significant keywords and actively search... this way it just 'poped-up' in the right place for me, as I was exploring the same or similar topic again... neat

Another one of those essays no one reads

26.04.2003, 02:18 · Martin · Permalink

is the product of my experimentations with Mozilla Composer. No, it's not about XUL or JavaScript, but about Keywords, Naive Bayesian and VectorSpace. And I'm pointing to it in all it's unfinished raw not-spellchecked glory. Please feel free to praise and comment it, as I won't have much time this weekend to work on it, but would be very glad for any thoughts you might have about this...

simon

26.04.2003, 00:41 · Martin · Permalink

has built a keyword based 'similar entry' feature for his blog.

Mozilla Composer as writing tool

25.04.2003, 13:26 · Martin · Permalink

Martin Spernau Once again I'm doing a self-experiment. 'Is the Mozilla Composer suited as a writing tool?'.
Usually I come away from these test with the 'well, almost, but not quite' kind of response. I'm hoping that the 1.3 release chaged a ferw things, but it doesn't seem likely. But then mybe the premise if unfair: would I judge Dreamweaver MX in such a way? No, of course not. But then Dreamweaver is a HTML/DHTML authoring tool, not a writing tool. And Composer? Well, it's not a real good HTML authoring tool, although it's by far better than most. It still does too much mucking with my code to be a good authoring tool. And for writing? Hmmmm... let's see.

[update:] About 2 hours into writing an essay about search approaches I could say that, yes Mozilla Composer can be used as a writing tool. Esp. the 'insert Table of Contents' function is mega usefull. COmposer is not a killer writing environment, but definetly usefull. Esp. if you plan to publish your writing in HTML...

[update:] A very early version of the mentioned essay can be found here

blozom

25.04.2003, 12:52 · Martin · Permalink

Blozom is a standalone Mozilla based weblogging and note management application. It uses plain text files. Blog entries have associated XML files with post metadata. Blozom will support wiki like markup and a link glossary.

I wish I could get it to run... almost there now.

moockblog: sweet page turning effect

25.04.2003, 10:30 · Martin · Permalink

trust master-of-the-mask erik-with-a-k natzke to come up with the world's coolest flash page turning effect in his latest work for hp. (the page effect is in the dreamworks section. but other stuff in the site is nice too.)

more VectorSpace thoughts

24.04.2003, 22:56 · Martin · Permalink

here's some timing info for my current setup:
[Thu Apr 24 22:49:50 2003] vs.pl: reading archive from ../html/blog/archive
[Thu Apr 24 22:49:50 2003] vs.pl: found 1257 postings
[Thu Apr 24 22:49:50 2003] vs.pl: setting up VectorSpace
[Thu Apr 24 22:49:50 2003] vs.pl: building Index
[Thu Apr 24 22:49:50 2003] vs.pl: Making word list:
[Thu Apr 24 22:50:01 2003] vs.pl: Finished with word list
[Thu Apr 24 22:50:01 2003] vs.pl: doing queries
[Thu Apr 24 22:50:01 2003] vs.pl: making cosines for doc 0,
2003-04-24_22-33
[Thu Apr 24 22:50:02 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_22-33.sim
[Thu Apr 24 22:50:02 2003] vs.pl:
[Thu Apr 24 22:50:02 2003] vs.pl: making cosines for doc 1, 2003-04-24_22-29
[Thu Apr 24 22:50:03 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_22-29.sim
[Thu Apr 24 22:50:03 2003] vs.pl:
[Thu Apr 24 22:50:03 2003] vs.pl: making cosines for doc 2, 2003-04-24_22-26
[Thu Apr 24 22:50:05 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_22-26.sim
[Thu Apr 24 22:50:05 2003] vs.pl:
[Thu Apr 24 22:50:05 2003] vs.pl: making cosines for doc 3, 2003-04-24_15-53
[Thu Apr 24 22:50:06 2003] vs.pl: writing ../html/blog/archive/sims/2003-04-24_15-53.sim
[Thu Apr 24 22:50:06 2003] vs.pl:
[Thu Apr 24 22:50:06 2003] vs.pl: making cosines for doc 4, 2003-04-24_10-41

so the first optimisation could be to go semi-incremental. No need to re-do all 1200+ entries in each run, as most of them don't really change.
next step surly would be to pre-generate the word-index phase, which could also be used for the keyword search.
each actual 'query' takes roughly a second to preform, that is rather fast if you consider you are comparing one doc to 1200+ others

And no, keeping the whole VectorSpace in-memory between user-queries is not really an option. The Perl VM size for this is about 125MB on my machine, which is ok for running it every so often, but not for running it contiously.

Another gripe is the fact that it's difficult to incrementally add new words (or docs containing new words) to a 'pre-generated' space.

I must say, those similar entries

24.04.2003, 22:33 · Martin · Permalink

are really shaping up. Just go into the [permalink] version of a post you find interesting, and start browsing. I at least can see two things emerging very fast:
a) some topics are re-appearing regularly
b) I do repeat myself

This is almost like a search function, but somewhat better. To search you ned to know what you are looking for, esp. in terms of keywords. VextorSpace similar entries form the other side of that. 'Search/Query by Example' sort of. Now the ultimate would be a combination of both, keyword search and similar entries...

Some ideas/wishes I have for that:

  • VectorSpace search should be älmost instant. Now One complete run takes about 30min. so it only get's generated every 6hours
  • Keyword search and VectorSpace search actually have the same prerequesites to a degree, both need keyword lists per doc, keyword search then uses a reverse keyword index...

The Lessons of Lucasfilm's Habitat

24.04.2003, 22:29 · Martin · Permalink

read later, print out...

wxLua 2.0 availeble

24.04.2003, 22:26 · Martin · Permalink

now built upon Lua 5.0... I'm very curious how it came out...

nexTView EPG Decoder Software

24.04.2003, 15:53 · Martin · Permalink

In this software package you find a decoder for nexTView - an Electronic TV Programme Guide for the analog domain (as opposed to the various digital EPGs that come with most digital broadcasts). It enables you to receive and browse free TV programme listings for all of the major networks in Germany, Austria, France and Switzerland.

requires a TVtuner card in the PC

XMLTV

XMLTV is a set of utilities to manage your TV viewing. They work with TV listings stored in the XMLTV format, which is based on XML. The idea is to separate out the backend (getting the listings) from the frontend (displaying them for the user), and to implement useful operations like picking out your favourite programmes as filters that read and write XML documents.

At present there are backends grabbing TV listings for Canada, the USA, the UK, Germany, Austria, Sweden, Norway, Finland, New Zealand, Spain, Italy and the Netherlands.

freshmeat.net: Project details for FreeGuide TV Guide

FreeGuide TV Guide is a TV guide program which grabs listings for many countries from the Internet and allows you to view them and construct a personalised guide to print out. It uses the XMLTV tools to grab listings, and requires Java 1.4 or above.

Nachdem TVGenial jetzt nur noch an zahlende Kunden das ganze Listing liefert...

Der Schockwellenreiter

24.04.2003, 10:41 · Martin · Permalink

ist auch schon drei? Man sind wir alle alt ;)


alles Bild, Text und Tonmaterial ist © Martin Spernau, Verwendung und Reproduktion erfordert die Zustimmung des Authors

Martin Spernau
© 1994-2026


amazon.de Wunschliste

Facebook me!
Google+

powered by Traumtank