Comments on: Browser.py http://www.kryogenix.org/days/2003/11/30/pybrowser scratched tallies on the prison wall Wed, 03 Dec 2008 23:10:39 +0000 http://wordpress.org/?v=2.6.5 By: Simon Willison http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1956 Simon Willison Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1956 It's not working for me - what do I need to install for the "from xml.dom.ext.reader import HtmlLib" bit to work? It’s not working for me - what do I need to install for the “from xml.dom.ext.reader import HtmlLib” bit to work?

]]> By: sil http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1957 sil Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1957 Darn, I thought that came with Python. :) It's the Python/XML distribution (Debian package python2.3-xml) -- I think the HtmlLib stuff is from 4DOM, the FourThought Python XML stuff. It appears that the Debian people collect together a load of Python stuff and bung it into one package, which is nice of them but makes it difficult to recommend a package to non-Debian-Linux-people.<br /> <br /> I'm open to other suggestions for an HTML parser; all I really need is something that can parse even broken HTML and give me a DOM tree out of it. I tried using the Twisted people's @microdom@ first but gave it up in favour of HtmlLib, and even that I had to patch twice in the code (once to cope with bad namespaced elements, and once to fix a bug). Any better suggestions for an HTML -> DOM parser in Python, say the word, especially if it's either easy to distribute with browser.py or comes with Python by default! Darn, I thought that came with Python. :) It’s the Python/XML distribution (Debian package python2.3-xml) — I think the HtmlLib stuff is from 4DOM, the FourThought Python XML stuff. It appears that the Debian people collect together a load of Python stuff and bung it into one package, which is nice of them but makes it difficult to recommend a package to non-Debian-Linux-people.

I’m open to other suggestions for an HTML parser; all I really need is something that can parse even broken HTML and give me a DOM tree out of it. I tried using the Twisted people’s @microdom@ first but gave it up in favour of HtmlLib, and even that I had to patch twice in the code (once to cope with bad namespaced elements, and once to fix a bug). Any better suggestions for an HTML -> DOM parser in Python, say the word, especially if it’s either easy to distribute with browser.py or comes with Python by default!

]]> By: Chris http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1958 Chris Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1958 You might want to take a look at Fredrik Lundh's ElementTidy, or mxTidy which both use a library version of Dave Raggett's HTML Tidy utility to fix any problems with the HTML before converting it to XHTML (the XML version of HTML)<br /> <br /> <a href="http://www.effbot.org/zone/element-tidylib.htm">http://www.effbot.org/zone/element-tidylib.htm</a> You might want to take a look at Fredrik Lundh’s ElementTidy, or mxTidy which both use a library version of Dave Raggett’s HTML Tidy utility to fix any problems with the HTML before converting it to XHTML (the XML version of HTML)

http://www.effbot.org/zone/element-tidylib.htm

]]> By: Henning http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1959 Henning Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1959 The webunit package contains a module called SimpleDOM:<br /> <a href="http://www.mechanicalcat.net/tech/webunit/README.html#simpledom">http://www.mechanicalcat.net/tech/webunit/README.html#simpledom</a><br /> It seems useful. Besides webunit seems <br /> in some ways similar to Browser.py The webunit package contains a module called SimpleDOM:
http://www.mechanicalcat.net/tech/webunit/README.html#simpledom
It seems useful. Besides webunit seems
in some ways similar to Browser.py

]]> By: sil http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1960 sil Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1960 <code>SimpleDOM.NestingError: Open tags <html>, <body>, <table>, <tr>, <td>, <table>, <tr>, <td>, <script> do not match close tag </iframe>, at line 76, column 290</code><br /> The major problem here is needing to parse invalid HTML, which most of the web is. The effbot interface to tidylib would work but requires a C extension. mxTidy certainly didn't used to be an interface to tidylib -- instead, it called the tidy executable directly. There are other tidylib interfaces, some of which need ctypes or similar... SimpleDOM.NestingError: Open tags
<html>, <body>, <table>,
<tr>, <td>, <table>,
<tr>, <td>, <script> do not
match close tag </iframe>, at line 76, column
290
The major problem here is needing to parse invalid HTML, which most of the web is. The effbot interface to tidylib would work but requires a C extension. mxTidy certainly didn’t used to be an interface to tidylib — instead, it called the tidy executable directly. There are other tidylib interfaces, some of which need ctypes or similar…

]]> By: Geoff http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1961 Geoff Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1961 The twisted guys have microdom, which is specifically designed to correct nonstandard HTML, and can probably be easily lifted out of the package. The twisted guys have microdom, which is specifically designed to correct nonstandard HTML, and can probably be easily lifted out of the package.

]]> By: Kevin Dahlhausen http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1962 Kevin Dahlhausen Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1962 This is great stuff. Thanks. The XML parser from:  <a href="http://pyxml.sourceforge.net/topics/download.html  ">http://pyxml.sourceforge.net/topics/download.html</a><br /> works well.<br /> I added a little patch for titles:<br /> <pre><br />   def getTitle(self):<br />   try:  <br />   titleNode = self.__HtmlDom.getElementsByTagName('title')[0]<br />   return self.__getInnerText(titleNode)<br />   except:<br />   return ""<br /> </pre> This is great stuff. Thanks. The XML parser from:  http://pyxml.sourceforge.net/topics/download.html
works well.
I added a little patch for titles:

  def getTitle(self):
  try:  
  titleNode = self.__HtmlDom.getElementsByTagName('title')[0]
  return self.__getInnerText(titleNode)
  except:
  return ""
]]> By: Stefan Champailler http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1963 Stefan Champailler Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1963 I used PyXML as well and indeed it works fine. Just noticed that if I install it with the setup.py's defaults, it gets installed in site-packages/_xmlplus... python can't find it there, it must sit in site-packages/xml. (I may be wrong, I'm a newbie at Python) I used PyXML as well and indeed it works fine. Just noticed that if I install it with the setup.py’s defaults, it gets installed in site-packages/_xmlplus… python can’t find it there, it must sit in site-packages/xml. (I may be wrong, I’m a newbie at Python)

]]> By: Martin Eliasson http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1964 Martin Eliasson Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1964 I have had similar problems. I choose to compile tidy as a standalone tool, pipe the data i needed to parse through it using popen2 and then parse it. No problem, except figuring out encoding stuff for strange sweedish letters. I have had similar problems. I choose to compile tidy as a standalone tool, pipe the data i needed to parse through it using popen2 and then parse it. No problem, except figuring out encoding stuff for strange sweedish letters.

]]> By: sil http://www.kryogenix.org/days/2003/11/30/pybrowser#comment-1965 sil Thu, 01 Jan 1970 01:00:00 +0000 http://www.kryogenix.org/adpb/2003/11/30/pybrowser/#comment-1965 Geoff: I did use microdom first, but I stopped using it in favour of HtmlLib (although annoyingly I can't remember why!). Geoff: I did use microdom first, but I stopped using it in favour of HtmlLib (although annoyingly I can’t remember why!).

]]>