little boxes made of words

1. On Sat, Oct 14 at 1am,
   fluffy said —

   I totally agree. I’ve actually written about this not too long ago on my weblog, and have been thinking about this a lot recently. People could have an intelligent RSS agent which checks per-page RSS feeds for every page they go (with some sort of mechanism to make it only check for updates at the rate at which updates actually occur), and then to extend that with people being able to publish and subscribe to each others’ clickstreams as a sort of emergent recommendation network (FOAF and OpenID would be good underlying transports for clickstream discovery).

   Several weblogs have started furnishing per-post comment feeds, which is a great way to actually syndicate comment updates without having to actually register for every single site or keep on overloading the poor, abused email infrastructure, but working with that is pretty unwieldy if you have to manually manage the feeds in a reader, and if there’s a forum which you watch which has a lot of short-lived topics, having a hundred users’ feed readers subscribed to a thousand threads and checking for updates every hour would be something of a problem.

   All of the current forum software out there is so ad-hoc and hacky, though, and getting people to upgrade their fora will be impossible. Personally I think the whole forum concept is a bit hacky to begin with, though. Distributed identity/authentication systems with smart filters based on a trust network, with comments dual-published via HTML and RSS/Atom/whatever, would be ideal. I’ve always thought of the Internet itself as one big community; why set up oodles of walled gardens which duplicate effort? Keep things lightweight and trim. Weblog comment systems can be made so much better, and conversations should be able to happen anywhere, not in The Forum For That Site That People Visit.

   The Internet is the medium, not phpBB or what have you.
2. On Sat, Oct 14 at 10am,
   Jonathan Deutsch said —

   Have you seen dappit? While I doubt it could solve all scraping woes, it looks like a pretty neat approach, letting users create scrapers by pointing and clicking.
3. On Sat, Oct 14 at 11am,
   fluffy said —

   If you just want to support a scraping model for update notification, there’s always stuff like RSSpect which just does a diff on the current and previous version of a page and adds a feed item if it’s different enough (and that mode of operation is also something which would make sense in the context of my clickstream ramble).
4. On Sat, Oct 14 at 12pm,
   Jens Alfke said —

   Re: RSSpect — the diff-the-page idea goes back to IE 4 (at least the Mac version) which had this as a built-in feature. It never worked well, because most popular sites include content that changes on every page view, whether it’s rotating ads or a hit-counter.

   Their more advanced model where you put SPAN tags in your content to indicate posts is precisely the same idea as the hAtom microformat (although the word “microformat” isn’t mentioned on that page.) It’s a reasonable idea, although it seems like nearly as much work as creating a real feed, only less efficient (since clients have to pull down all the non-semantic markup every time, too.)

   Also, the notion of commenting out the tags to make them not visible on the page is horrible — it means you can’t process the page using a real *ML parser, and implies they are simply grepping the HTML with regexps, which is even more fragile than DOM-based scraping.
5. On Sat, Oct 14 at 12pm,
   fluffy said —

   Well, yeah, I was referring to RSSpect’s “anysite” function, though it has various thresholds for the amount of a diff to consider an update. Simply changing links on an image won’t trigger it unless you set it to the most sensitive.

   I haven’t looked into the other stuff, I was only looking into anysite because I wanted to monitor content by a person who is, like, religiously opposed to RSS for some reason. The other methods are pretty silly because you’d have to have a pretty braindead CMS to not allow you to just provide an RSS view in addition to the normal HTML view.
6. On Sun, Oct 15 at 10am,
   leeg said —

   Don’t get me wrong, I absolutely love the idea of an elegant, unified user interface for distributed many-to-many discussions. I’ve been obsessed by it for years, in fact. But the real way to do this is to build protocols and servers that make it work, and then robust and reliable clients can be plugged into them

   Actually, I’m pretty sure Usenet already exists. ;-)
7. On Sun, Oct 15 at 11am,
   Matt Tavares said —

   What i hate most about my dream app is the fact that all the ideas down right suck or just cannot be implimented. I think their is one that actually puts a fake window on your computer to sync with the weather outside.

   If you can’t fucking get up to check a window, then you have problems.
8. On Sun, Oct 15 at 11am,
   fluffy said —

   Yeah, and then there’s all those apps which have slick professional sites which make it seem as if the app already exists, and they’re all like “hurr blurf duh we’ll do everything else too and maybe more and totally revolutionize the world.”

   The ONLY app I was even remotely interested in was Whistler and it still seemed pretty darn infeasible. Pitch detection and correction are actually difficult, especially when you’re dealing with people blowing on a $10 mic.
9. On Sun, Oct 15 at 11am,
   e:leaf said —

   The nice part about My Dream App is that it removes the ability to create programs from the top floor of “The Ivory Tower.” Perhaps that’s one reason why some programmers are against the idea. Non-programmers now have an outlet to get their ideas into practice, and some of those ideas may be damn good. If the programmers want to deal with feasability issues, then let them do it.

   I know that there have been a few good ideas that I have come up with over the years that I simply couldn’t implement because I don’t program. Why should application design and concept be left to coders alone?
10. On Sun, Oct 15 at 11am,
    julian said —

    e:leaf, as much as “The Ivory Tower” is something you don’t want users to feel they are up against, the concept Jens references isn’t just some notion programmers hold.

    In fact, in the field of Human-Computer Interaction (usability), we are taught over and over again that information about what to build comes from studying users, their environment, and their responses to questions… but NOT from directly asking them what they want.

    The central problem is that users think they know what they want, but they do not actually know what they want in terms of good solutions to do the problem.

    As much as people may believe that the programmers and designers are all ignoring the users, historically the programmers are the people who just go out there and implement anything the user asks for.

    This problem is reaching the point where it is so well-understood that it is now dealt with even in the Software Engineer’s world. A Google search these days turns up results from Agile Software Development sites as well as usability sites.

    Agile Software Development and Human-Computer Interaction programs alike teach that the solution to the problem you describe is more user involvement in the process of developing software—but not directly implementing what the user tells you they want, rather doing the harder work and figuring out what kind of solution would really solve their need.
11. On Sun, Oct 15 at 2pm,
    Colin said —

    If only there were some sort of message board system with a specific set of protocols.
12. On Sun, Oct 15 at 2pm,
    Jens Alfke said —

    lee, Colin: Yup, I was a big Usenet user back in the mid-80s (I remember the transition to the hierarchical newsgroup system, i.e. when net.music became rec.music.misc) and to a lesser extent in the early ’90s (I remember when the flood of people announcing Kurt Cobain’s death provided graphic evidence of how the lag of message propagation could make communication impossible.)

    Usenet had some nice properties, but sucked at least as badly as today’s BBS sites. Some of my early designs for The Perfect Social Software were based on extensions of NNTP, but I soon gave up on that as Usenet sank from view beneath the crushing weight of spam.
13. On Sun, Oct 15 at 3pm,
    Danny Cohen said —

    There is a demo that the Hijack-conciever came up with that shows that DOM-scraping could be as easy as clicking on elements once on a forum, and then it remembers what to look for on every page.

    stop being so negative, they are “dreams” after all. You have problems with dreams?
14. On Sun, Oct 15 at 3pm,
    Joshua said —

    Thanks for saying what many of us have been thinking all along. You have to remember too that half the people at MyDreamApp are 13 year olds from MacThemesForums who only care about two things, Forums and Pretty GUI’s.
15. On Sun, Oct 15 at 3pm,
    Jens Alfke said —

    Danny: I fail to see how a mocked-up demo can show the feasibility of implementing something.

    These aren’t “dreams”, they are entries in a competition to actually build a real app. Rather, to have someone else who can program build a real app. If the concept is going to drive that programmer insane, and if people who know nothing about programming keep insisting that they (somehow) know it’s “as easy as clicking”, I think that’s a valid thing for me to be negative about.
16. On Sun, Oct 15 at 3pm,
    Jon Aizen said —

    Dapper could definitely be used to build a meta-forum application.

    Yup. In an amazing feat of hand-waving, this app is going to Just Work with every BBS program under the sun, through the miracle of screen-scraping. An army of volunteers are going to code and maintain scripts that will strain through the murky tag soup and extract every bit of information you need.

    Dapper does a great job of solving this difficulty. Using a GUI, you can construct an API for any website without a writing a line of code (nor a single regular expression to parse that nasty tag soup).
17. On Sun, Oct 15 at 4pm,
    pwb said —

    What a joke! You rag on screen scraping and then propose that the “right” way to do it is for everyone to get together and agree on some common format and then throw in some OpenID single sign on, blah, blah, blah. Are you freaking kidding me?
18. On Sun, Oct 15 at 5pm,
    Jens Alfke said —

    Jon: Dapper is interesting, and might go some way toward making this workable (though the devil is in the details and I’m not sure enough of the semantics can be extracted reliably.) I do maintain that this entire approach is undeniably a kludge, though, the web equivalent of OCR.

    pwb: No, I am not fucking kidding. What’s your point? Plenty of advances have come about from people getting together to develop and standardize new protocols, and conversely, plenty of kludged-together systems have collapsed under the weight of their own messes.
19. On Sun, Oct 15 at 7pm,
    Scott Stevenson said —

    Where’s the fun in working on something that’s feasible?
20. On Mon, Oct 16 at 2am,
    Maciej Stachowiak said —

    How about getting forum software to add hAtom markup? Then you can unambiguously pull content from the actual web pages without screen scraping hacks, and the software does not have to be updated to provide the data via alternate formats or protocols.

    I’ll add though that I think “screen scraping” on the web (more accurately it would be called markup scraping) is fundamentally different than the original meaning — trying to infer information from pixels. You already have the text, you’re just trying to scrape the semantics - and markup is supposed to be about semantics.

    This is why I think microformats are a good approach because they make the markup richer to let you extract the semantics you want in a reliable way, without the need for out-of-band metadata.
21. On Mon, Oct 16 at 4am,
    SuitCase said —

    Part of the Daring Fireball flood, sorry if you have ten million messages to moderate.

    I am immensely in love with the idea of Hijack, I’ve been wishing for a rich client that accesses content from forums for many years and had fantasies about learning how to code myself just so I could try and make it.

    Yes, the collection of data will be hard, however I think you are exaggerating the difficulty it would involve. That Dapper thing looks promising enough, but it already seemed mostly feasible to me because some forums use semantic markup (or at least, informative CSS stylings), offer information-rich RSS feeds, or (probably the easiest thing) a default skin\the option to choose the default skin in the user prefs. Imagine - if you had support for the subSilver theme of phpBB alone, you’d probably then support 10% of the forums on the internet. Do the same for the default looks of vBulletin, punbb, Invision and.. heck, I dunno, UBB? Then throw in the ones that can’t be changed, like ezboard and proboards.. And you’ve probably covered 50% or more of all forums, and I bet those “forum definitions” for stuff like subSilver would work with a lot of custom themes as they are almost always heavily based on the default. So even if we take it as “50% can feasibly work”, it’s a useful app already, and all that needs to be accomplished after that is a training mode and\or definition file sharing community, both concepts which are being discussed in the forum for the app.

    Regardless of “rarely listen to the user” philosophy, I think your idea that people should start writing protocols and microformats or whatever for forums clearly shows a programmer-oriented perspective that is apathetic to what users like I really want from this application. I don’t care to use an app that works with the five hyper-cool Web 2.0 sites that bothered to implement a weird new protocol. I want something that works with the web as a whole, and I think that’s a lot more feasible than you’re indicating.
22. On Mon, Oct 16 at 8am,
    Jens Alfke said —

    Maciej: My guess is that adding semantic markup to the HTML page (like hAtom) is about the same difficulty as adding an RSS/Atom feed. But it’s less flexible, since it is constrained to show only the same set of entries that are on the page, and the content only in its post-themed state.

    SuitCase: You raise some good points, although I am still not convinced it will work well enough. But perhaps some others have a looser definition of “well enough”.

    By “apathetic” I think you mean “antithetical”? I don’t think we’re really at odds in what we want. I’m just looking at it from what I think is a broader perspective: I would rather not have a ton of work go into something that is at heart a kludge.

    Remember, today’s “hyper-cool Web _.0 … weird new protocol” often becomes tomorrow’s everyday reality. RSS used to be that weird new protocol. So did CSS, Flash, JavaScript, tables, inline JPEGs. I’m sure when Mosaic came out the Gopher/Archie folks said “forget this foo-foo ivory-tower ‘hypertext’ stuff, we just want something that works with the FTP sites we already have.”
23. On Mon, Oct 16 at 8am,
    Step said —

    Hmm, my links got eaten in my last comment, and I don’t see a way to edit.

    Either way, I see from your last response that you’re approaching this from a different perspective. I can appreciate that, and I agree that I’d like to see forum software change to include an appropriate standard. I’m sure it could happen, too, just not sure what would push it forward. You say “today” and “tomorrow”, but in reality it took a couple years to catch on. Forum software could take just as long, maybe it will transition faster, maybe slower. I’ve never created and moderated my own forum, so I don’t really know.

    I hope my comments have been helpful. I’m mainly concerned that it seems your article unfairly attacks two ideas (the whole MDA competition, and Hijack) based on their weaknesses, without seeing if those weaknesses were acknowledged or addressed in any way. Another words, I’d hope for more balance. But then again, you’re coming from quite a different perspective, and I’m not a regular reader of your blog, so perhaps I’m missing something vital to the discussion here.
24. On Mon, Oct 16 at 2pm,
    Eduo said —

    I keep scanning mydreamapp.com periodically and keep voting for the seemingly least-popular projects. There were some nifty ideas among all the crap sent and, to me, the filtering hasn’t been that good but, then again, that speaks about the voters more than anything.

    I was about to submit an idea to it, back when it started but after seeing what the people were raving about and what’s being the most successful pitches I realised mine wasn’t as flashy and, indeed, too techy for most of the reviewers.

    Several other people did try to do this, and seemingly failed because their mockups weren’t pretty (there isn’t any project right not in the semifinals that isn’t shiny and/or animated). In the end I didn’t even see if anyone had proposed anything like I had thought (some did propose other ideas I had thought of, and were summarily buried to the bottom of the lists).

    *sigh*, maybe next time.
25. On Mon, Oct 16 at 4pm,
    shadownight said —

    Dear blogger and commenters,

    Jason Harris, Developer at My Dream App has repeatedly stated that Hijack is is feasible. He suggests to do it not through HTML, but through DOM scraping. For more info, you can check out the Official Feasibility thread in the MDA forums and other threads in the Hijack section: http://mydreamapp.com/forums/viewtopic.php?id=1263
26. On Mon, Oct 16 at 6pm,
    fluffy said —

    Yeah, except most forum software doesn’t put out compliant XHTML, so you need to use a smile-and-nod HTML parser which will convert to DOM, or a prefilter which de-moronizes the HTML.
27. On Tue, Oct 17 at 5am,
    Eduo said —

    Shadownight: That may have been stated, but DOM scraping is really not diferent than HTML scraping unless you’re working on a standard (and then, it isn’t either, just more efficient).

    It’s not possible to build an “universal scraper” of any sort when there is no standard on what’ll be scraped.

    Unless every forum in the world decides to go the route of tagging specific parts of the forum with IDs or with classes this won’t be feasible without hordes of people creating “plug-ins” for each different forum (bear in mind that even generic forum software like phpbb, yabb and bbpress allow for theme customisation, where the theme can be pretty much whatever you want).

    I’d dare to say that probably bbpress would be the best candidate for an app of this sort, as it can provide RSS feeds of its forums and posts, and posting can be done through the methods defined in the RSS. But that means the best candidate is the newcomer in the arena. Existing major forum apps would need to change their releases to add a functionality like this and even then older forums that won’t be updated won’t benefit from it.

    The route to do this is not to make an app that does scraping, it’s to propose a standard for it and lobby this standard to the major forum programmers (and to bbpress, I insist. Wordpress’ weight is too much to ignore and integration with WP itself makes bbpress a probably quick-raiser to the top-five). Along with the protocol an app can be designed that takes advantage of it, but the protocolo needs to exist so other platforms and other forum bases can adopt it as well.

    That’s how RSS started, by the way.
28. On Tue, Oct 17 at 5am,
    SuitCase said —

    Jens - Hey, thanks for the response. Reading myself again I hope I didn’t come off too hostile, but I’m passionate about this idea and don’t want it killed :P

    Grammatically “apathetic” looks a bit awkward there but I still meant it, though by using the word I was exaggerating your position in an attempt to show that for this app your “programmer’s logic” didn’t fit. While the nicest way to do this would be through standards, this is not in the interest of a user who wants to monitor their favourite forums _now_, and not just a scant few that are modeled on some sort of new protocol (which, as a commenter above mentioned, is kind of already done with Usenet.)

    As for the idea that Hijack is a kludge, I dunno. A lot of apps I use feel like they’re messed beyond recognition and could be nicer in an ideal world, but they cope with the problems and I make use of them very effectively - my web browser and my mail client, for one. I really don’t see a future where web forums become abandoned for rich clients, I see them as a minority thing for the quirky power users on quirky platforms like the Mac, and thus I think it’s acceptable that they require some maintenance and not be as perfect and simple as they could.

    It’s funny that I’m arguing that for a “My Dream App” competition, but I honestly think it’s far less feasible for a client based on standard forum-provided data to be a good product than one that spends a lot of time trudging through messy forum templates. There’s not enough demand for a forum aggregator for a standard to be adopted, basically, and so I don’t think the examples you cited are directly comparable as that kind of ground-up approach would result in a client for a specification nobody cares about and never will.
29. On Tue, Oct 17 at 9pm,
    Jens Alfke said —

    SuitCase — it doesn’t take rewriting the forums, and it won’t require writing all-new protocols. Most of the job can be done using Atom syndication (which many forums already support, although buggily) with the special secret-sauce of the threading extensions.

    These happen to be implemented for WordPress already, as a plugin.

    That covers a lot of the subscription side of what Hijack would need. There’s more to do; I’m going to write a post describing this in more detail.
30. Name:
    Email:
    Website:
    Name and email are required. Thanks!
    
    [Or log in]    [Or register]

    Or use your OpenID:
    LiveJournal username DeadJournal username Other OpenID URL

    HTML: You may use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <code> <em> <i> <strike> <strong>

    Notify me of followup comments via e-mail

Design: Hemingway, modified by Chris Gerlach, further modified by me, Jens.

Entries feed   Comments feed