| LED Digest 1864: Keeping Spiders Under Control |
|
|
|
================================================== The LED Digest Moderated Discussion List "Effective Online Advertising, Since 1997" pair Networks: The LED's Web Host Hosting and Domain Reg. from a Trusted Leader pair.com for Hosting | pairNIC.com for Domains ================================================== Guest Moderator: Published by: Veronica Yuill LED Digest post,led-digest.com http://www.led-digest.com ................................................ Septmember 2, 2004 Issue #1864 ................................................ .....IN THIS DIGEST..... ==== CONTINUING ================= --== Email Autoresponders ==-- ~ Mike Koenigs "Over the past four years, I've investigated and/or used over 25 different services and have tried over 15 different open source or "low cost" DIY autoresponders..." ~ Joe Halbrook "There's no cross-paddling against the open standards for exchanging information taking place..." ~ Michael Martinez "All innovations have to start somewhere, and they should not be laughed at. They should be tested, evaluated, and then used if found useful or discarded if they prove to be unworthy..." <Guest Moderator Comment> --== Is Microsoft Crawling My Site? ==-- ~ Pat McCarthy "There are a few OSCommerce contributions that you could implement that help spiders browse an OSCommerce store correctly..." ~ Tom Aman "Possibly someone has managed to hijack one or more of their systems ... and the company will take appropriate action once they are informed of the problem..." ~ Paul Bedford "It could be a problem with MSNbot choking on session IDs..." ===== CONTINUING ================================= From: Mike Koenigs Subject: Autoresponders Hi Brett, Over the past four years, I've investigated and/or used over 25 different services and have tried over 15 different open source or "low cost" DIY autoresponders and come back to two solutions for their usefulness, affordability and (general) lack of blacklisting. Of course they all will get blacklisted by some services now and then because many users who sign themselves up are lazy, don't read when you state the "Click here to unsubscribe" and instead report your newsletter as spam. Everyone suffers for this. And obviously, there's a bad apple that purposely or innocently imports a few names that get them (and us) in trouble. But by and large, I've been using two services to mail to lists that have as many as 16,000 names at a time without any problem at all. Most of the time the lists are in the 200-2000 size. Here are my two choices that have continuously given me results: For years I've been using "Reply2It" (www.reply2it.com) as my autoresponder of choice. The system is flexible, easy to use, reasonably-priced and reliable. It's not filled with gobs of wild and crazy features, but it allows for up to 15 custom fields in addition to the usual suspects (name, address, city, state, zip, yadda, yadda). I even use it at times to create "forward to friend" capabilities for Flash presentations. This way you can capture information and send a personalized HTML message to the referred friend or contact AND track who sent it. If you're interested in seeing how I've done this, write to me directly and I'll send you a link to a live client presentation. If you'd like to see a simpler example, visit http://www.sanoviv.com/newsletter/august_newsletter.html and click on the "Send this newsletter to a friend" link. If you fill in the form, you'll see how it works. There's a nice "tracker" system built-in (actually a redirector) so you can set up basic metrics as well. The other autoresponder that solves my bigger business challenges for such sites as reByte www.rebyte.com (a startup), www.sanoviv.com, www.gegax.com, www.kidseatgreat.com (all clients), I use the built-in autoresponder that 1ShoppingCart.com provides. Their system is completely integrated with their shopping cart, lead generation system and a slew of other tools that allow a company to manage a fairly complex web business affordably. (BTW, I'm only a user of these systems, I have no business relationship with either of them). MAK MAK Koenigs "Breaking The Rules Without Breaking The Law" Results-driven marketing and strategy www.mrbz.com ===== CONTINUING ================================= From: Joe Halbrook Subject: Autoresponders In LED 1862, Veronica Yuill wrote: >> ... an increasing number of people are switching to other browsers. At least RSS feeds are browser- and OS-independent. << Hi Veronica, That's the beauty of this system (EZ-Feeds). It's an addition to your current mailings - not a replacement for them. The key is to increase deliverability and, thus, readership. Yes, Windows users will benefit from it tremendously. But the smaller subset of non-Windows users will still obtain your mailings as usual, via email. The goal was to allow email publishers to continue to publish the way they always have, without having to resort to RSS and other delivery technologies that require "re-tooling." This system provides a way to deliver those mailings seamlessly to the obviously larger audience out there who use the Windows- based OS, while not affecting the delivery of your mailings to those who do not use the Windows OS. There's no cross-paddling against the "open standards for exchanging information" taking place. Joe Halbrook Permission Technologies - Bypass the SMTP protocol. http://www.ez-feeds.com ------- new post - same topic ------- From: Michael Martinez Subject: Autoresponders In response to Joe Halbrook's ez-feeds post, Veronica wrote in Digest 1862: >> Be careful with this type of solution! A publication I subscribe to recently switched from email to a feed supplied via a browser toolbar. When I read the system requirements and discovered that it only works with IE/Windows, I gave a hoot of incredulous laughter and promptly deleted it from my list of subscriptions.<< Browser toolbars, unfortunately, are going to be with us for a long time to come, and many of them work in multiple browsers. I know people who install just about every toolbar they are offered, and their computers are messed up because of the toolbars. So, I am sure many companies will continue to offer toolbars. >> This type of proprietary system doesn't seem to me to be the way forward -- it runs counter to the entire ethos of the Internet, which is all about open standards for exchanging information. << This type of proprietary system is precisely how the Internet was built and continues to evolve. Innovations always start out as small changes which someone tries on a limited scale. While many good innovations die unfortunate quick deaths, virtually all the major pieces of Internet technology started out as proprietary implementations. There was a time when Microsoft's Internet Explorer didn't support frames, and now we have frames everywhere. There was a time when Microsoft's Internet Explorer didn't support Javascript, and now we have Javascript everywhere. There was a time when ... well, my point is that, even though Internet Explorer is itself a proprietary system, it is currently the driving force in many Internet-based applications. It wasn't always so, and hopefully, one day, we'll have some better choices than we currently do (and I know about Mozilla, Opera, etc. -- that isn't the point). Nonetheless, all innovations have to start somewhere, and they should not be laughed at. They should be tested, evaluated, and then used if found useful or discarded if they prove to be unworthy. Michael Martinez [Moderator comment] Sorry Michael, but I have to take issue with your claim that "This type of proprietary system is precisely how the Internet was built and continues to evolve ... virtually all the major pieces of Internet technology started out as proprietary implementations". The Internet originated as a US government-funded project. TCP/IP, the protocol that underlies everything on the Internet, is a non-proprietary open standard. So are FTP and SMTP, and all the other basic protocols that make the Internet work. When Tim Berners-Lee invented the Web in 1989, he was working for public research organisation CERN in Geneva -- his idea was to find a way of providing CERN with a "collective memory", making it easy to retrieve documents wherever they were. There's a wonderful moment in his book "Weaving the Web" where he describes his worry that CERN might choose to patent HTTP and HTML -- and his relief when they decided to make them freely available to the world without payment. A few years later, before Bill Gates "got" the Internet, Microsoft tried to launch a proprietary competitor to the Web, codenamed Blackbird, expecting it to be a runaway success -- it was buried before it even got close to release. Nowadays, Microsoft participates in open standards organisations such as the W3C. OK, rant over! Sorry, this is one of my hobby horses. The story of how the Internet evolved is really fascinating though. If you want to find out more, I can recommend two books: "Where Wizards Stay Up Late" by Katie Hafner and Matthew Lyon (a bit dated, but lots of interesting detail on the early days) and "A Brief History of the Future" by John Naughton, an easy read which brings the story (almost) up to date and explains why open standards are so vital to the continuing health of the Internet. ~ Veronica ------- new post - new topic ------- From: Pat McCarthy Subject: Is Microsoft Crawling My Site? In response to Sarah Hayes' question about what to do about Microsoft spidering her OSCommerce site, there are a few options. First, there are a few OSCommerce contributions that you could implement (or pay someone to implement) that help spiders browse an OSCommerce store correctly. Essentially they keep it from getting trapped and increasing your bandwidth load. Here is one of those contributions: http://www.oscommerce.com/community/contributions... Second, you could create or edit your robots.txt file to disallow spiders from traveling to certain areas of your site that are unimportant for a spider to be indexing. Here is an example for an OSCommerce site: User-agent: * Disallow: /admin/ #Block files that are secure or login oriented Disallow: /account.php Disallow: /account_edit.php Disallow: /account_edit_process.php Disallow: /account_history.php Disallow: /account_history_info.php Disallow: /account_newsletters.php Disallow: /account_notifications.php Disallow: /account_password.php Disallow: /address_book.php Disallow: /address_book_process.php Disallow: /advanced_search.php Disallow: /advanced_search_result.php Disallow: /checkout_confirmation.php Disallow: /checkout_payment.php Disallow: /checkout_payment_address.php Disallow: /checkout_process.php Disallow: /checkout_shipping.php Disallow: /checkout_shipping_address.php Disallow: /checkout_success.php Disallow: /create_account.php Disallow: /create_account_process.php Disallow: /create_account_success.php Disallow: /disclaimer.php Disallow: /download.php Disallow: /info_shopping_cart.php Disallow: /login.php Disallow: /logoff.php Disallow: /logoff.php Disallow: /password_forgotten.php Disallow: /popup_image.php Disallow: /popup_search_help.php Disallow: /product_notifications.php Disallow: /product_reviews_write.php Disallow: /redirect.php Disallow: /shopping_cart.php Disallow: /shopping_cart_help.php Disallow: /tell_a_friend.php Disallow: /contact_us.php Pat McCarthy Palo Alto Software http://www.paloalto.com/ ------- new post - same topic ------- From: Tom Aman Subject: Is Microsoft Crawling My Site? If you go to the DNS Stuff: DNS tools... site (http://www.dnsstuff.com/), you can find out not only who has these IP addresses, but everything you need to know to complain. The addresses you give (in fact the whole range 65.52.0.0 - 65.55.255.255), is shown as owned by Microsoft. But, more important for you, it also gives the email address to write to complain about abuse ( This email address is being protected from spam bots, you need Javascript enabled to view it ). Possibly someone has managed to hijack one or more of their systems (could even be a disgruntled employee) and the company will take appropriate action once they are informed of the problem. (But before complaining to Microsoft, you might contact osCommerce to see if they know what is happening). Tom Aman Aman Software (http://www.cyberspyder.com) Home of CyberSpyder Link Test ------- new post - same topic ------- From: Paul Bedford Subject: Is Microsoft Crawling My Site? It could be a problem with MSNbot choking on session IDs. Have a look at http://search.msn.com/webmasters/msnbot.aspx for possible remedies. Paul Bedford ------------------------------------------------------- The LED Digest is sponsored by pair Networks: pair.com for Hosting | pairNIC.com for Domains Copyright 1995-2004 Adam Audette. All Rights Reserved. ----------------------------------------------------------------- "Don't tell people how to do things. Tell them what to do and let them surprise you with their results." - George S. Patton |
