Advertisement:

Author Topic: More Spiders v1.2  (Read 87641 times)

Online SleePy

  • Let there be light!
  • Site Team Lead
  • SMF Master
  • *
  • Posts: 30,786
  • Gender: Male
  • Thats his happy face.
    • jdarwood007 on GitHub
    • @jdarwood on Twitter
    • SleePy Code - My personal site
More Spiders v1.2
« on: April 10, 2008, 11:01:45 PM »

Compatibility
For SMF 2.0 Beta 3 Public, 2.0 Beta 3.1 (and above)
It will not work for ANY earlier versions.

Introduction
Adds 77 more spiders/crawlers/bots to your Spiders section in SMF

Features
o 83 Spiders belonging to search engines, validators, checkers, crawlers, bots, software, etc.
- Including; Ask, Baidu, GigaBot, Google-AdSense, Google-Adwords, Google-SA, Google-Image, InternetArchive, Alexa, Omgili, Speedy Spider, Yahoo, Yahoo JP, DeadLinksChecker, W3C Validator, W3C CSSValidator, W3C FeedValidator, W3C LinkChecker, W3C mobileOK, W3C P3PValidator, Bloglines, Feedburner, SnapBot, Picsearch, Websnapr, AllTheWeb, Altavista, Asterias, 192bot, AbachoBot, Abdcatos, Acoon, Accoona, BecomeBot, BlogRefsBot, Daumoa, DuckDuckBot, Exabot, Furl, FyperSpider, Geona, GirafaBot, GoSeeBot, Ichiro, LapozzBot, Looksmart, Lycos, Majestic12, MLBot, MSRBOT, MSR-ISRCCrawler, Naver, Naver, NoxTrumBot, OmniExplorer, OnetSzukaj, ScrubTheWeb, SearchSight, Seeqpod, Shablast, SitiDiBot, Slider, Sogou, Sosospider, StackRambler, SurveyBot, Touche, Walhello, WebAlta, Wisponbot, YacyBot, YodaoBot, Charlotte, DiscoBot, EnaBot, Gaisbot, Kalooga, ScoutJet, TinEye, Twiceler, GSiteCrawler, HTTrack, Wget
(Remember SMF detects most Google/Yahoo/MSN bots by default)

It appears that most sites offering spider/bot lists have tonnes of inactive ones. I'm putting together my own lists by detecting them in the wild on my own sites. Plus any that get reported to me (after I've checked them out).

So if there are any ACTIVE ones I'm missing? Please let me know in the support topic.

Installation
Any previous versions of the mod does NOT need to be uninstalled.
This mod adds a row for each spider in the database only. It will ignore adding ones which already exist.
No theme edits required. No language strings to translate or editing. It adds database rows only.

Install and your done.
Note: Click uninstall to remove the mod from your mod list. But it won't remove the spiders from your database. You'll need to remove each one from your SMF Search Engines/Spider section.

For manual installation, just upload AddMoreSpiders.php to your SMF directory and run it in your browser (then delete the file).

Useful Links
SMF Package Parser (No you won't need this for this mod)
Manual Installation Of Mods
How Do I Modify Files?

Donate
Has this modification helped you? Support the developer by Donating

Support
Please use the modification thread for support with this modification.
(Please don't ask me to do the edits for you)

Changelog
1.0 - 9th April 2008
o Initial release.
1.1 - 4th May 2008
o Fixed Alexa/InternetArchive
o 25 More Spiders added (which have been detected on my forum in the past month)
1.2 - 20th June 2008
o Added a handful of new spiders which have been detected in the past month
« Last Edit: June 20, 2008, 07:00:50 PM by karlbenson »
Jeremy D — Site Team / SMF Developer
Support the SMF Support team!
Profiles:
GitHub

Offline SgtMic

  • Sophist Member
  • *****
  • Posts: 1,083
  • Gender: Male
    • -{FAST}- Squad
Re: More Spiders
« Reply #1 on: April 10, 2008, 11:09:19 PM »
Got it.
There is nothing more deadly than a US Marine and his rifle.
A close second is a US Marine and his K-BAR.
2/5 Fox Co.  (BlackHearts)
FAST Co. 5th Plt. (FIDO)

Offline Joomlamz

  • SMF Hero
  • ******
  • Posts: 2,639
  • Gender: Male
  • Suporte SmfPT
    • apllicmz on Facebook
    • apllicmz on GitHub
    • https://www.linkedin.com/in/aplic on LinkedIn
    • @aplichosting on Twitter
    • Webdesign
Re: More Spiders
« Reply #2 on: April 11, 2008, 03:14:00 PM »
great Mod



Offline shadow82x

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 11,280
  • Gender: Male
    • cmblaber on LinkedIn
Re: More Spiders
« Reply #3 on: April 12, 2008, 08:26:51 PM »
Works fine on my board. :P Nice work.
Colin B
Former Spammer, Customize, & Support Team Member

Offline D.S

  • Jr. Member
  • **
  • Posts: 133
    • ILLIMITED LCC WEBHOSTING
Re: More Spiders
« Reply #4 on: April 13, 2008, 06:02:49 AM »
I like your mod but I'm using  1.1.4

anyway , thank u karlbenson for effort

D.S

Offline coldy316

  • Jr. Member
  • **
  • Posts: 164
  • Gender: Male
  • G'day mate, how ya doin
Re: More Spiders
« Reply #5 on: April 13, 2008, 06:50:13 AM »
will this mod work on the 1.1.4 boards? seems okay just disappointed that there
isnt one for the 1.1.4
Would you believe it that i live in gods country
Australia

Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,630
  • Gender: Male
    • Lifeproof
Re: More Spiders
« Reply #6 on: April 13, 2008, 10:02:29 AM »
No.
This mod only adds rows to the default 2.0 feature of search engine spider tracking.

However I use 1.1.4 on my own forum, and what I've done is used the spiders from this mod in the Googlebot & Spiders mod (replacing the existing spiders in that mod which may be a little out of date/doesn't detect the newer ones).
let me known if you want me to attach that as an attachment with some manual directions.

Offline edi67

  • SMF Hero
  • ******
  • Posts: 1,532
  • Gender: Male
  • Italian Supporter
    • CrazyZone
Re: More Spiders
« Reply #7 on: April 13, 2008, 10:47:36 AM »
excellent as always
CrazyZone - My SMF Forum


From the difficult the hardening of the man you can see

Offline D.S

  • Jr. Member
  • **
  • Posts: 133
    • ILLIMITED LCC WEBHOSTING
Re: More Spiders
« Reply #8 on: April 13, 2008, 04:18:05 PM »

let me known if you want me to attach that as an attachment with some manual directions.
Yes, plz karlbenson if u can that , I wish to help us

D.S

Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,630
  • Gender: Male
    • Lifeproof
Re: More Spiders
« Reply #9 on: April 13, 2008, 04:26:14 PM »
Ok here it is attached with manual instructions.

There are a handful in here, not in the more spiders bot yet. I've been waiting to detect them on my forum.

Edit: Updated attachment 20th May 2008 - v1.2
« Last Edit: June 29, 2008, 05:52:06 PM by karlbenson »

Offline D.S

  • Jr. Member
  • **
  • Posts: 133
    • ILLIMITED LCC WEBHOSTING
Re: More Spiders
« Reply #10 on: April 13, 2008, 04:38:15 PM »
thank u dude for help me ..

D.S


Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,630
  • Gender: Male
    • Lifeproof
Re: More Spiders
« Reply #11 on: April 13, 2008, 04:40:04 PM »
Let me know if there are any problems.

I've been using it on my forum for the past week without issue.
And have noticed all kinds of different crawlers/spiders/bots on my forum.

I've now seen about 90%+ of all these bots on my own forum.  Including some more.  Like a Russian SE spider.  Not sure why as my forum is entirely in English.
« Last Edit: April 13, 2008, 04:46:16 PM by karlbenson »

Offline sorosh1

  • Semi-Newbie
  • *
  • Posts: 76
Re: More Spiders
« Reply #12 on: April 14, 2008, 09:35:46 AM »
karlbenson do you have the same mod for SMF 1.1.4? Awesome mod by the way!;)

Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,630
  • Gender: Male
    • Lifeproof
Re: More Spiders
« Reply #13 on: April 14, 2008, 10:41:29 AM »
As stated above. Its 2.x only because ALL this mod does is add more rows of spiders to the default 2.0 feature of search engine spider tracking.

There is no such feature in 1.1.x.

However I use 1.1.4 on my own forum, and what I've done is used the spiders from this mod in the Googlebot & Spiders mod (replacing the existing spiders in that mod which may be a little out of date/doesn't detect the newer ones).
That is what I've attached further above.

Offline sorosh1

  • Semi-Newbie
  • *
  • Posts: 76
Re: More Spiders
« Reply #14 on: April 14, 2008, 04:29:41 PM »
ok thanks i will do the same then:)

Offline christicehurst

  • Full Member
  • ***
  • Posts: 581
  • Gender: Male
  • Brisbane Lions United Admin
    • CJT Online
Re: More Spiders
« Reply #15 on: April 15, 2008, 04:06:05 AM »
Another great mod!
www.brisbanelionsunited.com - A forum for everyone!

Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,630
  • Gender: Male
    • Lifeproof
Re: More Spiders
« Reply #16 on: April 15, 2008, 08:55:29 AM »
Thanks.

Offline RoarinRow

  • Full Member
  • ***
  • Posts: 662
  • Gender: Male
  • Ride my AV!
    • Avalanche Style
Re: More Spiders
« Reply #17 on: April 16, 2008, 12:50:49 AM »
Very cool, thank you!   8)

SMF 2.0
TP 1.0 RC1.1
Wordpress 3.1.3

Offline coldy316

  • Jr. Member
  • **
  • Posts: 164
  • Gender: Male
  • G'day mate, how ya doin
Re: More Spiders
« Reply #18 on: April 18, 2008, 07:11:46 AM »
whats the chances of you doing a version for 1.1.4 spiderbot updates but just the spiders only
something like an addon  to the 1.1.4 version..
Would you believe it that i live in gods country
Australia

Offline karlbenson

  • SMF Friend
  • SMF Super Hero
  • *
  • Posts: 15,630
  • Gender: Male
    • Lifeproof
Re: More Spiders
« Reply #19 on: April 18, 2008, 10:43:13 AM »
I probably won't do it as a add-on mod to the googlebot mod.

But I'll continue to update the attached list (as long as my own forum remains at 1.1.4)
Its a single edit taking less than a minute to do.

http://www.simplemachines.org/community/index.php?topic=233636.msg1507179#msg1507179