Better plugin system / file management (with more details below)

[Arantor]

I'd love to see future versions of SMF have a stronger plugin system. Putting aside all the discussions related to hooks, there are two things I'd love to see.

1. A more standardised method of plugins being able to have all their files in one place, rather plugin authors needing to do things over and over in terms of file management.

2. Plugins being able to deploy to the server without having to mess about with CHOWN or CHMOD at all, so that you can just deploy a new installation and drop plugins in without having to mess around changing file permissions. This generally relies on point 1, though.


Thoughts?

[SoLoGHoST]

I'd love to see future versions of SMF have a stronger plugin system. Putting aside all the discussions related to hooks, there are two things I'd love to see.

1. A more standardised method of plugins being able to have all their files in one place, rather plugin authors needing to do things over and over in terms of file management.

2. Plugins being able to deploy to the server without having to mess about with CHOWN or CHMOD at all, so that you can just deploy a new installation and drop plugins in without having to mess around changing file permissions. This generally relies on point 1, though.


Thoughts?

Ok, I'm assuming that, by Plugins, you are referring to SMF Modifications, since that is kinda where Mods are moving towards these days, I suppose, with hooks and all.

I'm definitely in agreement here!  If Plugins could just occupy 1 single space (directory path) within the SMF Root, without having to spread itself out everywhere, just to be collected back up and disposed of upon uninstallation of the mod, that would be AWESOME!

My thoughts on this are...

1. It would definitely be a step forward in the right direction for what SMF's long-term plans are with smCore!  Would kinda be an smCore inside of SMF itself, but acting as an smCore for Plugins/Mods.

2. The way SMF 2.1 is being coded, I can see a step towards this approach, but not exactly getting there yet, as there is still hooks.  I think that hooks might have to be rethought for this approach!  I mean, SMF could still have hooks, but instead of a ton of hooks throughout the software, it should be 1 HUGE HOOK, with a ton of sub-hooks, IMHO to get this working right.  The 1 HUGE HOOK would have to be flexible enough to do just about anything!  Or maybe it can be done with just 1 Hook and no sub-hooks?

3. This would require a lot of changing for sure with how SMF deals with the index.php file, ./Sources/Load.php will need a great deal of thought involved in this.  But SMF would have to be much more flexible with how it deals with security in order to be able to do this successfully, IMHO.

Anyways, just some of my initial thoughts...

Mostly, a lot of restructuring would be going on here!  Starting with the index.php file ofcourse, and working it's way down the ladder.

[Arantor]

The 1 HUGE HOOK would have to be flexible enough to do just about anything!

Dear god please for the love of all that is holy NO.

how it deals with security

Um, why?

[SoLoGHoST]

lol, those are just my initial thoughts.  Ofcourse, when you take it to the drawing board, which I haven't done, could be more difficult and problematic...

Just trying to get the brain-juice flowing here.

[Arantor]

Oh, I'm coming at it from a completely different perspective, I don't have to speculate on how any of this might work - I have pretty much all of this working already but it's not under a licence I can share back to SMF's code (not that any of it would work without being rewritten anyway). The biggest hurdle is accepting the notion that mods would never be able to edit SMF files directly and once you get past that, you start figuring out all kinds of ways to clean up the core to make it more extensible by design.

So it's more a case of suggesting what I think SMF needs to do to catch up with what's going on that's already working...

[SoLoGHoST]

Thought you were asking for suggestions on what SMF might do to start a Plugin System for Modifications, that instead of Modifying SMF files, has their own files that modify SMF instead, by either, adding, editing, and/or removing features from it, however, in it's own files, not SMF Files.  In this case, the next logical thing for SMF to do is to make a Hooks change.  Or maybe continue to expand on hooks as they are doing currently.

Not sure what else they could be doing that wouldn't be a completely dramatic change, that would change just about everything and probably wouldn't be backwards-compatible anyways.  Could be the fate of SMF?  That is, that they have reached a Dead-End, where as, something needs to change completely and not partially.

Just need more Arantors out there to make this happen I suppose...

[Arantor]

Thought you were asking for suggestions on what SMF might do to start a Plugin System for Modifications, that instead of Modifying SMF files, has their own files that modify SMF instead, by either, adding, editing, and/or removing features from it, however, in it's own files, not SMF Files.  In this case, the next logical thing for SMF to do is to make a Hooks change.  Or maybe continue to expand on hooks as they are doing currently.

I am. I'm also establishing the fact that a lot of this has already been done in an SMF-like base, so that it isn't a pipe dream. It's the result of going away, trying it and coming back to share the results.

Not sure what else they could be doing that wouldn't be a completely dramatic change, that would change just about everything and probably wouldn't be backwards-compatible anyways. 

Why shouldn't future versions be dramatic changes? SMF 2 -> SMF 1 isn't backwards compatible, why should SMF ?? -> SMF 2(.1) be backwards compatible?

That is, that they have reached a Dead-End, where as, something needs to change completely and not partially.

There are some serious issues to fix architecturally to make this work. But it IS doable.

Just need more Arantors out there to make this happen I suppose...

More contributors who are aware of all that has transpired, all that is transpiring and where it is likely to go from there. These are in all too short supply.

[SoLoGHoST]

More contributors who are aware of all that has transpired, all that is transpiring and where it is likely to go from there. These are in all too short supply.

Yeah, well, it's a free project!  You can't earn a living developing for SMF AFAIK.  Don't get me wrong, those of us who love coding will do it for the pure pleasure of it all, but, honestly, at the end of the day, they are left with their nagging wives, telling them that they are spending too much time on the computer and not with them...  or they are wasting their time coding for a software without getting anything from it, or their parents are limiting their computer time each day...  Life always gets in the way, and family is even more in the way of us wanting to contribute...  But in the end, what's motivating us to keep reading hundreds of posts per day at the SMF website?

Anyways, just a thought...  Sorry for going off-topic here

Also, who really wants to genuinely share the knowledge?  Who, out there, has no ego, to the point where they can help someone understand what is going on without criticizing them the entire way down that road?

[MrPhil]

I was wondering if someone could summarize the differences between the current SMF mod system, hooks, and plug-ins (or point to a place where this has already been discussed). I understand the current mods, where the package manager actually edits the PHP source files. While the result is in-line code running at max speed, as we all know, it's prone to all sorts of problems (incomplete install/removal, collisions between mods, version changes, etc.).

I take it that hooks and plug-ins remove the need to modify base code files, but...

• Does this require that code be added to check for hooked routine/plug-in wherever you want to make a change? What if I want to add something at a point where there is no hook/plug?
• What is the performance impact of this? Will SMF have to check for the existence of a module at every hook/plug point? Is this a runtime late binding (elegant, but slow, compared to compile-time binding)?

[Kindred]

well, if you look at 2.1 and even more so at Wedge...   they have added hooks to all the points which they can conceive of any plugin needing to attach itself. 2.1 is still missing hooks from some key points and still allows direct file edits. Wedge seems to be pretty comprehensive, but has completely removed the ability to make direct edits.

In the case of wedge, if you come across a place where the hook doesn't exist, then you can't do anything about it.
I think this was a step too far in the other direction. The package manager's ability to perform direct edits to core files has always been one of SMF's greatest strengths over the other systems, IMO. Yes, it can lead to conflicting mod edits... but as more people use hooks, the edits will become less and will only be used in cases where the hooks are missing (or the mod author is lazy)

One thing which is a benefit of a "full plug-in" system instead of direct edits, or even linked hooks, is that - if a mod screws the pooch, you can remove it by just deleting the mod files form FTP or the file manager

I don't know about the performance impact....


There are two schools of thought on the installation format that Arantor mentions.
1- I like SMF's consolidation of the source files into one direcotry and template files into another directory (with images as a subdir). I find this logical and straight forward...
2- sometimes, it mitgh be easier, especially for newbies is a mod installed all of its files, plug-in style into a single mod-sub-directory like WordPress and Joomla do...   each mod has its own subdirectory and structure independent of SMF core and/or any other mod.

IMO, both variations have something going for them...

[Arantor]

1- I like SMF's consolidation of the source files into one direcotry and template files into another directory (with images as a subdir). I find this logical and straight forward...

It is... until mods need to add their own images. There is a method by which they can drop their files only into the default theme where they can explicitly call them from there. But most mods don't know about it, some can't even work that way (e.g. bbcode editor images) which means explicitly having to drop images in every theme's folder. Yay.

I personally like the notion of the core being cleanly separated and then plugins having all their files in a set place. Some of my plugins have a couple of files, some have hundreds - the former doesn't bother with an extra folder, the latter has subfolders for that purpose.

2- sometimes, it mitgh be easier, especially for newbies is a mod installed all of its files, plug-in style into a single mod-sub-directory like WordPress and Joomla do...   each mod has its own subdirectory and structure independent of SMF core and/or any other mod

Trust me, it's a ton easier.

Classic case example: an extension needs to add an action - but there's a bug. What's going to happen? SMF's going to be in the situation whereby you have to manually edit files to cover. I just rename the folder temporarily (not even remove it) and it's usable again. Tell me that's not an improvement.

How many support issues here are due to parse errors of mods blindly find/replacing code? Seriously, how many of them? Parse errors here turn up because of two things: people manually editing code and not knowing what they're doing and people having to cope with botched mod installs.

To me, having slightly reduced flexibility (mod author can always tell people to manually change code, that happens elsewhere, e.g. in XenForo installations where people have to do certain template edits), far far outweighs the support headaches that come with it.


There are a lot of other side effects to this.

1. It means you reasonably reliably can say that if files have been changed, something is likely wrong. (Yes, I even changed the caching code touch()ing Load.php to indicate a cache flush) It is then possible to track this and write a task to automatically check it.

2. Updates need not be distributed as patch files but for the bulk of users can be distributed as complete new files to be updated.

Does this require that code be added to check for hooked routine/plug-in wherever you want to make a change? What if I want to add something at a point where there is no hook/plug?

Yes, it does. However, an isset() to a hash-key array is very fast these days.

There are places where there are currently not hooks. 2.1 has added more than we have, but I'd argue ours are more powerful The other thing is that it's very easy for us to add hooks.

What is the performance impact of this? Will SMF have to check for the existence of a module at every hook/plug point? Is this a runtime late binding (elegant, but slow, compared to compile-time binding)?

It is a late run time binding, however it is not as simple as that. There are several things we have deliberately introduced around this.

Firstly, actions. In SMF, to register a new action, you have to load a file to be called by a hook which is running a 1 line function. In Wedge, the plugin-info.xml file (analogous to package-info.xml) indicates that the plugin declares one or more actions, and these are added to a list held centrally, activated when a plugin is activated at start-up. There is, of course, a performance hit to this but it is not actually that significant (seeing how there are no DB queries involved in this, nor any extra I/O that wouldn't be carried out anyway)

Secondly, only enabled plugins get their hooks attached. This is done at start-up, so a plugin that is not activated cannot get its hooks enabled. It's fractionally slower than relying on it having been done before 'once and for all' by the package manager, but from a usability perspective the few extra cycles involved are worth it.

Thirdly, when enabling a plugin, the hooks a plugin uses are validated. If a plugin wants to use a hook that doesn't exist, it isn't going to happen, including if a plugin depends on another plugin (since plugins can also indicate they provide certain hooks)

And now all of this stuff can happen without a single chmod in sight. Not a single 777 gets invoked for plugins. Users don't have to remember to put permissions back because they never have to change them in the first place, nor invoke chown.

[SoLoGHoST]

Wow, GREAT STUFF Arantor!!  Definitely need more minds like you!

Just something I want to add here...

In 2.1, I've noticed the addition of the loadCSSFile() and loadJavascriptFile() functions.  I just wanted to say that I'm so happy you guys added this in!  Beats using loadTemplate(false, {relative path from ./Themes/{theme} for CSS file}).  It's definitely a step forward, and the fact that there is also a $params array where we can set $params['local'] to empty, we can load up a CSS file from anywhere on the server, thus having css files into our own folders and images linked via css to our own sub-folders within our own folders.

I just wanted to say that this is a very HUGE step in the right direction of exactly with what Arantor is talking about!  Something that I plan on taking advantage of, in which all mod authors should learn to use also, IMO.

So, this opportunity does exist, it's just not that apparent yet.  But I don't see a way to do this easily with Template files, in SMF, yet, however...

****  EDIT ****
Perhaps, SMF should rethink their approach to the way themes work.  Instead of loading template files from a predefined Directory structure...  Just a thought.

[Arantor]

Wow, GREAT STUFF Arantor!!  Definitely need more minds like you!

It isn't the minds that are the problem. It's the attitudes that come with it. It is the same attitudes that discourage me from committing any of what I've done back to SMF, licenses aside. It is why, despite being asked, I will not join the SMF team as a developer.

In 2.1, I've noticed the addition of the loadCSSFile() and loadJavascriptFile() functions.  I just wanted to say that I'm so happy you guys added this in!  Beats using loadTemplate(false, {relative path from ./Themes/{theme} for CSS file})

It also solves a bug too if the file happened to be in a subfolder because the loadTemplate injection would also add an id to the <link> element that would be invalid.

It's definitely a step forward, and the fact that there is also a $params array where we can set $params['local'] to empty, we can load up a CSS file from anywhere on the server, thus having css files into our own folders and images linked via css to our own sub-folders within our own folders.

Eh, some of us were doing that without any problems in 2.0.

I just wanted to say that this is a very HUGE step in the right direction of exactly with what Arantor is talking about!

In the grand scheme of things that could (and IMNHSO should) be done, these are tiny steps. Important ones, but realistically tiny. When you get to the point of automatically deploying minification of the files, it suddenly becomes more of a big step, especially if you go to the trouble of saving them on the server already gzipped (and thus saving CPU cycles not to gzip them every request)

But again, none of this is theoretical. It's all been done, in some cases for over a year in a real environment.

[SoLoGHoST]

Yeah, I noticed the id attribute being set to the path in loadTemplate() while back.  Had to actually do an edit to the loadTemplate function to fix this within the function itself, for Dream Portal.  Yeah, suppose it is a tiny step, but huge in significance.

Well, if it's all been done, than why are we talking about it?  lol... j/k

[Arantor]

Well, if it's all been done, than why are we talking about it?  lol...

A cynical mind might suggest I were trolling SMF.

A more enlightened mind would take it that others and I went off adventuring, trying all these things out and reporting back on what we found works well and what doesn't work so well, sharing the experiences of the journey.

For example, more of the steps required in making it more modular require breaking templates up so they aren't massive leviathans of code but each module of content is a single unit and that $context['sub_template'] as a minimum needs to be an array rather than a single function surrounded by one or more layers. Though you can get infinitely more interesting than that too.

[SoLoGHoST]

Well, if it's all been done, than why are we talking about it?  lol...

...others and I went off adventuring, trying all these things out and reporting back on what we found works well and what doesn't work so well, sharing the experiences of the journey.

Sounds great!  Good Stuff!  I always enjoy reading your posts because they usually involve a great deal of thought behind them that only comes with experience!

For example, more of the steps required in making it more modular require breaking templates up so they aren't massive leviathans of code but each module of content is a single unit and that $context['sub_template'] as a minimum needs to be an array rather than a single function surrounded by one or more layers. Though you can get infinitely more interesting than that too.

hmmm, definitely interesting.