[storm] some questions about Storm (from the perspective of Grok)

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martijn Faassen wrote:

> Stuart Bishop wrote:
> [snip]
>> An application that doen't need to scale to vast amounts of users or massive
>> quantities of data is a Noddy application by my definition.
> 
> I don't know what "Noddy" means, so you can define it as anything you 
> like. :) I take it you don't want Storm to be a fit for Noddy applications?
> 
>> That doesn't
>> mean they are not useful (and this is why I don't use the real world term
>> here). You don't care about the scalability issues with generated schemas
>> because scalability isn't an issue. You don't care about integrating with
>> existing data sources because there are none. You don't care about
>> maintainability or upgrades because data model required by the application
>> is so simple it doesn't matter. 
> 
> Quibble: I can see a schema generation system care about maintainability 
> and upgrades, such as apparently RoR is able to do. Just not in the same 
> way as you'd do with a Vast Application.

We aren't even talking about "vast":  tables with more than a few
thousand rows apiece are enough to trigger the issues we've described.
Furthermore, getting the RDBMS schema right for the application logic
(performance aside) often means taking advantage of various
less-specifcied, or non-standard, features of the RDBMS in question:
triggers, custom index types, tablespaces, stored procedures, etc. In
such cases, a "least common denominator" schema generators can't
possibly succesed.

>> Just don't try to retrofit scalability later.
> 
> Actually, it may still be worthwhile to do this in some cases. Time to 
> market matters, and apart from that, many developers are encouraged when 
> they get something running quickly, even if it's a bit dirty. Such an 
> application may still grow into something scalable later on. The 
> transition will in many cases be painful, but it might nonetheless be 
> worthwhile to go this route for a project.

Somebody might make that tradeoff for their application.  I hope Storm
declines to trade that off in advance (although I'm not opposed to an
add-on which supplies the need).

> [snip expressing advanced features of relational database schemas are 
> hard to express in a Python-driven schema]
> 
> I am not debating that Python-driven schema are what everybody should 
> use, or that they should in all circumstances. I accept the arguments in 
> favor of hand-written schemas. I don't think that schema generation is 
> the unadulterated uselessness/evil that I am picking up from the vibe here.

Try working with one of the schema-generating frameworks for a bit, on
an application where you actually care about the outcome (as opposed to
the tutorial / demo applications which they all outline).  Pick
something like a timesheet, or a project management / work breakdown, or
even (for extra illumination) something which would be trivial to do in
a ZODB-based application (say arbitrarily recurseive contaiers).

FWIW, I've been doing relational-based apps for a *long* time (since
1990), and have yet to see the promise of generated schemas fulfilled:
in each project (five, I think, altogether) where we started from a
generated schema, we had do scrap not only the schema, but lots of the
application code, somewhere around halfway to two-thirds through the
project.  In each case, hindsight indicated we would have been better
off starting from a hand-rolled schema.

>>> Do you think that SQL *queries* created by tools such as Storm are
>>> also so inferior to hand-written queries that they are only suitable
>>> for toy applications? If not, why is query generation already there
>>> while schema generation is not?
>> The SQL queries generated by some ORMs are vastly inferior. Storm and other
>> ORMs targetting non-Noddy applications are designed to allow formulation of
>> complex queries with enough control to have them perform efficiently. 
> 
> I find the differences here fascinating. What makes generating queries, 
> with all their complexities, all right and useful but generating schemas 
> something that shouldn't be done? Is generating queries correctly that 
> much easier?

The semantics of the 'SELECT' statement are restricted / well-understood
 / commoditized, compared to the rest of the SQL language, largely due
to the prevalence of third-party reporting tools.  It is still
commonplace to have the DBA forbid the use of such tools on the "live"
data, or subsets of it, because the queries they generate perform so
poorly that everyone else (not just the person running the query) suffers.

>> And if
>> the Python syntax turns out to not offer fine grained enough control or
>> access to that proprietary feature you need to access, then you can fall
>> back to using SQL fragments but still have the results retrieved into your
>> object model. 
> 
> Yes, same argument applies to schema generation too, though.
> 
> [snip]
> 
> I think I'm dealing with a cultural issue here: it's an important part 
> of the Storm approach to want to write these schemas in SQL. There are 
> of course excellent reasons to do this in many circumstances. I do have 
> the feeling it's like arguing about threading to Twisted developers - it 
> might be instructive but it is not very useful. :)

One of the reaons I jumped on Zope in the first place was to get away
from the "always fighting with the DBA" syndrome which plagues large
relational projects.  That doesn't mean I think the DBAs were *wrong*,
by the way;  their interests and concerns just didn't align with those
of the application developers.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFH3VR3+gerLs4ltQ4RAsZIAKC7HlUorKulTmcLUM3ogt6sD6zcnACfQpcf
6ixAyYmfTGchftNC95lMgXU=
=6+k7
-----END PGP SIGNATURE-----