Software: simplicity and evolution (was: Ableton Live - has it lost its mojo?)

[Peter Korsten]

Kim Johnsson schreef op 8-6-2015 om 22:45:

> I agree on Simplicity. However, I am a big fan of starting from scratch. I have a pet project I’ve been working on for years, and every now and then I come to realise that I’ve been a moron, and that I have so much better ideas than I had when I dreamt up the current architecture, and that my software would definitely benefit from a major refactoring/rewrite. I absolutely love the moment when you start from a clean slate and get your first few classes working together with way less code than last time :-)
>
> Each rewrite/generation has so far been much simpler than the previous one, as in simpler to understand, develop further and use. Much less bugs as well. Of course, some “practical" code is reusable; usually I rewrite architecture level stuff, because that’s the most interesting part, and has the biggest impact on everything. Every now and then a piece of the architecture really falls into place, and I’m happy for a couple of years. I’m not strictly speaking a professional programmer, but I am a software guy at heart. I make Data Warehouses and DW-based reporting, which is reasonably far from being interesting, so I kind of need my pet project to keep me going (a Conceptual Model based automated DW and ETL-generating tool / application generator, so it’s not completely impractical). Obviously the goal is to produce something that doesn’t need to be re-wrtitten, but I don’t mind it if the end result benefits from it. I’m also not considering cost of development here, which obviously is a factor.

So... after all those years and all those rewrites, have you actually 
finished anything?

The question ought to be: is this good enough? Because it will never be 
perfect. If you strive for perfection, you're practically guaranteed to 
never achieve it.

An example at work is the Bundle Management System (BMS). You send an 
SMS, we charge your prepaid account or your bank card, and you get x 
free minutes, y free SMS and/or z free MBs of data, based on the 
contents of the SMS. This was written by a guy who has since moved on to 
greener pastures (thank God for that) and made working by someone else.

Recently, we decided to largely ditch our home-grown CRM system and 
replace it by Siebel. The friendly folks of Accenture (their Greek 
subsidiary) wrote all the extra bits, and with what they charged us we 
must have kept a significant number of Greeks from torching banks and 
German institutions.

Part of this was BMS. Some part of it is now done by Siebel/Fusion 
Middleware, and part of it by us. What we do is figuring out when 
bundles need to be repurchased, or expired, or given as a freebee, and 
we send messages regarding these events.

To make things a bit more interesting, I decided to use Hibernate 
instead of building all the SQL code myself. It also uses fancy stuff 
like thread pools. All in all, the whole project including BMS was an 
absolute disaster. There were some bugs in BMS, not many, and there were 
tons of bugs in what our Greek friends had written, or rather, what the 
Indian friends of our Greek friends had written. (They're in the 
building next to where our own Indian friends are.)

One mayor point of sorrow was getting information about the bundles from 
Siebel. They had written services for them, but these were 
excruciatingly slow. The result was an enormous backlog. We also 
couldn't send too many requests at one go to middleware, because it 
would choke. Given that I take pride in writing efficient code, you can 
imagine how many requests per second my system would give them.

Given the massive problems we had with the web services, I decided to 
change those to database views and move practically all database stuff 
from Hibernate (Java) to Oracle PL/SQL. Since sending SMS messages for 
us is simply inserting them into a database table (another application 
that I've written, years ago), the system became again simpler.

Accenture made some noise about not using their web services, and one of 
their architects foolishly used the phrase "spaghetti code", after which 
I patiently explained that a database view is as much as an abstraction 
as a web service is; the difference being that Accenture wrote the web 
service, and we wrote the database view. In an e-mail, obviously, with 
enough people in copy, and the implicit message that their stuff was 
crap, and mine wasn't.

Still, even though most of the work is now done in the database, the 
core of the program remains unchanged. All changes were done in an 
evolutionary manner. I also moved the whole thing to Maven (it was Ant 
based before), and last week I modified it again so that it uses one big 
Maven project with modules, instead of three separate projects.

Could it be improved? Oh, definitely. But not by starting over: the core 
of the design is sound, and simple. I can easily put it in a diagram on 
an A4 paper with a couple of boxes and arrows. It may not be perfect - 
but it's good enough.

- Peter