Or sprint and shatter?

For a while now I’ve been working on a new version of Freegle Direct, which is the way that most people use to Freegle (for valid reasons of history and democracy, there are others – Yahoo Groups, Norfolk Freegle, trash nothing…).  It went live today.

In a commercial environment, you’d take this nice and slowly – do your development, run your regressible unit tests and code coverage, do friendly user testing, test on all the standard hardware, operating systems, and browsers.  Then do a phased rollout to a few groups.  Leave that for a week or two, use Decibel Insight to do usability testing, tweak and repeat.  All nice and calm, with time for lattes and cake and cats and those fells which were beckoning me today.

Ha.  Let’s pause for a minute to dream about that.

Back to reality.  Freegle is a volunteer project, late evenings and weekends and half-snatched fragments of insomniac time.  We just don’t have the time or geeky resources to be able to do it in the same way as a commercial environment.

Nor, frankly, do we need to.  If people can’t reuse items, then who dies?  It’s not nice to annoy your users, it’s really not.  But, y’know, move fast and break stuff.  “Fit for purpose” doesn’t always mean perfect.

So with Facebook’s API changes breathing down our necks, we just had to take a deep breath, stick aforementioned neck out, and go for it.

So it’s been a high-wire-act kind of day.  Cunningly defer some function.  Be ruthless about what we need and what we don’t (no bad thing anyway).  Get other kind volunteers to test it, but accept that the testing will have gaps.

Then do the switchover, monitor everything, spot the howling bugs and get them fixed, wait for the first raft of problems to come in from our volunteers and users, then fix them too.  Development in real time.

It’s settling down now.  I might actually get some sleep tonight.

Where are the geeks?

Here at Freegle Towers we have a large team of highly paid IT professionals, in a rather swish office, sitting on inflatable furniture, surrounded by solid platinum cappuccino makers, where the main obstacle to getting things done is constantly being interrupted by someone asking if you’d like a free manicure.  No, hang on, that’s not right…

Let’s try again.  Freegle runs almost entirely on volunteers – we have a mighty 1.8 million members and we save about £1.3 million pounds worth of goods from going to landfill/incinerators a year for a budget of £10K, which is a Return On Investment (ROI) of 13,000%.  

That is amazing.  

But it’s a struggle, and there are some areas where it’s really tough to find volunteers with appropriate skills.  IT is one of them.  The people with the skills we need are usually in full time jobs, and if they have energy left when they get home and have changed the nappy on the hamster and fed the children to the dog, then they’d like to spend it coaching a football team or helping young adults knit their own vegan curry.  

And we can’t afford to pay for much, because, well, giving stuff away for free is hard to monetise, and we’ve not found any angels who see the value of what we do, or they assume that it must all be fine.

So that means that the number of people who can have a conversation like “Yes, it’s great that we’ve moved to a Percona cluster, and nginx continues to rock, but we really need to rewrite the UI to be responsive for mobile, probably in a backbone architecture, using bootstrap or similar until we can find a CSS3 expert, and with a checkout model rather than upfront sign-up” can be counted on the fingers of one han…no, wait, I can’t do that, I’m typing.

That’s a bit unfair.  There are a couple of us.  And we sometimes luck out – Bytemark, for example, have provided us some splendid hosting which has really helped with the kind of load spikes we get from TV coverage:

 

 

But still.  So please, have a look behind your sofa and see if there’s anyone with geekspertise twiddling their thumbs wondering what to do.  Because we could really, really use them.  Mail us at geeks@ilovefreegle.org.

TV coverage – there’s no gift horse into the mouth of which a geek cannot look

Freegle got some TV coverage on Kirstie Allsop’s Fill Your House For Free last Tuesday, and with a bit of luck we’ll get some more over the next few weeks.

This is brilliant, fantastic, wonderful stuff.  It’s also horrific, stressful, awful and expensive.  Why?

Normally, while some people are pottering around on the Freegle website, others are stroking their cat.  After a while, the cat wanders off to decimate the local rodent population, so those people start posting WANTEDs on Freegle for cat toys in the deluded belief this will make the cat stay inside.  By this time the people who were on the website earlier have posted their OFFERs for the cat toys they got a while ago which just got ignored, and have now started hoovering.  Elsewhere, some people have decided to cut out the middlemog and just hoover the cat.

Anyway, you get the idea.  It ebbs and flows.

Then you get some TV coverage.  Don’t get me wrong, it’s great, and someone has worked hard behind the scenes to make it happen.  But what it means is that all over the country, people who see the name “Freegle” on screen decide to check out your website.

All.  At.  Once.

Suddenly, your website has 50, or 100 times its normal traffic.  And it’s important that it handles it because, well, this is people’s first impression, and if doesn’t work, you’ve probably lost them.  And if it takes five minutes to load a page, by that time the cat has come back in with some entrails which need scooping up.

So what do you do?

Well, if you have lots of money, you just have a few hundred times more servers than you need.  Right, well, you just lost us at the whole “having lots of money” thing.

If you know the publicity is coming, you can usually add some more server capacity for a short period, if your hosting company allows that.  If they don’t, change hosts.

For the geekier readers, here’s some stuff you can do:

• Use NGINX.  Seriously.
• Use a PHP-FPM pool with a static size – you don’t want lots of processes spawning which chew up your memory and make you start swapping.
• Make sure you’ve increased OS limits on file handles – the defaults are too low.
• Don’t have slow MySQL logging turned on.  That can cause a death spiral.
• Get your SQL queries right.  High load is a good time to monitor the system to spot missing indices – we found one in the email confirmation step.

Here are some other things we’ve learned:

• It takes people 3 seconds to google.  Watch the screen, watch Google Analytics in real time, and you’ll see it.
• When there are three different words on the screen, people mostly don’t react.  When there’s just one, boy do they leap on it.
• Come the commercial break, the people who don’t google immediately start checking out the stuff they heard about in the previous segment.
• People really do watch the +1 channels.  Who knew?

 

Behind the scenes – server migration

 Warning: this blog entry is a bit geeky. If you’re the kind of person who puts on a polite smile and backs away carefully when people at parties start talking about Linux, then…who am I kidding, who talks about Linux at parties? Oh, that guy over there standing awkwardly on his own in the corner? I see.

 Actually, at Freegle we’d like to reclaim the word ‘geek’. Freegle helps people give away stuff they don’t need to people who do – and this just wouldn’t happen without some IT gubbins in the background, and some committed volunteer geeks trying to keep it going. So geekiness is good, right? Right? Please say yes.

 Anyway, we’ve just migrated our hosting. For the last few years we’ve been with VPS.NET, who have been lovely, and provided us with discounted hosting. Everyone likes a good deal, but Freegle is run on a shoestring – actually we’re saving to buy a shoestring – so this was particularly valuable for us as we found our feet. But we’ve gradually got to the point where we needed to take the next step up. Since the philosophy of Freegle is that you don’t know whether you’ll get something for free until you ask, we asked around, and Bytemark offered us some room ontheir BigV platform. Because we’re a good cause, they’re hosting us for free, and when we were discussing it they haggled us up from what we thought we could ask for without them spluttering to something beefier than we’d hoped. Lovely folk.

So obviously, now I have to big up BigV a bit. But in fact, I don’t need much encouragement. It’ll be a while until we need to scale up to a VM with 180GB of RAM, but it’s still rather nice if you’re looking at higher-end hosting, and you’re prepared to live with an admin side that’s a bit more focussed on CLI than a slick GUI. We particularly like the availability of SSD disks on a VM – that gives your database performance a welcome boost.

Talking of databases, as part of this migration, we switched over to using a Percona XtraDB cluster, and it keeps emailing us in the night to express its gratitude for decent write latency. If you’re still grappling with MySQL master/slave replication, and you’ve not looked into XtraDB, stop reading right now and do so (it’s nearly the end of this post anyway). If you don’t know what master/slave replication is and why you’d grapple with it, it’s a kind of Greco-Roman wrestling.

 If you read to the end of this, you might well be the kind of geeky volunteer we could use. Drop us a line at geeks@ilovefreegle.org.