- cross-posted to:
- meta@jlai.lu
- cross-posted to:
- meta@jlai.lu
Firstly, apologies to everyone for the extended downtime. Unfortunately, it was for a pretty bad reason. We were hacked.
The bad news is that it was a comprehensive attack, and the attackers had privileged access to our database system, across all of our services (except for writefreely, which doesn’t use postgres). From what we can tell, the attacker did not do anything with that access, so we don’t believe any user data was accessed, but we can’t be certain of that. For lemmy, the impact of this should be minimal. If you registered with a real email address, they may have that. User passwords are encrypted in the database, so if you were using a secure, non trivial password, it should be safe, but you should still change it. You should also reset your 2 factor authentication if you had it enabled, as the seeds for these are not encrypted.
Our understanding is that the attacker used a peertube exploit, then a postgres exploit and then a kernel exploit to systematically gain access to different layers of our database server. A side effect of the hack was that it filled up our database servers hard drive, and caused it to fail over to our backup, which we believe mitigated some of the potential fall out.
We have had to reset activitypub keypairs for every account and community on lemmy, so there may be some federation hicoughs for a day or so, until remote servers have dropped any cached copies of our users public keys. This is uncharted territory though, so hopefully it’s as smooth as we think it will be, but we can’t be sure!
As stated earlier, our writefreely instance is still up and running as it wasn’t impacted by this attack. Vernissage (our pixelfed replacement) has been brought back online, as has our matrix server.
We will be bringing up Sharkey, and then Piefed hopefully later today, but we have to rotate keypairs on those services too, which is also uncharted territory, so the timelines are hopes, not guarantees. At this point in time, we don’t plan on bringing pixelfed back online, as it was slated for shutdown in August in any case. If people still need access to pixelfed to export data, we can spin it up briefly if needed, so please reach out if this is you. We also won’t be bringing peertube back up at this point. It was not heavily utilised, and it was the source of the attack, so Kaity is a bit gun shy about spinning it back up on shared database infrastructure. If there is a strong desire to bring peertube back, we can consider doing that on isolated hardware, but at the current utilisation level, it doesn’t seem worth the cost/effort to run it isolated.
in any case, you can read a fuller explanation of the attack by Kaity here https://pen.blahaj.zone/supakaity/weve-been-hacked
Edit - Piefed is back now!
You must log in or # to comment.
Thank you so much, Ada and Will. Appreciate the transparency! :3
To all curious, for the future: if you cannot go to your account on the Blåhaj instance or open up any stuff from there, check the desktop website of the instance, just go to lemmy.blahaj.zone (or its piefed equivalent).
Chances are, that there may be something on it. If you have an alt, I’d recommend one on an instance that’s mutually federated with the Blåhaj one.
For changing passwords, your app may not support it - use the desktop environment.
Consider donating to the Blåhaj instance - kofi link!
(Might be good to put that in the sidebar too…)
This feels like an inappropriate time & place to plug my instance, but I have a bot set up to mirror instance bans from blahaj (and dbzer0), so my instance can be a safe space as a back up for folks who need one. Important differences in that regard though are that I do have downvotes enabled, federate with hexbear, and could only copy bans going back a little over a year (which is over 3000 accounts banned btw! They do so much work to keep a safe space it’s jaw dropping- I donate monthly as a thanks for the ban list lol)
Is there any way to change the email address on an existing lemmy.blahaj account? I’m not the best at research, sorry about that - but I wasn’t able to find an answer through a search engine and I haven’t found the option on any of the listed frontends.
Appreciate the steer - I was able to change my password, but nothing I change in the ‘Settings’ field saves when I click the button. Sorry for asking but really appreciate the support.
Edit: working now, not sure what was different - either way, thanks for the help, appreciated
Kaity fixed it. Sorry, forgot to let you know!
thank you for all your hard work Kaity, Ada, and the rest of team, and for the transparency. even tho this was a horrible thing, the honesty and work makes me hopeful in a dark world. lots of love <333
waow o.o;
i sure showed up at a doozie of a time… LITERALLY making my first comment right here!
i applied for the account like a month ago and finally managed to check on the status, only literally just now finished setting things up the rest of the way X3 feel like i lowkey dodged a bullet that I didn’t put anything up before it would’ve been stolen.
my condolences for the hardship you’re going through D:
I’m curious if they hardened their SE Linux, and I wonder what kernel bug was used and if its public. I know the recent batch of copy fail style exploits were patched. If these are zero days I fear the only way to accurately protect yourself is to run a full VM level hypervisor. And even then how long before we have zero days for hypervisor. I noticed C abd C++ applications seem more vulnerable so maybe moving to rust will help.
As ever, both of your dedication to transparency and communication is both excellent and very gratefully received! I did see you said in the post above Ada that keypair rotation and all that that entails would make Piefed a tricker recovery but I was wondering if you had any updates for Piefed recovery? If you don’t, you don’t :) I fully appreciate how time consuming dealing with all this is, I just thought I’d ask.
We just ran out of time to get Piefed back yesterday (Australian time). We’re also navigating around moving house and the Kaity’s day job. It will be up as soon as we can today (It’s currently 7am here)
Yeah I thought the timezone thing might be playing a part! I hope I didn’t come across like I was complaining in any way as I definitely am not :)
Thx for resurrecting us back.
Had a feeling that this was the fact. Glad to be back
Are IP addresses stored? And if so are they affected?
It looks like IP addresses are stored in the DB in lemmy. It’s possible that the attacker had access to those IPs, but we don’t believe they accessed them.
This is the sort of thing we would turn off if we could :\
I believe IP addresses are anonymized on hexbear although I dont know how it’s done.
We could do that by direct DB manipulation.
I just use a VPN. Paid one, but not one that’s advertised everywhere.
I would like to bring up my onion service post from earlier https://lem.lemmy.blahaj.zone/post/22655537
Neither side of the connection knows the ip address.
I don’t know anything about server management.
Oof. People can be shitty. It sounds like not collecting data spared a lot of potential damage, though, so good on you for that. I know was the point, and this just proves why that was such a good decision.
Oh, minor thing: there’s some piece of punctuation in the name pgcrypto that your blog software is interpreting as markdown to start and stop italics. Luckily you mention it twice in fairly quick succession, so the passage in italics is relatively short, but you might want to escape out that character. I realize this is an unbelievably low priority right now, and possibly forever. It’s just something I would want pointed out to me if it were my post.
It’s good to be back
So happy you’re back! I was so desperate as to visit Reddit a few times. It was horrible. You’re amazing! ❤️❤️❤️
Guess we now know where the database problems where coming from.
Yikeeeesss. Wishing you all the best, and many thanks for the work you do for the community!
I was wondering what was going on. First noticed it with matrix, then Lemmy.
