Rendered at 05:21:47 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
NewJazz 14 hours ago [-]
musl’s crypt() returns the string * as an error indicator when the key is over 256 bytes long. * is also exactly what the stored field contains for every account with no password, so the “hash” matches and the login succeeds.
Does anyone know why musl does this? Are they the only C library that behaves this way?
For a C library that famously tries to adhere to POSIX as much as possible, this seems like a crazy choice. From POSIX standard:
Upon successful completion, crypt() shall return a pointer to the encoded string. The first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Thanks yeah just woke up, finished making my way through the single-article website and still unclear if other libcs do the same thing (the article says libxcrypt picked it up, but only after musl did it).
NewJazz 9 hours ago [-]
Some more clarity ... Multiple implementations have these sentinel values (not OpenBSD). But unfortunately only musl had a sentinel value that was also used in shadow files...
This is either a bad engineering choice or an intentional bug. An empty string would have been better. Why '*' instead of, say, a tilde, @ symbol, or something that generally isn't used?
2 hours ago [-]
worksonmine 13 hours ago [-]
Apparently because some devs didn't check for NULL which resulted in crashes. Weird choice to fix other peoples' bad programming by deviating from the spec.
cyphar 13 hours ago [-]
To be fair (as the sshole site mentions) it is also necessary to fake a password hash check to avoid timing side channels revealing to remote clients whether users exist, so returning a sentinel value is not a horrifically unreasonable solution to make sure even naive code does approximately the right thing.
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a lengthy description of a long-deprecated and utterly broken hashing algorithm using DES that is even implied to be broken in the text itself (and the warning message is so old it refers to "parallel computers" as though they were a new concept). Given that the default mode is to use a broken hashing algorithm, I'm shocked anyone actually uses the libc crypt(3) -- surely it's trivial to just write your own that isn't awful...?
mik1998 12 hours ago [-]
Constant time strcmp doesn't fix it. The problem is that the user doesn't have a password at all (the hash is *). But which hashing algorithm is used is encoded in the password. So they pick a different user's hash to not leak the information that there is no password.
Alternate solution would be to hardcode a specific algorithm, but that could also leak info in many cases, I guess.
cyphar 3 hours ago [-]
No, I get that -- my first point was meant to allude to that but maybe it wasn't phrased particularly well. The bit about constant-time strcmp() was about comparing the hashes afterwards, which was a classic attack to extract the password hash for offline cracking (though salts probably make it untenable?).
In any case, the fact you need access to an existing hash to not select something stupid like DES is all due to crypt(3) being a wacky API and while callers could work around it by selecting more reasonable defaults it's a pretty big foot gun.
sysguest 10 hours ago [-]
hmm why not make some fake-users just for this stuff?
forthac 4 hours ago [-]
you mean like daemon?
api 14 hours ago [-]
TBH that sounds like a straight up bug in that it deviates dangerously from the spec.
shakna 9 hours ago [-]
The spec only defines a single error state - ENOSYS, that is basically only for systems that don't implement crypt at all.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
NewJazz 12 hours ago [-]
Which is notable because musl is more known for adhering to the spec to the point where they break programs that assume nonstandard behavior.
silisili 11 hours ago [-]
> adhering to the spec
Pedantic, but more "author's understanding of the spec" which isn't always correct.
> assume nonstandard behavior
I go back and forth on this one (spec vs standard behavior). It feels like at some point the most widely used impl -is- the spec, as we've seen in other places (web standards).
> Upon successful completion, crypt() shall return a pointer to the hashed password [...]. Otherwise, it shall return a null pointer and set errno to indicate the error.
shakna 9 hours ago [-]
You need to add that the spec _also_ only defines a singular error state, that isn't applicable here.
> The crypt() function shall fail if:
> [ENOSYS]
> The functionality is not supported on this implementation.
mort96 5 hours ago [-]
I don't think those error sections are generally meant to be exhaustive?
pdw 5 hours ago [-]
To the best of my knowledge, only OpenBSD and Fedora have a crypt(3) that returns NULL. Everybody else returns a string starting with "*" on error.
For example on Debian the manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an _invalid_ hash to the output field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
amluto 17 hours ago [-]
Wow, what a bug. I expect the exploit is literally as simple as trying to ssh in as a user with the password has ‘*’ in /etc/shadow and typing anything as the password.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
bombcar 14 hours ago [-]
Alpine matches the first.
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
otterley 1 hours ago [-]
Which distro has password authentication enabled by default? I haven’t seen that in years.
dpoloncsak 14 hours ago [-]
So it's essentially one of the 'recommended configurations'?
I'm seeing people I generally trust in the scene memeing on twitter, while others are freaking out. Its hard for me to tell how fucked we are, I guess
bombcar 9 hours ago [-]
It's always 50/50, you're either fucked or you ain't.
I think this time around it's mainly Alpine Linux and a few custom-rollers (Gentoo with musl could easily get to this place).
dpoloncsak 9 hours ago [-]
Yeah, as the dust starts to settle a bit it seems this is the largest case. Many 'stock' Alpine images, Docker images built on those, and embedded systems seem to be the main candidates.
ferngodfather 14 hours ago [-]
No, no don't let all of that distract you. We gotta hype this. This is the new SSH2SHELL dude. Have we registered the domain yet? I'm scheduled on Fox News tomorrow.
> We are making this release as fast as possible (we learned of the problem yesterday) because we have recently observed multiple cases of rapid bug rediscovery by AI agents, sometimes within hours, and so now consider long embargo periods to be a dangerous practice that primarily benefits attackers.
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
throwa356262 19 hours ago [-]
musl only, the quick fix:
PasswordAuthentication=no
Which is a good advice in general.
solyin 17 hours ago [-]
How do you break back into your digital life? You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop. How would you get back into your digital life given only a laptop with access to the Internet, unless something, somewhere is passwordauthenticate=yes.
jareklupinski 17 hours ago [-]
> You wake up in a strange place. Something happened. Everything's gone. No wallet/keys/smartphone/tablet/laptop.
at this point the _last_ thing i care about is my email.
where am i? am i ok? how are my friends/loved ones?
once those are in place, it becomes clearer how to get back to "normal"
solyin 12 hours ago [-]
With nothing, how do you reach them? Your friends and loved ones are what's at the other end of the email/Internet access.
jareklupinski 11 hours ago [-]
i guess if i have some sort of internet access (otherwise why would i be worrying about passwords), i'd set up a new account(s) and message those friends
otterley 1 hours ago [-]
Sometimes you just have to start over. It’s not the end of the world.
fn-mote 17 hours ago [-]
The somewhere for password authentication is a place you have physical access. Not ssh.
solyin 11 hours ago [-]
There's been a natural disaster and your home is now inaccessible. With global warming going the way it has, that's looking more and more likely each day. Fires, hurricanes, earthquakes, war, tsunami, etc.
gfv 16 hours ago [-]
PasswordAuthentication=no usually means that ssh stops directly reading /etc/passwd to authenticate, not that it stops accepting passwords completely. You'll still be able to log in with a correct password, provided KbdInteractiveAuthentication is still on, PAM is available, and the PAM policy allows for it.
nucleardog 15 hours ago [-]
The email at the root of my identity is tied to a country code domain in my country. There are robust legal processes for enforcing my status as the registrant and regaining control of it. Once I have access to that, I can use that to gain access to everything downstream.
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
viraptor 17 hours ago [-]
Remotely: An encrypted file with a seed / master passwords to all the other data, published under a public url you can remember, but not linked anywhere and not listed.
Locally: Same, but printed out, laminated, in a safe place with other documents.
denkmoon 17 hours ago [-]
I walk to my pc and type ssh-keygen. If i can’t do that I’ve got bigger problems. “I must be able to recover remotely using a memorised credential” is a terrible requirement.
solyin 11 hours ago [-]
You go travelling in a foreign country and get mugged. You have no phone no wallet no keys, just something you know and the goodwill of strangers.
BigTTYGothGF 11 hours ago [-]
> goodwill of strangers
Who can presumably get you in touch with the relevant embassy or consulate or, failing that, local authorities.
MPSimmons 16 hours ago [-]
Does this happen to you a lot? You may have other issues
INTPenis 17 hours ago [-]
Yeah your vault has password authentication. Your password manager in other words. Your vault is where you store your private ssh keys, and or the passwords for the keys.
mr_mitm 17 hours ago [-]
I log in via console, either physical or virtual (the latter requires MFA though)
worksonmine 16 hours ago [-]
A scenario where all the backups are gone and the server is somehow the only thing that survived is not a very plausible one.
pixl97 14 hours ago [-]
Depends if you're backing up somewhere else online (also is that protected by 2FA?). You must not be in a place with many huge natural disasters.
And this is assuming the server is somewhere else on the cloud.
worksonmine 13 hours ago [-]
> You must not be in a place with many huge natural disasters.
Or I have proper backups synced across the world on different providers. The likelihood of everything failing is not a vector I worry about. If that happens I have bigger problems to worry about.
petesergeant 17 hours ago [-]
1Password subscription and the memorized key. 1Password is the only place my ssh private keys are stored.
uresolame 16 hours ago [-]
[flagged]
illusive4080 16 hours ago [-]
Avoid this kind of behavior on HN; we are not the community for attacking each other. Your comment is not helpful to further the conversation. You could have made a point about subscriptions being antithetical to the ethos of open source culture, but instead you attacked the user and mocked him/her.
sva_ 15 hours ago [-]
Another thing I would recommend is having your servers in a wireguard net and only allowing ssh from inside that virtual network. There is no need to publicly expose ssh.
somat 14 hours ago [-]
Why is wireguards auth better than openssh auth?
TheDong 14 hours ago [-]
Wireguard allows effectively no configuration around auth, and so it's more likely that it is configured in a secure manner.
Openssh allows: password auth with weak passwords, PAM with potentially insecure PAM modules, gssappi, configurable ciphers, etc. Wireguard has none of those.
Each of those auth methods adds more possible vulnerable code paths and accidentally insecure configurations. It's very reasonable to believe that wireguard auth, since it has far fewer moving parts and configuration knobs, is likely to have fewer ways it can be insecure.
praash 14 hours ago [-]
Layered security is better (as long as their weaknesses don't overlap).
vadansky 14 hours ago [-]
Not only that, wireguard doesn't even reply if they don't have the right key, while SSH does so the attacker knows there is at least a target.
INTPenis 14 hours ago [-]
Yeah until they discover a vuln in wireguard.
sva_ 9 hours ago [-]
Then you still have your ssh safeguards
14 hours ago [-]
vadansky 14 hours ago [-]
I was always uncomfortable that it's on by default and surprised anyone uses it it. It's also more cumbersome compared to just using a key. I feel like using keys is the password manager analogue for SSH.
XorNot 11 hours ago [-]
It's irrelevant. There's no reason this should be any more effective in the face of software bugs then anything else.
patadura 17 hours ago [-]
imo this is a good advice only if the devices that connect to the server have their home encrypted and ssh private key protected.
Kuraj 7 hours ago [-]
> At least one local account has a $5$ or $6$ password hash.
I'm not sure this gets the point across when (according to https://sshhole.com/) it effectively means "any other user with a real password"
MPSimmons 16 hours ago [-]
Is this actually effective against Alpine? The default install seems to use ! as the placeholder in /etc/shadow instead of * and trying the attack doesn't seem to work.
feearent 10 hours ago [-]
Not really related to the topic, but AI-popular sentences and language structure stand out to people right away these days right? I noticed some.
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
monk_grilla 4 hours ago [-]
Agreed, I think all the tells you identified are correct. It is really off-putting to me still.
beepbooptheory 10 hours ago [-]
An aside but one nice thing I realize about not using LLMs is, I guess, I am not tortured by this kind of thing, because presumably I am not seeing it all the time. I can't even imagine quite being in the mode of reading this kind of text so closely like this, really. Its a security advisory, I don't really have my critical prose-reviewer hat on.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
feearent 9 hours ago [-]
Thank you for your comment, it's very refreshing (and comforting for me actually). Have a good day.
0127 11 hours ago [-]
IMO this is where languages like Rust shine. The design patterns built into the language for handling errors (Result) - force you to handle errors / function results with sentinels, null pointers, or errnos are so much more error prone than handling a Result<T,E> using a match statement.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
adrianN 11 hours ago [-]
I guess this is where the culture around rust shines. You can write crypt that returns a string in rust just fine, just like you can write crypt that returns a tagged union in C. Cultural differences and syntactic sugar change which version is idiomatic.
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
mort96 11 hours ago [-]
The culture is downstream of the design IMO. In general, people write the API that is obvious. In C, returning a string is obvious, while defining a custom OptionalString type is high friction. In Rust, returning Option<String> is as easy and obvious as returning a String.
asveikau 9 hours ago [-]
In a C function returning a string, two or three idioms are common for signalling an error.
One is to return NULL for errors.
Another is to make the string result a parameter and use a return code to signal error.
A third, less common but still out there, is an error parameter.
Failure to observe these is on the caller. Forcing the caller to consider the error is probably where rust shines.
But a good C programmer is thinking about these idioms when they encounter a function signature for the first time, or they write one.
mahboi 6 hours ago [-]
All the standard Rust libs use results. The standard C libs use errno, return 0, return -1, return EOF, return nullptr, segfault, or of course UB. Other libs dream up things like returning "*". C++ has exceptions in some areas but not others, and if you turn them off, they're segfaults. I actually like C, at least more than C++, but the error handling is the worst part.
dabinat 8 hours ago [-]
Sure, you could write deliberately bad code in Rust by unwrapping everything everywhere, but the point is that it is a distinct choice to do so. You can’t forget to handle errors - you have to make the choice to disregard them. If you forget to check for NULL in C it won’t complain and it’s dependent upon you to remember to do so.
mahboi 6 hours ago [-]
Unwrapping everything isn't even bad, that's what most webservers will do
spullara 11 hours ago [-]
always has amazed me that there was so much hate for java checked exceptions where handling errors ought to be the norm.
sieve 10 hours ago [-]
In the vast majority of programs, errors and non-results are irrelevant at the call site. You want to program for the happy path and not wrap every single call with error checking code or `try/catch/finally`.
h = file_open(path)
data = file_read(h)
file_close(h)
return data
is the code you want to read. This essentially is:
return file_open(path).success(h -> h, file_read(h)).success(h, data -> file_close(h); data)
A language with thoughtful error handling primitives will support such code.
zmmmmm 9 hours ago [-]
> You want to program for the happy path
This is the root of a lot of the evil. Yes, humans want to program for the happy path. It is a lot more fun. However, that bias leads to so much lack of diligence to properly deal with the unhappy path.
sieve 8 hours ago [-]
Syntax is not going to solve logic errors. No amount of checked exceptions or forced unwrapping of a result will help with TOCTOU or any number of similar vulnerabilities in code.
You ideally want to attack the problem at two ends:
- make the syntax noise-free so you can see what is happening
- build primitives at the syscall and stdlib level for common footguns
You would still need to deal with domain-specific logic errors, but you have at least reduced the surface area exposed to casual attacks.
groestl 6 hours ago [-]
> This is the root of a lot of the evil
It is not. Being extremely restrictive in the happy path, throwing all over the place, but have generic error handling code at the top level (e.g. HTTP entry point) makes for some pretty robust, stable, easy to read software. Albeit it typically has somewhat rough user experience.
The thing is, that's a pretty good place to start. From there on, you can handle exceptions one by one, surgically, and make things nicer.
mahboi 5 hours ago [-]
I keep hearing how diligent it is to handle errors at the call site, always by people who end up making services that either crash for preventable reasons or bury errors. Python is actually safer than Golang, and I don't mean with type hints.
mahboi 6 hours ago [-]
People using exceptions want to handle the unhappy path, just usually not at the call site.
spullara 9 hours ago [-]
that is the same as saying the method throws IOException
sieve 9 hours ago [-]
It might throw one. There is a difference between documenting what exceptions can be thrown and forcing every consumer to handle them.
spullara 8 hours ago [-]
returning a non-success you either handle it or pass it up the chain, potentially with not a lot of information about what originally caused it. with an exception you can retain the full information about what happened by either rethrowing it, throwing your own exception with it embedded as the cause or declaring that you don't handle it by saying you can throw it.
sieve 8 hours ago [-]
I generally prefer exceptions to result codes. You can even use the mechanism to throw useful results across nested function calls.
I do prefer Maybe/Option/Either style returns in some cases though. Makes for a cleaner API.
So each method has its uses.
mahboi 6 hours ago [-]
Yeah, checked exceptions are weird. Almost everything has some way to fail. The exact type of exception is usually meaningless, more like there are a couple of broad categories analogous to HTTP 4xx vs 5xx, and even then you usually don't want to handle it right there.
pandemic_region 8 hours ago [-]
IOException must be #1 on the list of most ignored exceptions. Like okay what am I supposed to do really I'm just trying to run a select against my db
pie_flavor 6 hours ago [-]
Well, for example, if you had opened a file to put the data into, you might delete the file or restore from a backup. But even if the answer is nothing, bubbling it all the way up to main is not the wrong answer. The point isn't whether you can do something, it's whether someone can do something.
groestl 11 hours ago [-]
The question is _where_ the error needs to be handled. It's almost never directly at the call site. And then there's a few issues with checked exceptions and Java language things (RMI, closures, etc) which make checked exceptions cumbersome. An exception (checked or not) might be worse than sum types, but they're still a thousand times better then returning an error in-band.
drdec 9 hours ago [-]
> The question is _where_ the error needs to be handled. It's almost never directly at the call site.
Which is why exceptions in Java can bubble up the stack? You don't have to catch it in a particular method you can let it go to the caller of the method, and so on.
mahboi 6 hours ago [-]
Checked exceptions must be handled or declared. You end up with everything saying it can throw several random things like IOException.
spullara 9 hours ago [-]
Rust and Go literally have you handle it at the call site by default and if you don't want to you have to pass it on.
mahboi 6 hours ago [-]
Go doesn't require you to handle or pass an error upwards. You can call something that returns err and just ignore it, compiler won't care. The "if err != nil" thing is annoying too. Rust made the right compromise with !
groestl 6 hours ago [-]
Which (at least the way it was done in Go) is ignoring 20 years of experience in programming language design.
0x457 8 hours ago [-]
Better design breeds better habits. In Java, you can end up handling a DNS exception 20 layers up the stack, even if your call never actually triggers a DNS lookup, simply because some method deep in the call stack declares it as possible. The culture is to propagate or wrap exceptions rather than translate them into domain-specific errors near the source.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
kaoD 11 hours ago [-]
I agree but I think was also because Java didn't have sum types (or still doesn't? years I haven't read Java) and rethrowing is super easy, while Rust kinda nudges you into the right direction (if you need to rethrow at all, think if you want to rethrow verbatim or wrap with a semantic error type).
Right idea at the wrong time.
zmmmmm 9 hours ago [-]
yep, 90% ergonomics in my view. If it had its time again with modern ergonomics thing could look very different.
javier2 11 hours ago [-]
They were overused, but some actually make sense, such as the IOException and SQLException
cloverich 9 hours ago [-]
The problem with them all always comes when the ever-steady march of abstractions, features, and edge cases results in one routine throwing the same two error classes but in different places, and how you handle them is different despite being the same wrapper. Add a thousand developers and several years of churn, and poof you get extremely complex bugs that result from errors seeming to come from one place, but actually coming from another.
Having been on the receiving end of debugging that a few times, I no longer mind the in-band error handling, although admittedly I'm inexperienced enough to be confident its better. Just for the random code I've had to read where I just need to know exactly how something I didn't write and will never read again works, and whether the error is handled and how, i love reading in-band error handling, even if I too dislike the way an error laden routine must be written. Over time I've come to appreciate elegance "in the large" as opposed to the code I see in any individual function.
_flux 10 hours ago [-]
I always thought the ideas was great, but type system was holding it back. If you could have expressed that your thrown exceptions are the same as some other method's (e.g. a method of a class given as a parameter), and some other classes, then it could have been a lot less annoying. Powerful, even! But Java did take its sweet time to get even type parameters.
I have written very little Java, though, so this is more of an armchair analyst position.
Terr_ 10 hours ago [-]
Amen, people saw the symptoms of "needs more syntactic sugar" and misinterpreted it into "rigorous error handling is bad".
The same way that layered code should react/wrap/translate when it comes to return types, it should do the same for thrown exceptions, and checked-exceptions helps you do that. They're both results of computation.
Lt_Riza_Hawkeye 11 hours ago [-]
It seems it used to return NULL, but they changed it for "convenience"
mahboi 6 hours ago [-]
How is this more convenient, being able to display it as text?
10 hours ago [-]
lakeviewterrace 7 hours ago [-]
Yeah too bad your adult-diaper language can't even host itself on most of the architectures where musl is relevant.
Another Rust self-suck session rendered completely irrelevant. Again.
drob518 12 hours ago [-]
Given Alpine’s broad use in the Docker ecosystem, this is going to hit a lot of folks, I think.
apexalpha 11 hours ago [-]
People generally don't expose SSH from within a Docker container I think.
eddyfromtheblok 8 hours ago [-]
no, ssh INTO a running alpine container. who runs sshd in a container when you can just attach a terminal?
folmar 7 hours ago [-]
you'll be surprised, ssh gives more ergonomic terminal, multiple connections, more convenient file sync and so on
AtNightWeCode 10 hours ago [-]
Considering how much people whined about the MinIO images I would expect that it is not that uncommon.
11 hours ago [-]
NewJazz 11 hours ago [-]
Eh, most folks aren't hosting ssh servers with alpine containers and password auth enabled.
asdfsa32 17 hours ago [-]
Who does PasswordAuthentication=yes in this day and age anyway.
dspillett 16 hours ago [-]
We have clients (regulated banks and insurers of various sizes) that send us feeds via SFTP. You might be surprised what proportion of them ignore our “we strongly recommend key based authentication” and ask for password auth instead.
somat 14 hours ago [-]
When I set up a client sftp drop I had the luxury of saying "key based access only"
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
pixl97 14 hours ago [-]
About to say, users just drop the key on their desktop and then download random exe's from the internet and run them.
Still safter in general, but not bulletproof.
dspillett 13 hours ago [-]
At least in that case, if the account is compromised we definitely didn't leak the credentials. That we don't have them to leak is one of the advantages of key-based auth.
[of course not having the credentials to leak does not protect us from this sort of bug, it is just one factor in a multipart security stance]
dns_snek 14 hours ago [-]
If you refused, would you get overruled internally?
dspillett 13 hours ago [-]
Previous company: probably. It could be a dealbreaker for the client, and we bent over quite readily on a number of issues considered minor (keys are best practise, but passwords are so common it is difficult to argue that they are so bad they should be refused).
Current state (now we have been bought and are effectively more than an order of magnitude bigger): I definitely don't have opportunity to make a unilateral decision like that. Even though the larger company has a more “fair enough, feel free to go elsewhere” attitude when a small client says “if you can support × that is a dealbreaker for us” then we could previously afford to take as our stance, some of the bigger clients are definitely paying us enough, and do use passwords, that someone above me would overrule a keys-only decision.
krystalgamer 16 hours ago [-]
it's the default for most systems. I can't remember a time I had to not turn it off.
bombcar 14 hours ago [-]
Don't most cloud providers default to that? I know I had to force some to deploy with a key and even then PasswordAuth was often on and had to be disabled.
mhitza 16 hours ago [-]
If you run a cockpit service on your server you will need password authentication enabled, unfortunately.
Curufir 15 hours ago [-]
I went and checked this. I can admin my server with Cockpit using either the web service (using my username/password) or the desktop client (via private key) without issue even with SSH password authentication disabled. I do usually disable the web service so had to reconfigure to check.
15 hours ago [-]
ectospheno 17 hours ago [-]
People who use AuthenticationMethods "publickey,password"
You must present a key first after which you must enter a password.
rvz 17 hours ago [-]
Just do no password at all.
ectospheno 17 hours ago [-]
Putting the one in 1fa? No thanks.
walrus01 17 hours ago [-]
As this is apparently affecting openwrt, probably a fair number of openwrt users, if I had to guess
drdexebtjl 15 hours ago [-]
I thought OpenWrt used Dropbear by default?
7 hours ago [-]
getfacl 10 hours ago [-]
I run Alpine on hardware. I was not effected by this as while I do use ssh keys, there are no user accounts without passwords on my server.
aborsy 13 hours ago [-]
I often use "AuthenticationMethods publickey" which would disable all authentications except public key.
Isn't this safer than "PasswordAuthentication no", even if it's a different setting?
nodesocket 10 hours ago [-]
This is brutal because a lot of docker containerized apps are very slow to update the base image if at all.
ls612 2 hours ago [-]
Docker images are generally not affected because they don’t ship ssh.
pishpash 5 hours ago [-]
They shouldn't be including any sshd.
zajio1am 16 hours ago [-]
I would say that PasswordAuthentication=yes is by itself a vulnerability.
crest 19 hours ago [-]
I'm sure some people running lightweight containers just had a heart attack.
smw 18 hours ago [-]
I would imagine the vast majority of lightweight containers don't contain an ssh server?
dormento 16 hours ago [-]
You'd be surprised by the amount of people using containers that don't know/care what is inside.
K0nserv 17 hours ago [-]
This looks terrible for musl, what a collection of terrible decisions on their part. I guess I'm staying away from it.
sheepscreek 17 hours ago [-]
To err is human. There’s no escaping it, I’m afraid.
K0nserv 16 hours ago [-]
I agree entirely, but I'm not sure changing `crypt` to return `"*"` instead of `NULL` on failure qualifies. It's much more of a considered decision, which happens to be very wrong.
12 hours ago [-]
alanfranz 3 hours ago [-]
> set PasswordAuthentication no in your sshd configuration and restart sshd.
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
11 hours ago [-]
SV_BubbleTime 14 hours ago [-]
> because we have recently observed multiple cases of
rapid bug rediscovery by AI agents, sometimes within hours, and so now
consider long embargo periods to be a dangerous practice that primarily
benefits attackers.
Seems quite reasonable. It’s quite obvious that black hats have day jobs as White hats and have been privy to embargoes for probably decades.
InsidiousWombat 13 hours ago [-]
Embargo should last at most to the moment fix is made and code passes test suite.
Anything longer is putting systems at danger. I can recompile the package with a patch (even if Ubuntu/RHEL takes 3 weeks to do it) myself in a pinch, and I'd like to have a chance for it before the bug gets re-discovered by malicious party
tremon 12 hours ago [-]
What if the test suite passes in the middle of the night? Should all administrators interrupt their sleep and get up to recompile their servers immediately?
tsimionescu 7 hours ago [-]
All times are the middle of the night somewhere. Now, it's true that some times are the middle of the night to more people than others, but that's rarely a usable piece of info.
SV_BubbleTime 3 hours ago [-]
Right, can’t be temporally bigoted in <current year>
stonogo 11 hours ago [-]
That is generally the practice in professional environments, yes. Many places have an on-call rotation, and those that don't soon discover why. An embargo wouldn't fix that particular problem anyway, because time zones exist.
Xylakant 10 hours ago [-]
Also: There’s always midnight somewhere.
pgporada 11 hours ago [-]
[dead]
aomix 14 hours ago [-]
I think this is a long held belief of the project and why they have been excluded from bug embargo’s in the past.
4 hours ago [-]
jmclnx 17 hours ago [-]
>We consider the root cause of this bug to be outside OpenSSH itself, but we're in a good position to deliver a mitigation quickly.
So not a ssh issue but an issue specific to Linux using "musl libc without PAM".
mpyne 15 hours ago [-]
Well, OpenSSH contributes to this by mixing hashes from the shadow file. The musl code would be correct if OpenSSH passed in the hash from the user account that is being logged-in.
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
bombcar 14 hours ago [-]
Randomly using a different users's hash seems so wrong to me, but maybe I've just been too deep in PII stuff for too long.
0x457 12 hours ago [-]
I keep saying that no one should be using alpine or musl, but people keep replying that static linking is totally worth it.
Sohcahtoa82 5 hours ago [-]
Static linking makes distribution easier, but it's an absolute security nightmare.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
muvlon 3 hours ago [-]
So then rebuild everything? I don't see the problem. NixOS shows how it's done.
pheggs 11 hours ago [-]
both alpine and musl are great in my opinion, and as far as I know theres no empirical evidence that it is more or less secure overall than alternatives. would you like to elaborate your opinion?
0x457 8 hours ago [-]
IMO only good thing about alpine is that it's a good environment to build stuff that links musl statically.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
thewillowcat 11 hours ago [-]
More evidence than allowing root to log in with any long password?
ceejayoz 11 hours ago [-]
This particular flaw sat latent for a decade.
There's a good chance similar flaws will be discovered elsewhere.
cluckindan 8 hours ago [-]
”Flaws” or ”intentionally planted backdoors” a la xz? That one was targeting openssl as well.
pheggs 10 hours ago [-]
I have to admit that I am a bit disappointed in this kind of thinking. There's vulnerabilities everywhere as especially the past few weeks have shown.
One occurrence is not a statistic and should not be used as evidence - it's just an example of a vulnerability that due to various bad circumstances lead to a particularity bad outcome. This logic is in my opinion equally flawed as calling alpine safe now because this particular vulnerability has been fixed.
I'm not a big fan of blaming and I think it's just a result of bad circumstances, but it's interesting to see how some seem to blame musl, but one could argue further that musl documented the behavior of their API which was ignored by openssh and so on
Muromec 11 hours ago [-]
Docker is a static binary and HTTP is a new command line. I don't care what's inside, nothing good anyway.
AtNightWeCode 10 hours ago [-]
I always use certificates with SSH. Saved me this time.
9 hours ago [-]
j45 11 hours ago [-]
It's nice to see how quickly this is being addressed, and likely monitored for any unintended side effects.
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
phoronixrly 19 hours ago [-]
I presume Dropbear, thus musl-based OpenWrt routers are also not affected? Not that Dropbear has been an example in that regard.
mijoharas 17 hours ago [-]
There is this write-up/fancy page from the person who found the vulnerability[0] (HN discussion[1], how I found this thread).
At the bottom of that it says dropbear is unaffected.
> on debian, ubuntu, fedora or anything else with glibc? going through pam? dropbear? openbsd? you're fine
Except in the case where their images are built with OpenSSH instead of Dropbear, which is the case for some manufacturers. (Turris, was noted as one such vendor)
manwe150 19 hours ago [-]
Since after you posted, various people in the thread have reported dropbear is not affected
slopinthebag 12 hours ago [-]
i think it's just unbelievable there isn't a bigger push to move away from c to languages that can prevent stuff like this from happening. this and just the pure unsafety. it's 2026. it's like pure negligence at this point, the equivalent of building a bridge with matchsticks because you like using it more.
my-huge-pony 12 hours ago [-]
How would using a language different than C help here? At it's core this bug is a logic bug, where "*" == crypt(x), because crypt in musl started returning "*" in some cases some years ago. This could be Rust. This could be Go.
Not every C vuln is a memory safety problem.
rhdjebejdbd 12 hours ago [-]
Another comment said that returning a * was to avoid returning null which would crash users of the library who didn't check for null. That seems like a C smell to me
ethin 11 hours ago [-]
Not really. This is like if the char* were an Option<String> but you had programs which stupidly didn't check if the Option had something and assumed it always did. As others have said, Rust would not prevent this.
K0nserv 11 hours ago [-]
`Option<String>` or `Result<String>` forces the check at the callsite. Unlike C, you cannot forget that the function can error. This absolutely helps prevent issues like this.
Now, the decision to return `"*"` instead of letting poorly written code crash on `NULL` is not good either.
pheggs 10 hours ago [-]
I don't think it's that people forget it while writing the code, it's that people want their program to work with as little effort as possible and decide to ignore it "for now". And that definitely can happen with Option and Result as well. There's no protection for not checking return values, and it's probably not even possible to protect against (in entirety)
_flux 10 hours ago [-]
Well they would use `unwrap` if they were lazy, but at least that would be quite visible in the code and would not result in undefined behavior due to operating on a null value. And there might be a group of developers who first choose to use .unwrap. I hope it is the minority.
Given how easy it is to just forward that error to the caller, I think most Rust developers would just choose to do that.
uecker 10 hours ago [-]
While dereferencing a null pointer is technically UB, it practice it leads to a safe trap. So .unwrap is the right comparison. If you always forward the error to the caller, it will also just lead to some random error message at some point, not really different to a crash.
K0nserv 10 hours ago [-]
> There's no protection for not checking return values, and it's probably not even possible to protect against (in entirety)
That's exactly what you get with Result and Option, it's impossible to forget.
Compare:
char* hash = crypt(key, salt);
// We crash here on the hash derefence, it's no clear from the code
// that we were supposed to check for NULL or that we forgot.
if (!strcmp(hash, expected)) {
}
To:
let result: Result<Vec<u8>, Error> = crypt(key, salt);
// We MUST write this line, which is a very clear signal
// that we can panic here.
// This can be grepped for or forbidden.
let hash: String = result.unwrap();
// if result == expected { would type error
if hash == expected {
}
pheggs 10 hours ago [-]
That's exactly what I mean. And I have seen a huge amount of unchecked unwraps in production code in the past years
K0nserv 10 hours ago [-]
Sure, but it's far better than the C code and I don't think in Rust a library would start returning `Ok` with sentinel values to "fix" shit calling code. Maintainers would, rightfully, tell people to fix their code.
pheggs 10 hours ago [-]
Don't get me wrong, I think rust has some great features and does some things extraordinarily good. But at the same time I can see that for 99% of cases it's just as prone to bugs, including security and supply chain attacks, as any other language. I think I could just as much argue that any "->" is clearly de-referencing that needs checking, which is also pretty easy to see.
K0nserv 9 hours ago [-]
Agree to disagree I guess :)
slopinthebag 9 hours ago [-]
this is theoretically, empirically, and objectively wrong. akin to saying that you can get into an accident in a truck as well as a motorcycle, so in 99% of cases there is no difference in safety.
pheggs 8 hours ago [-]
different kind of programming languages change the kinds of bugs and their properties, but it does not change the amount of bugs per line of code. in respect to C and rust for example, rust may have less memory bugs - which LLMs seems to be able to spot now - but it did increase supply chain attacks for example.
orf 7 hours ago [-]
> but it does not change the amount of bugs per line of code
You are really uninformed: different programming languages have very different volumes of bugs per line of code
>because crypt in musl started returning "*" in some cases some years ago
In Rust, the type of the return value would be a `Result<T, E>`, which means the chain of events that led to this would never have started in the first place.
my-huge-pony 12 hours ago [-]
In C the return value would be NULL, except it was changed by musl at some point.
Edit: at the beginning, crypt always returned a value. So hypothetical Rust return type would be a String, and then it would be retained for backward compatibility. Or something else would happen, this is a complex logic bug triggered by a 25 year old chain of events.
realharo 12 hours ago [-]
The type of the value in C is `char*`.
That's what led some people to write code where it crashes without checks. Which is still their mistake. But if the type was something that can't be implicitly used like an always-present value, that would have prevented the situation, even with those same people.
uecker 10 hours ago [-]
A pointer is a type where a null pointer can't be used as a regular value but leads to a trap when dereferenced.
atomicUpdate 10 hours ago [-]
You mean could be a `Result<T, E>`. There’s nothing in Rust that forces that.
bmacho 11 hours ago [-]
> How would using a language different than C help here?
Easier to formulate invariants and check/prove them. Better language correlates with safer software.
Gibbon1 9 hours ago [-]
I write firmware where the program has to run for years unattended.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
slopinthebag 12 hours ago [-]
why does this logic bug exist? why are they returning a string, why not a result type, (nil, err), or something else?
oh right, because c is a crusty old language with barely any features.
ethin 11 hours ago [-]
Rust would not save you here as this is a logic bug. Nor would, well, really any other language. I suppose someone might claim that Rust's result type would save you but it really wouldn't: I can easily make the error type a String or pretty much any other type with little consequence.
K0nserv 11 hours ago [-]
Rust would absolutely have saved you here. Even with `Result<String, String>`, which is pretty poor design, there's no confusing `Ok(s)` with `Err(s)`. Sure you can go out of your way to write something like
let hash = match result {
OK(s) => s,
Err(s) => s,
};
but at that point I feel like the library is forgiven (except for the poor error design).
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
iknowstuff 11 hours ago [-]
It would, due to Result<T> and #[must_use] on it. Nobody would return “*” as a sentinel in Rust.
0x457 7 hours ago [-]
[dead]
slopinthebag 9 hours ago [-]
languages with more features would mean developers aren't choosing between unchecked NULLs or sentinel strings.
rpdillon 12 hours ago [-]
musl's crypt function could be written in Rust, Python, or JavaScript, and still return * when the key is too long.
oguz-ismail2 12 hours ago [-]
What exactly do you mean "push"? What do you think is going to make musl developers switch languages?
slopinthebag 12 hours ago [-]
it's unbelievable to me that there isn't a larger movement of developers to move away from c towards better & safer languages. look at how much pushback the rust4linux folks get, primarily from c developers who are simply afraid of change. imagine engineers not adopting safer building practices because "they prefer the old way"....it's disgusting honestly.
what will it take to move away from c? older devs retiring. Planck said that "science advances one funeral at a time" and i fear it will be the same for software.
hollowonepl 10 hours ago [-]
Moving away from C in any POSIX system is in fact stop being POSIX system as POSIX and linux being unix like system is built on standard that is defined as C library specification.
Rewriting it to Rust is fine attempt, just hard to call it is unix/posix anymore. It may not matter but semantically true.
bigbadfeline 11 hours ago [-]
> what will it take to move away from c? older devs retiring.
I'll take forking and porting, thank goodness for LLMs.
10 hours ago [-]
anonreplier 12 hours ago [-]
guess what, this kind of sentinel value mistake could even be made in rust
slopinthebag 12 hours ago [-]
no rust programmer would ever return a sentinel string like this.
ethin 11 hours ago [-]
Are you sure about that? When backwards compatibility comes into the picture people tend to bend over backwards.
_flux 10 hours ago [-]
Backwards compatibility with what? Rust has always has had enum also for this exact case, to avoid encoding sentinels as strings or pointers.
slopinthebag 9 hours ago [-]
im quite positive the only rust programmer who would do this is a C programmer who doesn't understand rust
9 hours ago [-]
anonreplier 9 hours ago [-]
why, because they all use agents to do the coding and don't check it?
slopinthebag 9 hours ago [-]
because rust has features that makes doing it pointless.
in theory could someone do it? yeah, just like in theory someone could program a calculator to randomly rm -rf ~ when pressing +.
but in practice would anyone do it? no.
woah 12 hours ago [-]
Cars without seatbelts and airbags chopping off driver's heads in relatively slow collisions? Well guess what, that could happen in a modern car as well if the driver sticks their head out the window while going through a very tight tunnel!
anonreplier 9 hours ago [-]
you followed up an incorrect comparison with another one, are you sure you're a rust programmer?
9 hours ago [-]
tonetheman 12 hours ago [-]
[dead]
TZubiri 13 hours ago [-]
Here's an unrelated security issue I opened with Alpine earlier this year
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
Mantis99 13 hours ago [-]
[flagged]
486sx33 5 hours ago [-]
[dead]
slukmbr 10 hours ago [-]
[dead]
entuno 18 hours ago [-]
Seems slightly confusing that the OpenSSH release notes page says:
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
SAI_Peregrinus 14 hours ago [-]
musl libc violates the POSIX documented behavior of `crypt()`¹
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
OpenBSD is not affected, so apparently they are doing something right. But also glibc linux is not affected and pam linux is not affected, so it is only musl linux with the remote hole.
bigbadfeline 11 hours ago [-]
You became a victim of a very misleading post title. This isn't an OpenSSH problem at all, it's a musl problem.
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
The OP should be flagged.
mldbk 15 hours ago [-]
OpenBSD doesn't provide musl-linked OpenSSH by default, so not yet[x2]
heavensteeth 15 hours ago [-]
No, the bug is in musl but manifests in OpenSSH when compiled against it.
rpicard 16 hours ago [-]
This vulnerability doesn’t exist on non-Linux systems, so not yet!
15 hours ago [-]
jordiburgos 18 hours ago [-]
Workaround? Why not a fix?
ale42 18 hours ago [-]
AFAIU because the bug is in musl and not in OpenSSH
Does anyone know why musl does this? Are they the only C library that behaves this way?
For a C library that famously tries to adhere to POSIX as much as possible, this seems like a crazy choice. From POSIX standard:
Upon successful completion, crypt() shall return a pointer to the encoded string. The first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
https://pubs.opengroup.org/onlinepubs/9699919799/functions/c...
https://hachyderm.io/@ncopa@fosstodon.org/117564809789053485
Though I am surprised that all of the code examples do not use constant-time strcmp(). Actually, I wonder if even constant-time strncmp() would still leak information about whether the account is disabled because in that case both arguments are much shorter? Food for thought.
But yeah, the crypt(3) API is very 80s C, the man page even features a lengthy description of a long-deprecated and utterly broken hashing algorithm using DES that is even implied to be broken in the text itself (and the warning message is so old it refers to "parallel computers" as though they were a new concept). Given that the default mode is to use a broken hashing algorithm, I'm shocked anyone actually uses the libc crypt(3) -- surely it's trivial to just write your own that isn't awful...?
In any case, the fact you need access to an existing hash to not select something stupid like DES is all due to crypt(3) being a wacky API and while callers could work around it by selecting more reasonable defaults it's a pretty big foot gun.
Error handling is underspecified. The spec doesn't give you enough well-defined behaviour, meaning everyone lists pages of error-handling extensions they have added.
Pedantic, but more "author's understanding of the spec" which isn't always correct.
> assume nonstandard behavior
I go back and forth on this one (spec vs standard behavior). It feels like at some point the most widely used impl -is- the spec, as we've seen in other places (web standards).
> Upon successful completion, crypt() shall return a pointer to the hashed password [...]. Otherwise, it shall return a null pointer and set errno to indicate the error.
> The crypt() function shall fail if:
> [ENOSYS]
> The functionality is not supported on this implementation.
For example on Debian the manpage says:
> Upon error, crypt_r, crypt_rn, and crypt_ra write an _invalid_ hash to the output field of their _data_ argument, and crypt writes an invalid hash to its static storage area. This string will be shorter than 13 characters, will begin with a ‘*’, and will not compare equal to _setting_.
To the extent that systems that are configured in the vulnerable manner exist in the wild, I’m surprised they weren’t exploited years ago by trivial ssh bots.
P.S. Wow, I’m reminded of just how awful traditional UNIX-y C APIs and code are.
P.P.S. The musl code in question is nuts. The API it implements is awful. Seriously, here’s the manpage:
https://man.archlinux.org/man/crypt.3.en
You can’t call this function correctly. sshd’s “fix” is basically to assume that a too-short return is an error return instead of a success return.
PAM is also awful, but this is worse.
"It starts with a bold claim: "ssh accounts that should not accept a password will accept any password between 256 and 1024 characters". But then it keeps adding qualifiers:
- sshd compiled with musl
- not using PAM
- password logins enabled
- at least one account with '*' as the password "hash"
- at least one OTHER account with a valid password
The conditions are all ANDed, miss a single one and you're fine."
This doesn't seem to be nearly as big as an issue as they're making it out to be
Disabling PAM has been "generally good security advice" for decades; I remember suggestions to not use it in the 2000s.
Password logins are often enabled, especially on multi-user and default setups.
If alpine uses the * feature, it's going to be pretty common. Gentoo has it, as does Ubuntu.
If password logins are enabled, you're almost guaranteed to have at least one user with a password (root on all my boxes has one, even if I never use it and it's random garbage).
So the cheese lines up pretty well, this affects quite a bit.
If password logins are enabled, someone almost certainly has a password.
I'm seeing people I generally trust in the scene memeing on twitter, while others are freaking out. Its hard for me to tell how fucked we are, I guess
I think this time around it's mainly Alpine Linux and a few custom-rollers (Gentoo with musl could easily get to this place).
I think this will likely become standard. Nevertheless, a few stormy months are ahead.
at this point the _last_ thing i care about is my email.
where am i? am i ok? how are my friends/loved ones?
once those are in place, it becomes clearer how to get back to "normal"
Is it fast? No, but that's a feature.
But it does mean I get to lean on the existing processes and identity verification already part of the government bureaucracy and the courts as my backdoor.
Locally: Same, but printed out, laminated, in a safe place with other documents.
Who can presumably get you in touch with the relevant embassy or consulate or, failing that, local authorities.
And this is assuming the server is somewhere else on the cloud.
Or I have proper backups synced across the world on different providers. The likelihood of everything failing is not a vector I worry about. If that happens I have bigger problems to worry about.
Openssh allows: password auth with weak passwords, PAM with potentially insecure PAM modules, gssappi, configurable ciphers, etc. Wireguard has none of those.
Each of those auth methods adds more possible vulnerable code paths and accidentally insecure configurations. It's very reasonable to believe that wireguard auth, since it has far fewer moving parts and configuration knobs, is likely to have fewer ways it can be insecure.
I'm not sure this gets the point across when (according to https://sshhole.com/) it effectively means "any other user with a real password"
"If you cannot update yet, at least disable password logins: set PasswordAuthentication no in your sshd configuration and restart sshd. Either one closes the hole." (only AI would characterize a critical SSH implementation vulnerability as a hole closing game)
An attacker who can reach your SSH port can log in without knowing any password. It takes one connection, works every time, and needs no retries. (It sounds like the writer is bragging about the exploit instead of describing it. AI does this)
The attacker needs the name of an account whose password field is * (an account with no password set, which is normal for key-only accounts), and needs the machine to have at least one other account with a $5$ or $6$ password hash. The attacker then sends a very long password (257 to 1024 bytes). One user with a password is enough: it makes every account without one reachable. (same as above made stronger with the 'statement: statement2' format)
"Users coming from a directory service through musl-nscd (nslcd, winbind, sssd), since they have no shadow entry." (a human would be more clear about what a shadow entry is and give more context, I think)
Hmm the rest of the article (the remaining 70-80%) seems quite human.
Anyway, not sure why I spent 10+ minutes on this. It just jumps at me right away.
As a grad student I read and graded a lot of student philosophy papers (well before LLMs), and I just remember the exhaustion of it. It made me appreciate how much I could otherwise just read something, get information or enjoyment out of it, and not focus on each sentence construction.
Just to say, I do believe you, I get that if you have to put up with it so much in general, it must prime you to find it everywhere. You have to read so much more now, I'm sure, so you're trained to be in that mode. It makes sense! I am just sorry its like this for you and most everyone else now. It sounds exhausting.
crypt() returning `*` is insane especially since these kinds of sentinels are all over /etc/passwd
I suppose if people started writing lots of code that crashed on error in Rust some enterprising developer might improve the function to always return a non-error value and set it to ‘*’ in error cases.
One is to return NULL for errors.
Another is to make the string result a parameter and use a return code to signal error.
A third, less common but still out there, is an error parameter.
Failure to observe these is on the caller. Forcing the caller to consider the error is probably where rust shines.
But a good C programmer is thinking about these idioms when they encounter a function signature for the first time, or they write one.
This is the root of a lot of the evil. Yes, humans want to program for the happy path. It is a lot more fun. However, that bias leads to so much lack of diligence to properly deal with the unhappy path.
You ideally want to attack the problem at two ends:
- make the syntax noise-free so you can see what is happening
- build primitives at the syscall and stdlib level for common footguns
You would still need to deal with domain-specific logic errors, but you have at least reduced the surface area exposed to casual attacks.
It is not. Being extremely restrictive in the happy path, throwing all over the place, but have generic error handling code at the top level (e.g. HTTP entry point) makes for some pretty robust, stable, easy to read software. Albeit it typically has somewhat rough user experience.
The thing is, that's a pretty good place to start. From there on, you can handle exceptions one by one, surgically, and make things nicer.
I do prefer Maybe/Option/Either style returns in some cases though. Makes for a cleaner API.
So each method has its uses.
Which is why exceptions in Java can bubble up the stack? You don't have to catch it in a particular method you can let it go to the caller of the method, and so on.
Rust makes handling and mapping `Result` locally more ergonomic, even though `?` makes propagation easy too. Rust, however, also makes it easy to lose the context during propagation and end up with useless `Error`.
Right idea at the wrong time.
Having been on the receiving end of debugging that a few times, I no longer mind the in-band error handling, although admittedly I'm inexperienced enough to be confident its better. Just for the random code I've had to read where I just need to know exactly how something I didn't write and will never read again works, and whether the error is handled and how, i love reading in-band error handling, even if I too dislike the way an error laden routine must be written. Over time I've come to appreciate elegance "in the large" as opposed to the code I see in any individual function.
I have written very little Java, though, so this is more of an armchair analyst position.
The same way that layered code should react/wrap/translate when it comes to return types, it should do the same for thrown exceptions, and checked-exceptions helps you do that. They're both results of computation.
Another Rust self-suck session rendered completely irrelevant. Again.
Then it was a surprisingly customer service intensive process to get the keys from the customers. I never did manage to automate it fully. Weird enterprise systems I have never heard of with strange key formats. ssh1 keys, email mangling them. Their internal processes requiring a password.
And way to many "I am sorry but that was the private key you sent me, That key is now compromised and you will need to remove it and generate new keys anywhere it was used"
I almost regret not allowing passwords, almost, despite the headaches key based is still much better.
Still safter in general, but not bulletproof.
[of course not having the credentials to leak does not protect us from this sort of bug, it is just one factor in a multipart security stance]
Current state (now we have been bought and are effectively more than an order of magnitude bigger): I definitely don't have opportunity to make a unilateral decision like that. Even though the larger company has a more “fair enough, feel free to go elsewhere” attitude when a small client says “if you can support × that is a dealbreaker for us” then we could previously afford to take as our stance, some of the bigger clients are definitely paying us enough, and do use passwords, that someone above me would overrule a keys-only decision.
You must present a key first after which you must enter a password.
Isn't this safer than "PasswordAuthentication no", even if it's a different setting?
Funnily enough, they don’t recommend checking pubkey auth is enabled and working. By blindly (or agenticly) following those instructions the risk is getting locked out of some machine.
Seems quite reasonable. It’s quite obvious that black hats have day jobs as White hats and have been privy to embargoes for probably decades.
Anything longer is putting systems at danger. I can recompile the package with a patch (even if Ubuntu/RHEL takes 3 weeks to do it) myself in a pinch, and I'd like to have a chance for it before the bug gets re-discovered by malicious party
So not a ssh issue but an issue specific to Linux using "musl libc without PAM".
They instead pass in a different user’s hash (to avoid timing attacks) but end up comparing the result from musl against the actual user account, defeating musl’s simplistic safeguard to avoid this very issue.
I’d probably still call this a musl thing. Errors should result in a NULL return even if the application crashes from there, but at the very least a return value intended to not match any actual hash in /etc/shadow should be a random string or predefined UUID or something other than *.
With dynamic linking, you just upgrade the library on your system and you're done. With static linking, every binary that uses the library has to be rebuilt.
Never understood why Alpine is used to actually run anything in a container - if you statically link binary, then you don't need alpine as a base and if you don't what is the point? Don't @ me about image size, even without layer caching we're talking about tiny difference that is often dwarfed by size of your runtime.
Musl itself is a great example of taking "simple and lightweight" so seriously that everyone else gets to deal with the complexity. Slow allocator, underoptimized memcpy/string routines, limited locale support, no NSS, and endless glibc compatibility headaches.
But hey, at least libc is small. %%Proceeds to pull in megabytes of <any other>malloc, and other libraries to replace the slow parts of musl.%%
And now sshhole.com is a particularly spectacular example of what happens when "simple" meets the rest of the software ecosystem.
There's a good chance similar flaws will be discovered elsewhere.
One occurrence is not a statistic and should not be used as evidence - it's just an example of a vulnerability that due to various bad circumstances lead to a particularity bad outcome. This logic is in my opinion equally flawed as calling alpine safe now because this particular vulnerability has been fixed.
I'm not a big fan of blaming and I think it's just a result of bad circumstances, but it's interesting to see how some seem to blame musl, but one could argue further that musl documented the behavior of their API which was ignored by openssh and so on
Looking at the unpatched Ubuntu things in my life, and I'm pretty close to asking Claude to update things to see how quickly Alpine can get close, assuming responsiveness can stay this fast and quality.
At the bottom of that it says dropbear is unaffected.
> on debian, ubuntu, fedora or anything else with glibc? going through pam? dropbear? openbsd? you're fine
[0] https://sshhole.com/
[1] https://news.ycombinator.com/item?id=50378228
Except in the case where their images are built with OpenSSH instead of Dropbear, which is the case for some manufacturers. (Turris, was noted as one such vendor)
Not every C vuln is a memory safety problem.
Now, the decision to return `"*"` instead of letting poorly written code crash on `NULL` is not good either.
Given how easy it is to just forward that error to the caller, I think most Rust developers would just choose to do that.
That's exactly what you get with Result and Option, it's impossible to forget.
Compare:
To:You are really uninformed: different programming languages have very different volumes of bugs per line of code
In Rust, the type of the return value would be a `Result<T, E>`, which means the chain of events that led to this would never have started in the first place.
Edit: at the beginning, crypt always returned a value. So hypothetical Rust return type would be a String, and then it would be retained for backward compatibility. Or something else would happen, this is a complex logic bug triggered by a 25 year old chain of events.
That's what led some people to write code where it crashes without checks. Which is still their mistake. But if the type was something that can't be implicitly used like an always-present value, that would have prevented the situation, even with those same people.
Easier to formulate invariants and check/prove them. Better language correlates with safer software.
You are right, C with manual memory management, no lifetime management, lack of phat pointers and a culture of leet has problems with deranged maddening bugs that other languages don't have.
But this bug first shows how inband signaling is bad news usually. But worse it's a failure to adhere to the spec properly. If you implement a library and you don't adhere to the spec you're going to bone someone.
Perhaps even worse is obviously the spec doesn't come with any standard testing framework. That was also the root of Apples infamous goto bug.
oh right, because c is a crusty old language with barely any features.
@my-huge-pony does make a good point elsewhere in the thread: if the potential error was not foreseen and the code was written originally to return `String` instead of `Result<String, Error>` then even Rust would be in a pickle.
what will it take to move away from c? older devs retiring. Planck said that "science advances one funeral at a time" and i fear it will be the same for software.
Rewriting it to Rust is fine attempt, just hard to call it is unix/posix anymore. It may not matter but semantically true.
I'll take forking and porting, thank goodness for LLMs.
in theory could someone do it? yeah, just like in theory someone could program a calculator to randomly rm -rf ~ when pressing +.
but in practice would anyone do it? no.
https://gitlab.alpinelinux.org/alpine/aports/-/work_items/16...
It didn't give me the impression that I was dealing with a secure piece of software, it seemed like they were dealing with very complex issues while laughably failing at the industry basics that were settled years ago.
Similarly Ubuntu was also the only other distro that fucked this simple permissions config up, and lately with the latency to fix the TCP vuln, it seems to confirm that these are not reliable distros for the coming security era.
> This release contains a critical security workaround that affects Linux systems that use the musl libc and PAM disabled.
But that there's nothing about it on their security page.
Even if it's not technically a bug in their code, the fact that they've made a "critical" release to address it suggests that it should probably be listed on the security page as well. But maybe (hopefully) they just haven't updated the security page yet.
> Upon successful completion, crypt() shall return a pointer to the hashed password; the first two bytes of the returned value shall be those of the salt argument. Otherwise, it shall return a null pointer and set errno to indicate the error.
Musl returns '!' or '*', not null.
¹https://pubs.opengroup.org/onlinepubs/9799919799/
“Only two remote holes in the default install, in a heck of a long time!”
https://www.openbsd.org/
I'm pretty sure the misleading wording is deliberate, OpenSSH is an excellent but non-commercial project which a bunch of entities would love to gain control of, in order to bug it with their spyware - thus the constant nagging on the bot-infested web.
The OP should be flagged.