A naive question: Are KDFs meant to improve the security of systems or merely preserve it? I was under the impression that the purpose of KDFs is twofold:
a) Preserve entropy of the password material (up to a limit - size of the input to the next stage)
b) Derive a fixed-length output suitable for use as an input for the next cryptographic stage.
If the password is, say, 'abcd', then it could be converted into an AES-128 key in the following ways:
1). Use 'a', 'b', 'c', 'd' as the first four bytes of the AES-128 key (32 bits) in ASCII form, and then pad the rest with zeros or ones or use some other well-known pattern (based on convention). This method preserves entropy but is not computationally expensive.
2). Use a KDF like PBKDF2.
On paper, PBKDF2 is a better option because an attacker would need to perform the PBKDF2 computation before each decryption attempt, which would be time-consuming. Therefore, as long as the KDF is implemented correctly, it should offer better protection than the first option.
However, if we're talking about an attacker has the resources to brute-force a large number of generated passwords (based on real-world use or derived from hand-crafted or ML-derived criteria), they can precompute KDF outputs for each of those passwords and reuse them. This would make the second scenario (using PBKDF2) as easy or hard as the first one (using simple padding).
So there are different types of kdf's. For passwords, the goal of the kdf is to "stretch" the entropy by making the kdf function hard to compute.
PBKDF2 basically works by hashing the password over and over again. Using pbkdf2 with 100,000 rounds, means bruteforcing is 100,000 times slower, because you have to hash each guess 100,000 times.
Argon2 takes the idea further by being memory hard and resistent to parallelism. People briteforcing will use gpus to try lots of guesses in parallel (or asic if money is no object). These can handle pbkdf2 really fast but not argon2
>they can precompute KDF outputs for each of those passwords and reuse them
I'm not sure what is exactly meant by "precompute", but any sane implementation would use salts to prevent an attacker from precomputing common passwords so they can be used across multiple targets.
However, if we're talking about an attacker has the resources to brute-force a large number of generated passwords (based on real-world use or derived from hand-crafted or ML-derived criteria), they can precompute KDF outputs for each of those passwords and reuse them. This would make the second scenario (using PBKDF2) as easy or hard as the first one (using simple padding).
PS: Not a cryptographer, please don't shoot!