I've decided I'm a trendsetter, so I've been playing with Jev after hearing the hype. I assume this is how trendsetting works.
Jev is a model from TypeSafe that answers small, structured questions. I'm using its yes/no question type, which returns a number between zero and one for how likely it thinks “yes” is. It doesn't write me an explanation. I have to decide what to ask.
That limitation is half the appeal. How much can I find out about a program from a pile of very narrow answers?
I'm trying it on deliberately broken Python applications, with problems I can check against the code. The questions are cheap enough that I can keep trying variations. I'm particularly interested in whether it can find the security holes I've planted, without accusing every harmless bit of code alongside them.
Give it something smaller than “secure”
If I ask an LLM whether an application is secure, I leave it choosing what to examine as well as judging the result. A checklist pins down the things I care about, but a long response can still glide past a check or spend most of its explanation on something else.
What I'm working towards is a layer of facts. A fact is a small statement about what the program does, without a verdict on whether that behaviour is good.
A function hashes a password with MD5, turning it into a fixed-length value. Another uses MD5 for a cache key. A handler returns a customer's email address.
Those statements give me separate things to examine. Whether hashing a password that way is acceptable comes afterwards. Finding MD5 somewhere in a file hasn't answered what it's being used for.
With an explicit list, I can require an assessment for each item and see which still need one. The list stays put while I change the model or how I ask. I'm no longer relying on the model to decide which concern deserves another paragraph.
I still have to check Jev's answer against the source, even when it comes back at 0.97.
I'm trying generative models to extract those statements. Jev judges them against the source code.
A program with an answer key
The larger test application is a customer-feedback product with 27 Python files, and I've given it 26 planted problems alongside six harmless cases that resemble them. I wrote the answer key before running the tools. There are missing access checks, customer records going into logs and a cleanup function that announces a purge without deleting the records. Alongside them are things it should leave alone, including an MD5 cache key and a public profile returning its intended public fields. With those cases written down, I can check what the tool finds and what it leaves alone. Otherwise I'm left with a convincing-looking report I have to grade from scratch.
One line, a very different answer
I start with a smaller four-file case. A route returns a full user record, including email and phone, without requiring a login. Its neighbour has the login check. The fixed version adds one decorator, the wrapper that checks authentication before the handler runs:
@admin.route("/admin/users/<int:uid>")
+@login_required
def admin_user(uid):
I give Jev both versions separately, with questions about the code:
| Question | Before patch | After patch |
|---|---|---|
| Does any route return personal data without login? | 0.930 | 0.900 |
| Does this named route require authentication? | 0.070 | 0.980 |
| Same authentication question, unchanged route | 0.060 | 0.060 |
The broad question barely reacts. Naming the route gives me an answer that follows my change. The other route's answer stays put.
High scores mean “yes”. The first row asks about an unsafe condition, the second about protection.
That's more than I expected from this interface.
The information is spread across the route, the login helper and the record's fields. With the route named, Jev connects enough of it here to tell the versions apart.
Both functions use MD5. Now what?
Back in the larger application, these functions share a file:
import hashlib
def hash_password(password):
return hashlib.md5(password.encode()).hexdigest()
def cache_key(*parts):
return hashlib.md5("|".join(str(p) for p in parts).encode()).hexdigest()
MD5 is a poor choice for storing passwords. The cache key has a different job, and I've deliberately included it as a case that should survive the password check.
The whole file scores 0.97 on the question about insecure password hashing. That's reasonable, but where's the problem? I ask again with each function as the context:
| Context for the password-hashing question | Answer |
|---|---|
| Whole file | 0.97 |
hash_password | 0.97 |
cache_key | 0.36 |
Now I have somewhere useful to look. Both functions call the same MD5 hash function, one deals with a password, the other makes a cache key, and those different jobs are enough to pull the scores apart. That's a useful distinction.
For something returning one number, it's telling me a lot.
Less code doesn't always help. It's tempting to keep going, from a file to a function, then a line, then.... An assignment to a string literal can resemble several kinds of credential without the surrounding code. A function keeps enough of the job together to be useful here. I can follow the function's job beyond the call that caught my attention.
The sweep can find a candidate file, then narrower questions help locate the function. And the source stays beside the answer, so I can check the statement myself. Jev has already returned its number. The explanation is my job.
Making the exam a little too convenient
Some of my first questions describe the planted bugs a little too closely. I'm calling these general questions, but one even mentions authentication on sibling endpoints in the same file. That describes my example, but it's a suspiciously helpful hint for a supposedly general check.
Very thoughtful of me to make the exam resemble the answer sheet.
The source-code comparison covers 25 of the 26 planted problems. The remaining one concerns dependency versions, a separate lookup task.
I rewrite 19 of the 25 statements to make them more general. On this fixture, all 25 target issues still clear the 0.5 cut-off.
Encouraging. But I'm also finding problems in what the questions mean.
A handler can select a record using an identifier from the request, then check the caller's permission. My question about selecting the record can correctly answer “yes”. On code outside my fixture, that is exactly what happens. I've found the mechanism I asked for, then treated it as a vulnerability without asking about the protection around it.
This is why I like keeping the statements small. I can go back to the particular question and see what I forgot to ask.
Cheap enough to keep poking
Jev lets me batch named questions against the same context. I don't need a network request for every answer. My recorded sweep and function-localisation run makes 52 requests across 21 non-empty Python files. It takes about 73 seconds and costs roughly $0.001. A tenth of one US cent.
At that price, I can afford to ask the wrong question a few more times, so another variation is awfully easy to justify. I could happily spend ages seeing how many ways I can phrase the same question, then start wondering what happens if I change one word back, and.... That's an evening gone.
The numbers can wobble too. Across five identical requests, one question varies enough to cross my 0.8 cut-off. An unchanged file can move into or out of the report because I've turned a shifting number into a hard decision.
So I keep the answer beside its question and evidence. The facts and checks keep stable names, even while the scores under them wobble.
Someone else could write the questions
In theory, a compliance officer could maintain these checks without writing code.
“Customer records must not appear in application logs” is a normative statement, a rule about what should happen. “This function logs the full customer record” describes behaviour I could check against it.
They could build a catalogue from their own requirements and shared sources. A rule could keep its name, the questions used to check it and the code behind each answer, so the officer can inspect the findings and revise the question. The record-selection example is why I'd keep the rule separate from the question. Wanting an access check is sensible, and the question that rule actually needs is whether the caller is allowed to read the record, rather than which identifier selected it. Keeping the wording beside old answers would let me change a question without losing what those results were based on.
The interface I'd like is “no programming required”, for real this time. The person responsible for the requirements gets to maintain them without waiting for me to turn each one into a scanner.
An insurer could give clients a set of required checks. Those questions could travel between teams, with each team using its own tools. Plain text doesn't tie everyone to one scanning product.
A month or two developing that list could be tiny beside building equivalent security tooling. It puts control with the people who need it, without first funding a multimillion-dollar tooling project. Model checks might cost more per run than equivalent static checks. My bet is that the saved development work could dwarf that extra running cost.
The officer wouldn't need to find the vulnerable line or understand every detail.
A coding LLM could check the code against that requirement and propose a fix. Jev needn't explain the whole bug. I'd rerun the check and the application's tests after the repair. The question keeps its name, so I can compare its answers before and after.
The statements I'm trying
Here are the 25 example statements. I ask Jev to assess each against the supplied code.
The [absence] entries ask about missing protections. The record-selection example above is why a “yes” still needs interpreting.
An endpoint that returns personal data has no authentication decorator.
A SQL query is built by concatenating a value taken from the HTTP request into the query string.
An endpoint requiring no authentication returns names or email addresses.
Cross-origin requests are allowed from any origin while credentials are also permitted.
A handler selects records using an identifier taken from the request rather than from the authenticated session.
A query selects all columns rather than named columns, and the result is returned to the client.
[absence]An incoming webhook payload is acted upon without verifying a signature.[absence]An endpoint that verifies credentials has no rate limiting.A secret, token or password is compared with a plain equality operator rather than a constant-time comparison.
Passwords are hashed with a broken algorithm such as MD5 or SHA-1.
An exception is caught and discarded without being logged or re-raised.
A framework secret or signing key is assigned a hardcoded literal.
A third-party service credential is assigned a hardcoded literal that is not a documented-public key.
A token used to authorise privileged access is assigned a hardcoded literal.
A database connection string disables TLS or certificate verification.
Personal data is passed as an argument to a logging call.
Raw database rows are written to the application log.
Personal data is transmitted to a third-party API.
A file containing personal data is written into a directory served as static content.
A shell command is executed using a string built by interpolation.
Personal data beyond the recipient's own details is embedded in an outgoing message body.
A function reports that it deleted or purged records but executes no delete statement.
Source contains personal data literals that are not reserved test values.
The web server is started with debug mode enabled.
[absence]Session cookies are configured without the secure, httponly and samesite flags.
For now, I can get a list of functions worth opening, with the statement that drew attention to each one. The fun is seeing how much that modest interface can find, and then discovering which apparently difficult problems become manageable when I ask something smaller. I'm still surprised how far “yes or no” gets me.