Talk:Fuzzing

Acoording to this article, open-source software is inheritently less secure than closed-source software -- this is quite surprising since one of the major claims of the open source enthutiasts is that open source is more secure than closed source.

The problem is the following sentence: Since major customer and enterprise management software is starting to be open-source, database-based security attacks are becoming more credible.

Whomever wrote that, should clarify why, being open source, this is so.

After having read the sentence again, I think it should also be clarified what exactly is ment -- are open source database systems less secure or are open source databases (systems?) used somehow in attacking other systems? And more importantly, just why is this realted to fuzzing at all? FrederikHertzum (talk) 14:32, 25 September 2008 (UTC)

Oddly, there is a security connection between open source software and fuzzing, but it is not what you might expect. Fuzzing treats the software as a "black box", in which none of the internal workings (including source code) are visible. Because fuzzing is so effective at finding glitches without source code and because some of the earliest fuzzing experiments showed the FSF's GNU suite of Unix tools to be more robust than any proprietary equivalents, open-source proponents sometimes cite fuzzing as yet another reason people should not rely on the false security through obscurity provided by closed-source software. I'm not sure about adding this to the article, however. While this information is citable, I'm not sure that it has encyclopedic worth. Thoughts? Ben (talk) 06:05, 19 August 2014 (UTC)

I clicked on the "fuzz testing" link in a user page (it read "I've been fuzz testing MediaWiki"). Although I'm far from a technological illiterate, the opening paragraph still doesn't quite make the meaning clear to the layperson. My questions are as follows:

• The basic idea is to attach the inputs of a program ... Given that software is essentially intangible, it is unclear how one would "attach" anything. Additionally, it's unclear what "the inputs" means. I know the term "input," but can't make sense of it in this usage.
• to a source of random data ("fuzz") ... need an example or clarification there, too, though it may become more self-evident once "attach the inputs" is cleared up.
• (for example, ... by failing built-in code assertions) ... I submit that a typical layperson doesn't know what "code assertions" are.

I know that one of the Wikipedia guidance documents mentions that there's only so far you can water down some very technical subjects, and that some articles are likely only to be accessed by people who already have some background in the terminology and concepts of the field. However, I think that this introduction is trying hard to use simple terms, is currently unsuccessful in that attempt, and is very close to success. Lawikitejana 02:32, 7 September 2006 (UTC.)

The introduction to the article appears to give credit for the idea of fuzz testing to Barton Miller and students in 1989. Yet at the bottom of the page, there is a link to folklore.org describing how fuzz testing was being done on the Mac on 1983! It seems that fuzz testing might have originated independently several times, perhaps with the Wisconsin group being the first to publish it, at least in a major journal (CACM, I think?). It might be more appropriate to have a "history" section on the page which discusses these multiple origins, instead of giving all the credit to one group.

I agree with the dubiousness of this being created in 1989; I know that I personally used it in 1985, and I didn't invent it at that time, as I had heard of others doing it. Not sure if it was published, but certainly it was already widely known in 1985. - Glenn

Precisely, someone started a page on fuzzing history. It has been unchanged since 2008 but seems correct. The contents of this page should be merged into the Fuzz testing page, and the Mac 1983 folklore.org you mention should also be added. 188.111.75.64 (talk) 18:17, 1 May 2013 (UTC)

Actually, George J. Carrette, the author of the 1991 UNIX fuzzer "crashme", cites his design as coming from the formal specification in Cybernetics by Norbert Wiener. That book was published in 1948, so it would seem to have precedence if credit is to be given to anyone. Here is the quote from the crashme webpage:

A bit of background on crashme. It is a tool for testing the robustness of an operating environment using a technique of "Random Input" response analysis. This I first saw formally proposed in the book Cybernetics by Norbert Wiener, but which any parent who has observed his children playing and learning would be well disposed to describe in detail.

I am not an expert in the field, so I will not edit the page directly, but I think it would make sense to at least make mention of Norbert Wiener and his book. And while the true originator of "fuzzing" may never be found I suggest changing this sentence in the article's introduction:

The field of fuzzing originated with Barton Miller at the University of Wisconsin in 1988.

It is pretty clear that, while the term fuzzing may be from 1988, the field of fuzzing has been around a lot longer than that. Perhaps, if Carrette is right, fuzzing originated when the first child played with the first toy by poking at it to see what it would do. Ben (talk) 05:48, 19 August 2014 (UTC)

why the two way links between Glitching and fuzz testing. i fail to see the connection. —The preceding unsigned comment was added by 189.172.38.108 (talk) 22:22, 10 February 2007 (UTC).

Article says "There are at least two different forms of fuzz testing:" and proceeds to show 3 bullets.

The last bullet should be outside the list, or the text above should say "...three..." —Preceding unsigned comment added by 61.11.49.253 (talk) 11:55, 28 December 2007 (UTC)

I think whoever wrote this article didn't know what mutation analysis was. It's not comparable to fuzz testing at all, as M.A. is for testing the test suite, not (directly) for finding bugs. I've deleted the whole paragraph discussing differences between the two. --Povman (talk) 00:32, 21 May 2008 (UTC)

I´ve kicked out MuDynamics company marketing because of unnecessary blabla. They should improve their systems before they go and make marketing for low price fuzzers that are only available on an apliance that is not scalable or otherwise working good. They have focus on a very small band of protocols and they cannot be used on mobile things (would you carry an appliance weighing multi pounds to test a wireless access point - hahahaha). - Anonymous

But shouldn't there be an overview of the Fuzz testing products available? For example, I would add gremlins.js , which does this for JS apps. - Dsernst (talk) 13:15, 17 July 2014 (UTC)

i propose to remove this text since it's not really related to this topic. this page is allegedly about fuzz testing then this section talks about how fuzz testing isn't good enough (are value judgments germane?) and goes on to promote a particular company's technology:

"Robustness testing" was introduced by PROTOS researchers in 1999 (most of them now part of Codenomicon, headquartered in Oulu, Finland) to increase the test efficiency through systematic tests.<ref>[http://www.ee.oulu.fi/research/ouspg/protos PROTOS - Security Testing of Protocol Implementations<!-- Bot generated title -->]</ref> Robustness testing is a model based fuzzing technique, an extension of [[syntax testing]], that systematically will explore the input space defined by various communication interfaces or data formats, and will generate intelligent test cases that find crash-level flaws and other failures in software. <ref>The technique is described in a University of Oulu white paper on [http://www.ee.oulu.fi/research/ouspg/protos/analysis/WP2000-robustness/ Robustness Testing] published in 2000, by Kaksonen et al.</ref>

i really don't see why this belongs on this page at all; go create a page on robustness testing and say whatever is relevant to that subject on that page —Preceding unsigned comment added by T0pgear09 (talk • contribs) 07:53, 28 March 2009 (UTC)

thuis pg is in my watchlist and so i noted recent changes; other than removing Peacock terms per WP:PEACOCK and re-wording a bit seems ok to me

i cant helep but wondeer if hte person means this org in finland: http://www.ee.oulu.fi/research/ouspg/ which is not exactly what is referred to in the text provided but maybee the name the person wrote is the orig name? come to think of it, i think the mention of a company in this line is outside the bounds of an ext link and i;ll do another edit to remove just that bit and ill chng the name of the group to whats on the website —Preceding unsigned comment added by T0pgear09 (talk • contribs) 05:55, 1 May 2009 (UTC)

I don't visit here often but I just caught up on edits over the last year. Maybe the Network Statgey Partners reference I added back was originally removed by mistake? It now has a home near all the other links to commercial papers on fuzzing. Informationh0b0 (talk) 17:28, 30 September 2009 (UTC)

The computer science litterature does not equate random testing with fuzz testing. Only Wikipedia does this.

In general, fuzzing is a particular form of (black-box or white-box) random testing.

See, e.g.,

  Patrice Godefroid, Michael Y. Levin, and David Molnar
  SAGE: Whitebox Fuzzing for Security Testing
  Communications of the ACM, 2012.
  http://research.microsoft.com/en-us/um/people/pg/public_psfiles/cacm2012.pdf

which defines "blackbox fuzzing" as "a form of blackbox random testing, which randomly mutates well-formed program inputs and then tests the program with those modified inputs".

Yet, Wikipedia redirects "random testing" to "fuzz testing". Because Wikipedia takes things in the other way round, the explanations are nothing but confusing.

Vasywriter (talk) 19:37, 8 April 2013 (UTC)

The fact there is a redirect between random testing and fuzz testing is not an endorsement that they two terms mean the same thing. I suggest that you look at the history of that "article" to determine how that happened. Since the term is not frequently used in the software development world and since no articles point there, it's not really an issue. If you actually want to resolve the issue, we can nominate that article for deletion. Walter Görlitz (talk) 20:26, 8 April 2013 (UTC)

I can't follow your point. Which term is not frequently used in the software development world? Random testing is universally used. Fuzz testing is also intensively used (cf. Microsoft SAGE, just for one instance). 188.111.75.64 (talk) 18:03, 1 May 2013 (UTC)

As of today, I see that the "random testing" page is separate, and includes a link to here. But it is also important to clarify in this article what makes "Fuzz testing" different than "random testing" - as my impression from reading the article was that it was just random testing. On the other hand the explanation by Vasywriter indicates that it does have a distinct meaning. AmirOnWiki (talk) 11:50, 5 February 2014 (UTC)

Actually computer science literature does not provide any clear distinction between monkey testing, random testing and fuzz testing. The older (and trivial) term is Monkey testing, the most accepted one is Random testing, and finally the more popular one these days in CS is Fuzz testing, but this just a term popularity. The particular paper cited by Vasywriter actually only describes one randomization strategy, that their authors think it a definition of fuzzing does not make it one. OWASP for instance lists a much wider variety of randomization strategies under Fuzzing, including "for binary: random ones", cf. https://www.owasp.org/index.php/Fuzzing 89.158.120.107 (talk) 07:30, 18 March 2016 (UTC)

Then computer science literature should get updated. Software testing literature often provides clear distinctions. Check StickyMinds.com Walter Görlitz (talk) 17:37, 18 March 2016 (UTC)

Maybe it would be a good idea to implement this information into the article. Here a few links:

• https://blog.hboeck.de/archives/868-How-Heartbleed-couldve-been-found.html
• https://fuzzing-project.org/
• http://www.golem.de/news/fuzzing-wie-man-heartbleed-haette-finden-koennen-1504-113345.html (German)

Some tools which are mentioned (and could be added to the article) are: American Fuzzy Lop, Address Sanitizer --rugk (talk) 19:21, 27 April 2015 (UTC)

Those sources for Heartbleed are not RSes.

We would have to decide if we only wanted to include notable tools (i.e., those with articles) or open it up to any tool that claims to be a fuzz testing tool. Walter Görlitz (talk) 03:55, 28 April 2015 (UTC)

The sources are reliable. They are all from an author of a more or less well-known German news website (third link) which is golem.de. Golem.de was also used in quite many English Wikipedia articles as a source. --rugk (talk) 19:48, 28 April 2015 (UTC)

They are? I don't think a blog is a RS, more-or-less well-know is less. Feel free to take it to WP:RSN though.

That we have a single author pushing the idea is also problematic.

You didn't discuss whether we wanted to include notable tools (i.e., those with articles) or open it up to any tool that claims to be a fuzz testing tool. Walter Görlitz (talk) 04:33, 29 April 2015 (UTC)

Golem.de is well-known (in 2012 it was on the 5th place of websites with focus on technical topics). If or what tools to include I leave up for others to decide.

BTW also the Linux Foundation now supports the fuzzing project:

* http://www.golem.de/news/open-source-linux-foundation-bewilligt-geld-fuer-weitere-security-projekte-1506-114833.html

* http://www.linuxfoundation.org/news-media/announcements/2015/06/linux-foundation-s-core-infrastructure-initiative-funds-three-new

--rugk (talk) 08:41, 28 June 2015 (UTC)

How about tools that have been specifically discussed in the course of academic research? Another option is creating a tools comparison entry. MikeAntares (talk) 22:37, 16 May 2016 (UTC)

e.g., this paper on fuzzing, with a table comparing tools on page 33. — Preceding unsigned comment added by MikeAntares (talk • contribs) 22:34, 17 May 2016 (UTC)