User talk:The Transhumanist/StripSearchSorted.js

Script to format search results as a list of page names with bullet list wikicode provided

(Originally posted to User:Evad37).

I've written a script called StripSearch.js that unclutters search results to make them bare lists of page names.

Now I'm writing a sequel to it called StripSearchInWikicode.js.

I would like the output of search results to look like this:

* [[Benjamin Franklin]]
* [[Larry Page]]
* [[Carl Sagan]]
* [[Hillary Clinton]]
* [[Warren Oates]]

...for easy copying and pasting into articles.

I'm having trouble manipulating the elements of class "mw-search-result-heading".

I gather that you put them into an array like this:

var x = document.getElementsByClassName("mw-search-result-heading");

I'd like to subject the items in that array to a regex, using the jQuery .each method, or the .each function, but I don't know how. The documentation is confusing as hell.

I think the search string (<a.*a>) and replacement string * [[$1]] ought to work.

Any pointers would be most appreciated.

Sincerely, The Transhumanist 12:58, 29 September 2017 (UTC)

@The Transhumanist: You don't really need anything that complicated – you can just insert content before and after each element with class "mw-search-result-heading" using jQuery's prepend and append methods:

$(".mw-search-result-heading").prepend('* [[').append(']]');

just about does the trick. - Evad37 [talk] 13:51, 29 September 2017 (UTC)

Or even better

$(".mw-search-result-heading").children().before('* [[').after(']]');

(this avoids leaving a space before the ]]) - Evad37 [talk] 13:53, 29 September 2017 (UTC)

You are right, the first method would be perfect if it didn't insert an extraneous space.

The second method inserts * [[]] unexpectedly on the same line after various entries, like this (searched for "genre"):

* [[Genre]]

* [[Genre art]]

* [[Rapping]] * [[]]

* [[Pop music]] * [[]]

* [[Trap music]] * [[]]

Is there a way to apply regex, to avoid both problems?

Another feature I would love the script to have is to strip redirected entries out of the search results. Those are the mw-search-result-heading entries that include searchalttitle inside their divs. I would like to remove just those instances of mw-search-result-heading.

Adding that feature would probably also solve the bug in the second method you presented above.

Would .not work for this, to hide divs with the class mw-search-result-heading except for those that do not contain searchalttitle?

Unfortunately, I don't know how to apply regex to facilitate matches for this type of thing. I can construct regex strings, I just don't know how to put them into play.

Forgoing jQuery, I think a for loop could be set up like this:

    // Strip out redirected entries
    var x = document.getElementsByClassName("mw-search-result-heading");
    for (i = 0; i < x.length; i++) {
        // somehow remove this entry if
        // it contains element of class "searchalttitle"
    }

But I don't know how to write the guts.

By the way, the script failed when I ran it with that empty for loop, and it failed when I tried sorting the array, like this:

    // Sort the search results
    var x = document.getElementsByClassName("mw-search-result-heading");
    x.sort();

It's enough to make one's head spin. :) The Transhumanist 20:50, 29 September 2017 (UTC)

Loops and regex aren't always the best tools, especially when working with collections of elements. jQuery has several ways to filter and refine results. One way would be to only apply * [[]] to the first-child elements within .mw-search-result-heading like so:

$(".mw-search-result-heading").children().filter(':first-child').before('* [[').after(']]');

Another way, like you alluded to above, is to first remove the searchalttitle elements, and then the * [[]] can be added safely:

$(".searchalttitle").remove();
$(".mw-search-result-heading").children().filter(':first-child').before('* [[').after(']]');

Or to remove instances of mw-search-result-heading which contain searchalttitle you can use .has():

$(".mw-search-result-heading").has(".searchalttitle").remove();
$(".mw-search-result-heading").children().before('* [[').after(']]');

Which can also be written slightly more succinctly like so:

$(".mw-search-result-heading").has(".searchalttitle").remove().end().children().before('* [[').after(']]');

Note that you can use .hide() instead of .remove() if you want to be able to show those elements again at some point. - Evad37 [talk] 02:52, 30 September 2017 (UTC)

Wow. You make it looks so easy. So, you chain methods to a selector. Nice. That sure is convenient. jQuery is simpler than I thought. When you chain methods to a class, they work on all the elements of that class. I was doing that with hide, but was just copying the examples and didn't really grasp the underlying structure. Thank you. And on retrospect, with loops and regex, it looks like I was trying to conduct surgery with an icecream scoop. :)

I try to follow along in the documentation during these discussions, so that I can grasp the jargon. While doing so, I noticed this:

$(".searchalttitle").remove();
$(".mw-search-result-heading").children().filter(':first-child').before('* [[').after(']]');

can be refactored to this:

$(".searchalttitle").remove();
$(".mw-search-result-heading").children(':first-child').before('* [[').after(']]');

It seems to work!

The script is now operational, thanks to you. But, I came across an unforeseen obstacle. The results look great on the search results page, but when you copy and paste them into an edit page, there is a blank line between all the entries. That requires that the user regex them all out in WikEd. I'd like to eliminate that manual operation by removing the blank lines in the search results.

Also, when we remove the .mw-search-result-heading entries that contain .searchalttitle, additional blank lines are left behind. Is that a clue that can help us track those newlines (\n) down?

It is not apparent where the newlines are inserted in the page source for the search results page. So, I assume they are specified on a style sheet somewhere. What is the most effective way to hunt down the style sheet which defines a particular class used on a Wikipedia page? The Transhumanist 21:24, 30 September 2017 (UTC)

It all seems to be very much browser dependent. Chrome gives me the expected result:

There is a page named "Genre" on Wikipedia
* [[Genre]]
* [[Yuri (genre)]]
* [[Film genre]]
* [[Literary genre]]
* [[Harem (genre)]]
* [[Genre studies]]
* [[Music genre]]
* [[Western (genre)]]
* [[Bara (genre)]]
* [[Genre fiction]]
* [[Biblical genre]]
* [[Epic (genre)]]
* [[Genre art]]
* [[Thriller (genre)]]

Firefox adds spaces at the start of each line:

 There is a page named "Genre" on Wikipedia

    * [[Genre]]
    * [[Yuri (genre)]]
    * [[Film genre]]
    * [[Literary genre]]
    * [[Harem (genre)]]
    * [[Genre studies]]
    * [[Music genre]]
    * [[Bara (genre)]]
    * [[Western (genre)]]
    * [[Genre fiction]]
    * [[Biblical genre]]
    * [[Epic (genre)]]
    * [[Genre art]]
    * [[Thriller (genre)]] 

IE adds several newlines between each item:

There is a page named "Genre" on Wikipedia

* [[Genre]] 




* [[Yuri (genre)]] 




* [[Film genre]] 




* [[Harem (genre)]] 




* [[Literary genre]] 




* [[Genre studies]] 







* [[Music genre]] 




* [[Bara (genre)]] 




* [[Western (genre)]] 




* [[Genre fiction]] 











* [[Biblical genre]] 




* [[Epic (genre)]] 



* [[Thriller (genre)]] 







* [[Genre art]] 

That's all on windows 7. And you're presumably using some other browser/OS combination. Not really sure what the solution is though. - Evad37 [talk] 03:28, 1 October 2017 (UTC)

Since the removed items each leave behind a newline, my guess is that it's one newline per div. But what div? There is other formatting there, including alternating background colors, and a solid border between entries. If I can remove the divs that the removed entries were in, that might get rid of some of the extraneous new lines. The rest I won't know until I get a look at the style sheets. But I can't find the style sheets. Is there a way to trace a class back to the style sheet it is defined on? The Transhumanist 04:54, 1 October 2017 (UTC)

I got rid of the blank lines for the removed items by changing one of your lines of code to this:

// nuke "li" instead of ".mw-search-result-heading"
$("li").has(".searchalttitle").remove();

I'm wondering why the double spacing (extra newline) between list items doesn't show up in the page source. In WikEd, newline characters ("\n") are invisible, but its regex feature finds/replaces them anyways. Maybe the same concept can be applied. The Transhumanist 05:40, 1 October 2017 (UTC)

I tried this to get rid of each \n, and it didn't work:

var str = $(".mw-search-results").html();
var regex = /\n/gi;
$(".mw-search-results").html(str.replace(regex, ""));

But then I tried it on \s instead, and it got rid of the extra linefeeds (along with all other white space, turning the entries to mush -- separated list items of mush! This shows that the extraneous linefeeds are potentially specifically accessible.). Any ideas? The Transhumanist 08:35, 1 October 2017 (UTC)

Tracing styles: A lot of browsers have Web development tools ("dev tools" or "inspectors" or similar) that can show what styles an element currently has, and where they come from (e.g. in Chrome you can right-click on an area you're interested in and select Inspect).

Regex: \s is equivalent to [\r\n\t\f\v ], so one of those should work. There are various regex-testing website you could use to test, analyse, explain, and experiment with regex patterns – I use https://regex101.com/ (just need to make sure the 'flavor' is javascript), but there are others out there.

I'm wondering why the double spacing (extra newline) between list items doesn't show up in the page source. – Since I didn't have the problem with Chrome on Win 7, and FF/IE had different problems to what you're describing, I think its basically down to either browser bugs (or "features") – possibly MediaWiki is serving up (or the JavaScript modification is making) non-standard/non-compliant code, and the browsers have to decide for themselves how to handle it (thus some insert phantom spaces, others don't). - Evad37 [talk] 13:21, 4 October 2017 (UTC)

Fixing the doublesspacing problem, and sorting it too

User:The Transhumanist/StripSearchInWikicode.js – the recent script you helped me on, which strips WP search results down to a bare list of links, and inserts wikilink formatting for ease of insertion of those links into lists. This is useful for gathering links for outlines. It still has the interlaced CR/LFs problem. Aside from that, I'd like this script to sort its results. So, if you know how, or know someone who knows how, please let me know.

The Transhumanist, I've had a thought on how to fix the stripsearch script: What it should do is make an array containing the search result titles - which can be sorted and otherwise manipulated using standard array methods - and then remove all the search result stuff, and rebuild the links from the array in the format you want. jQuery's .map() or .get() functions should be able to make the array. - Evad37 [talk] 00:34, 27 October 2017 (UTC)

Thank you for the guidance. How would you "rebuld the links from the array"? The Transhumanist 02:19, 27 October 2017 (UTC)

You can make links from page titles using code like I've got in User:Evad37/extra.js's makeLink function. But in your case you need to also surround the link with *[[ and ]], and have the whole thing within a block tag like <div> or <p>. Do that for each item in the array, and then you can add them all to (or next to) an element on the page using a jQuery method like .before(), .after(), .prepend(), or .append(), each of which can take an array as the input. - Evad37 [talk] 02:40, 27 October 2017 (UTC)

Adding a filter to StripSearchSorted.js

Originally posted to Evad37's talk page:

There's a really annoying design flaw in WP's search's intitle feature. Common words like "of" are ignored, even though they are included within a quoted phrase. So, intitle:"of Boston" is interpreted as just intitle:Boston. And the search results are filled with non-matching results. To make matters worse, the search results include matches of the phrase in the contents of pages, watering the results down even more to inlcude pages that don't even have "Boston" in the title. What I need is for results to strictly match the term provided after "intitle:".

For StripSearchSorted.js, you wrote a long sequence of chained methods (which I modified ever so slightly):

// Replace the search results by hiding the original results and use .after to insert a modified version of those results
            $('ul.mw-search-results').hide().after(
                $('<div id="Stripped"></div>').append(
                    $('ul.mw-search-results')
                    .children()
                    .map( function() {
                        return $(this).find('a').text();
                    })
                    .get()
                    .sort()
                    .map( function(title) {
                        return $('<div></div>').append(
                            '* [[',
                            $('<a>').attr({
                                'href':'https://en.wikipedia.org/wiki/'+mw.util.wikiUrlencode(title),
                                'target':'_blank'
                            }).text(title),
                            ']]'
                        );
                    })
                )   
            );

Is it possible to continue adding to this chain in order to filter the array down to elements that only include the intitle search string?

Assume we've put the search string into a variable, say var intitleString;

After the closing parenthesis (included below), the .filter chain continuation might look something like this:

).filter(function () { return this.
})

The problem is, I don't know what to put after "this." to match intitleString. I know regex generally speaking, but I don't know how to include it in a chain, or how to match a variable with it.

By the way, would this nuke the array if intitle wasn't specified in the search? Can an if control structure be put in a chain? (Like: If "intitle" is in the title, do this). The Transhumanist 12:05, 7 December 2017 (UTC)

Filtering is possible, but it's easier to do the filtering before the .map(), because at that stage you have a plain array of strings (each of which is a title), rather than an array of jQuery objects (which you have to drill down into to get the title string). When filtering on a plain javascript array, don't use this (that only works with jQuery objects) – the basic syntax is

newArray = oldArray.filter(function(arrayElement) {
    // do stuff, and return true (or a truthy value) to keep the array element,
    // or return false (or a falsey value) to remove the array element
});

To check if a string contains a test string, you can do string.indexOf(testString), which returns -1 if not found, or a number of where it is found. To convert to a true/false value; you just do string.indexOf(testString) !== -1. That's for case-sensitive results, and dosen't care about word boundaries. To do more advanced matching, you have to make a regex object, and then test for a match using regex.test(string), which returns true or false accordingly.

So putting it all together, elswehere in your code you make your regex pattern var intitlePatt, then

// ... same as code block above ...
                    .get()
                    .filter(function(title) {
                        return intitlePatt.test(title);
                    })
                    .sort()
// ... same as code block above ...

To stop things blowing up, you just have to make sure everything passes the filter when there's no intitle: in the search, i.e. set intitleString = '' or intitlePatt = /./ for that case. You can't really have control structures in a chain – you would have to store the intermediate value of the chain in a variable, then put in the control structure, and resume the chain from the intermediate variable. Like

var intermidateFoo = $(foo).bar().baz();
if (condition1) {
   intermidateFoo.qux().foobar().barbaz();
} else {
   intermidateFoo.barbaz();
}

- Evad37 [talk] 02:34, 8 December 2017 (UTC)

So, let me see if I got this straight...

You store the search's intitle value in a variable, and if there isn't one, the variable's value would just be null.

Then, in the chain, filter out non-matching entries. If the variable has a null value, meaning that intitle wasn't included in the search, all entries would match.

Is that correct? The Transhumanist 21:50, 10 December 2017 (UTC)

Not quite... null doesn't match anything, so no entries would match. To get all entries to match, you either have to set the variable to something that does actually match any entry (intitleString = '' or intitlePatt = /./ depending on whether you use indexOf or regex matching inside the filter); or else have an explicit check inside the filter which will just return true if the variable is null. - Evad37 [talk] 03:02, 11 December 2017 (UTC)

So, there is no way to match null in regex? So you can't match null or whatever the string is, using the pipe character? The Transhumanist 04:26, 11 December 2017 (UTC)

If you want to check if a variable is null or undefined, just do someVar == null (gives true if someVar is null/undefined, false otherwise). You can combine this with other logical tests using || , && , ! as usual. - Evad37 [talk] 04:37, 11 December 2017 (UTC)

Adding TrueMatch to StripSearchSorted

Originally posted to Evad37's talk page

I'm in the process of trying to fix the intitle bug in Wikipedia's search, by providing the solution as a function within StripSearchSorted.js.

The intitle bug is that when you enter a search phrase in WP's search box with a common word (like this: intitle:"in Germany"), the titles in the search results don't match.

I'm almost done, but I can't figure out how to get :contains to accept a variable:

        function TrueMatch() {
            // The purpose of this function is to filter out non-matches

            // Activation filter:
            // Run this function only if 'intitle:"' is in the page title
            // Notice the lone " after intitle:
            if (document.title.indexOf('intitle:"') != -1) {

                // Body of function
                // Create variable with page title
                var docTitle = document.title;

                // Display on screen for checking
                //alert ( docTitle );

                // Extract the intitle search string from the html page title
                // We want the part between the quotation marks
                var regexIntitle = new RegExp('intitle:"(.+?)(")(.*)','i');
                var intitle;
                intitle = docTitle.replace(regexIntitle,"$1");
                //alert ( intitle );
               
                // Filter out search results that do not match
                $( "li" ).not( 'li:contains(" + intitle + ")' ).remove();               
            }
        }

It works fine up until that last line. I want to remove all li elements that do not contain the text in the intitle variable. The Transhumanist 07:51, 19 January 2018 (UTC)

Instead of passing a single string for the selector, you need to build a string up around the variable:

$( "li" ).not( 'li:contains("' + intitle + '")' ).remove();

The 'li:contains("' + intitle + '")' gets processed first, and the result is passed through to .not(). Or if you wanted to be a bit more explicit, you could do something like

var intitle_selector = 'li:contains("' + intitle + '")';
$( "li" ).not( intitle_selector ).remove();

- Evad37 [talk] 13:59, 19 January 2018 (UTC)

I tried both methods you posted above. I tested it on intitle:"of Australia". The script runs, and strips out the details as it is supposed to. And it is sorting the results. But it isn't removing the non-matches. It's like it's matching everything, and therefore removing nothing. (When it matches nothing, like in my version above, it removes everything, leaving the results blank). I reactivated the alerts, and those show up fine. It's still the last line that isn't working. When you replace it with $( "li" ).remove();, it removes all results. The Transhumanist 02:16, 20 January 2018 (UTC)

Testing further on the "of Australia" search...

$( "li").not('li:contains( "of" )').remove(); resulted in blank results (ie, none).

$( "li").not('li:contains( of )').remove(); resulted in no matches (ie results unaffected).

So, it looks like the first one is matching nothing, causing all li elements to be removed, while the second one is matching everything, causing no li elements to be removed. The Transhumanist 03:11, 20 January 2018 (UTC)

I stared at the current page source, and discovered spans with the class "searchmatch", the contents of which appear to have been causing false matches. So, I blasted those with:

   // First, strip out the searchmatch class elements (they match).
  $( 'li').find( '.searchmatch').remove();

Then, with the above line in place, I tested your solution again, but it didn't work:

$( "li" ).not( 'li:contains("' + intitle + '")' ).remove();

The results turned up blank, which means it removed everything.

Ironically, doing the opposite works:

$( 'li:contains("' + intitle + '")').remove();

Unfortunately, this removes precisely the entries the user wants to keep. The Transhumanist 08:26, 20 January 2018 (UTC)

I think we need to be more specific, and target the main link of each result - since the value of intitle will be somewhere within the li, just not neccesarily in the title. Plus we can limit the searching of lis to just the search results, rather than the whole page:

// Mark true results with a class
$('.mw-search-results').find('li').has( 'div > a:contains("' + intitle + '")' ).addClass('truematch');
// Remove other results
$('.mw-search-results').find('li').not('.truematch').remove();

Which basically means: In the mw-search-results, find the lis which have a div that itself has (as a direct child element) an a that contains the text intitle, and add the class truematch to those lis. Then, in the mw-search-results, find the lis which do not have the class truematch, and remove them. - Evad37 [talk] 09:25, 20 January 2018 (UTC)

That did the trick. It works beautifully. Thank you.

This leads the way to the development of two related programs:

1. StripSearchFilter.js – will allow the user to enter additional search terms to filter down the results, including a term to keep or a term to discard. Can use it multiple times to further refine the result.
2. SearchSuite.js – will put selected features on their own switches so they can be turned on and off. Like the details stripping, and the inserted wikicode. It will also include the search filter feature mentioned above.

I'll keep you posted. The Transhumanist 11:28, 20 January 2018 (UTC)