Building a 'search as you type' engine for a Python Flask application using Javascript

In this blog post, I will describe how to build a 'search as you type' feature for a Flask application together with a Javascript frontend. 'Search as you type' means that the search results are displayed while the user is typing into the search field. Google has now included this feature in their search engine.

In my case, this search feature is used to search job advertisements for jobs in academic research and you can see it here in action.

We start with the HTML markup. All we are going to do here is to define a search field, where the user can type something, and a target field, where we are going to show the result:

<fieldset>
     <div class="form-group">
         <div class="col-xs-10 col-sm-8 col-md-6 col-lg-6">
             <input type="text" class="form-control" id="job_query" 
             placeholder="search term" value="" oninput="update_job_table()">
        </div>
    </div>
</fieldset>
<input id="timestamp" value="0" hidden>

<table class="table table-striped">
    <thead>
        <!-- whatever -->
    </thead>
    <tbody id="job_market_table">
    </tbody>
</table>

You can see that if the user types something in the search field the javascript function update_job_table() is called. In this function, we now have to take the current search string, send it to our Flask application, perform a quick search in the database and return the results for this search term.

To send the current search string to Flask we use Javascript. We are going to show a spinner symbol while the user is waiting to get the results back, so I include the spinner.js package. This package has only about 4kb when minimized, so does not add much overhead.

You can set up the spinner by defining the following options:

var opts = {
    lines: 9,    // The number of lines in the spinner
    length: 10,  // The length of each line
    width: 3,    // The line thickness
    radius: 6,   // The radius of the inner circle
    corners: 1,  // Corner roundness (0..1)
    rotate: 58,  // The rotation offset
    direction: 1, // 1: clockwise, -1: counterclockwise
    color: '#000000 ', // #rgb or #rrggbb or array of colors
    speed: 0.9,  // Rounds per second
    trail: 100,  // Afterglow percentage
    shadow: false, // Whether to render a shadow
    hwaccel: false, // Whether to use hardware acceleration
    className: 'spinner', // The CSS class to assign to the spinner
    zIndex: 99,  // the modals have zIndex 100
    top: 'auto', // Top position relative to parent
    left: '50%'  // Left position relative to parent
};

Now we can write the Javascript which sends the search term to the Flask app using Ajax:

function update_job_table() {
    var job_table = document.getElementById('job_market_table');
    job_table.innerHTML = "<td colspan='5'>Updating table</td>";
    var spinner = new Spinner(opts).spin(job_table);

    var search_term = document.getElementById('job_query').value;
    doc = { 'search_term': search_term };
    var timestamp = +new Date();
    doc['timestamp'] = timestamp;
    $.ajax({
        url: $SCRIPT_ROOT + '/search_job_market',
        type: "POST",   
        async: true,
        cache: false,
        dataType: 'json',
        contentType: "application/json; charset=utf-8",
        data: JSON.stringify(doc, null, '\t'),
        success: function(data) {
            if(data.error){
                console.log("Error = ", data.error);
            }
            else{
                var results_timestamp = document.getElementById('timestamp');
                if(data.timestamp > parseInt(results_timestamp.value)){
                    spinner.stop();
                    results_timestamp.value = data.timestamp;
                    job_table.innerHTML = data.html;
                }
            }
        }
    });
}

So we just access the search term through the id and send it with a post request to the Flask /search_job_market view as a JSON object. The $SCRIPT_ROOT variable is set to the home path of the application.

Now we have to handle this request within the /search_job_market'  view.

@app.route('/search_job_market', methods=['POST'])
def search_job_market():
    ''' 
    We get here if the user types into the job market search field
    '''
    if request.method == 'POST': 
        html = get_job_market_table_html(request.json)
        return json.dumps({'success': 'search successful', 'html': html, 
               'timestamp': request.json['timestamp']}),\
               200, {'ContentType':'application/json'}

The function get_job_market_table_html() retrieves the necessary information from the database and renders the HTML. My setup uses an Elasticsearch database, but of course, any database system can be plugged in here. You should probably make sure that your setup is able to return some meaningful result to the user in a reasonable time.

You might have noticed that I pass a timestamp through to all requests. The reason for that is to ensure that we only update the user view with new results. Imagine that the user types 'science' in the search field. This will trigger searches for 's', 'sc', 'sci', 'scie', 'scien', 'scienc' and 'science'. It is possible that the result for 's' takes longer and therefore gets sent back to the page after 'sc' has been displayed. Using the timestamp, our setup just ignores the older result.

I posted the entire code on GitHub. Feel free to leave comments or questions below.
cheers
Florian

Lazy load images with Javascript depending on the user view window

In this blog post, I will describe how I added lazy load of images to my website. If you have a page with many images, it is very useful to not load all images at page load but to load the images on demand. This is especially useful for people with bad internet connections or people who visit your page on mobile devices.

The first step is to load a spinner symbol instead of the actual image, but to write the image in the data-src property so that the HTML looks something like this:

<img class="lazy_load_image" src="/static/spinner.gif" 
data-src="{{ path_to_image }}" alt="{{ image_name }}" 
style="width:300px;hight:400px;">

With this markup, only the spinner image will be loaded on page load and you should, of course, make sure that this is a small image, but there are many examples you can find online.

Now after page load, we want to load the images and we do this with Javascript by adding an event listener

// This code lazy loads the images in the viewport after the page load
window.addEventListener('load', function(){
    load_lazy();
}, false)

The load_lazy() function just writes the data-src property of the HTM tag above into the src property for all images which are in the viewport:

// This function loads all images in the viewport
function load_lazy(){
    // we expand the viewport by 200px
    var images = $('.lazy_load_image').withinviewport({top: -200, bottom: -200});
    for(var i = 0; i < images.length; i++){
        var image_src = images[i].getAttribute('data-src');
        if(image_src){
            images[i].setAttribute('src', image_src);
            images[i].className -= 'lazy_load_image';
        }
    }
}

For this to work you have to download and include withinviewport.js (bower install within-viewport).
Here I extended the viewport by 200px, just to make sure that the user does not see a spinner forever if less than 50% of the image is in the viewport.

The last step is to make sure that images get loaded if the user scrolls down, which we can do by adding

// This code lazy loads the images in the viewport after scrolling
$(window).scroll(function(){
    load_lazy();
})

This function gets called whenever the user scrolls, and if a new image has entered the user view, it will be loaded by the load_lazy() function.

Hope that was useful. I posted the entire code on GitHub.
cheers
Florian

Combine and minify CSS/JS files with the minify API to minimize page loading time

Page loading time is a major factor in customer satisfaction with a website as many studies have shown, e.g. read this blog post. One step towards fast loading time is to minimize the data which need to be transferred before the page can be shown to the user.

In this blog post, I will talk about my approach to combine and minimize all CSS and javascript files. Combining the files reduces the number of requests and compressing reduces the number of bytes required to send to the customer. There are build tools like  GRUNT and Gulp, which allow you to do this in a more structured way, but here I will just discuss my own quick fix.

I wrote a small python script for this task which makes use of the minify API to process the CSS and javascript files. This is how it looks:

def compressor(api_url, output_name, filenames):
    '''
    Here we use the minify API 
    js files need to be sent to api_url = https://javascript-minifier.com/raw
    css files need to be sent to api_url = https://cssminifier.com/raw
    '''
    code = []
    total_cost = 0
    for fn in filenames:
        if fn.startswith('http://') or fn.startswith('https://'):
            response = requests.get(fn)
            if response.status_code == 200:
                code.append( response.text )
            else:
                print('ERROR: "%s" is not a valid url! Exit with "
                      status code %d' % (fn, response.status_code))
                return False
        else:
            if not os.path.isfile(fn):
                print('ERROR: "%s" is not a valid file!' % fn)
                return False
            code.append( open(fn).read().decode('utf-8') )
        cost = len(code[-1]) / 1024.0
        total_cost += cost
        print("added %s with (%.2fK)" % (fn, cost))

    payload = {'input': u' '.join(code)}
    response = requests.post(api_url, payload)
    if response.status_code == 200:
        outfile = open(output_name, 'w')
        outfile.write(response.text.encode('utf-8'))
        outfile.close()

        print('-' * 50)
        print('>> output: %s (%.2fK) from (%.2fK)' %
              (output_name, len(response.text)/1024.0, total_cost))
        return 
    else:
        print('ERROR: "%s" is not a valid url! Exit with "
              status code %d' % (fn, response.status_code))
        return False

This function gets the API URL, the output file name and a list of input files. It will concatenate all input files and send the code to the minify API, where it is processed and sent back. If you provide a URL instead of a file, it will download the data for you so you can build your js file entirely with CDN links.

So what does the minify API do? Let's take an example. I have a function in my javascript which changes the style display status of an HTML element like this:

// Every question mark on the site is handled through this function
function toggle_help(target_id) {
    var target_object = document.getElementById(target_id)
    if( target_object.style.display == 'none' ){
        target_object.style.display = 'block';
    }
    else{
        target_object.style.display = 'none';
    }
} 

After feeding it into the minify API it looks like this:

function toggle_help(e){
    var t = document.getElementById(e);
    "none"==t.style.display?t.style.display="block":t.style.display="none"
}

So minify removed all comments and unnecessary whitespace. It also renames the variables to one letter names. The purpose of this is of course just to minimize the number of bytes.

It is certainly not a good idea to work with the minified code. It's an unreadable mess (in the example above I introduced some whitespaces and indentation, to make it more readable). So my approach with the function above is to write my code nicely separated in many files logically separated by functionality and only before deployment I put them all together.

To process js files you have to call the function with

api_url = https://javascript-minifier.com/raw

while for javascript files, you have to call the function with

api_url = https://cssminifier.com/raw

I put this code on GitHub. Let me know if you have any questions/comments below.
cheers
Florian