Python Client

Summary

The following examples assume you have downloaded the API client into your working directory.

Basic usage

If you already have created an index you'll need to use your index name to instantiate the client:

from indextank.client import ApiClient

api = ApiClient('<YOUR API URL>')

index = api.get_index('<YOUR INDEX NAME HERE>')
					

Once you have an instance of the client all you need is the content you want to index. The simplest way to add a document is sending that content in a single field called "text":

id = '<YOUR DOCUMENT ID>'
text = '<THE TEXTUAL CONTENT>'

index.add_document(id, { 'text': text })
					

That's it, you have indexed a document.

You can now search the index for any indexed document by simply providing the search query:

query = '<YOUR QUERY STRING>'
results = index.search(query)

print '%d results' % results['matches']
for doc in results['results']:
    print 'docid: %s' % doc['docid']
					

As you can see, the results are the document ids you provided when indexing the documents. You use them to retrieve the actual documents from your DB to render the result page.

You can get richer results using the fetch fields and snippet fields options:

query = '<YOUR QUERY STRING>'
results = index.search(query,
                          fetch_fields=['title','timestamp'],
                          snippet_fields=['text'])

print '%d results' % results['matches']
for doc in results['results']:
    docid = doc['docid']
    title = doc['title']
    timestamp = doc['timestamp']
    text = doc['snippet_text']
    print "<a href='%s'>%s</a><p>%s</p>" % (docid,title,text)
					

Deleting an indexed document is also very easy:

id = "<YOUR DOCUMENT ID>"

index.delete_document(id)
					

Additional fields

When you index a document you can define different fields by adding more elements to the document object:

#### INDEX MULTIPLE FIELDS
index.add_document(id, { 'text': text,
                         'title': title,
                         'author': author })
					

By default, searches will only look at the "text" field, but you can define special searches that look at other fields by prefixing a search term with the field name. The following example filters results including only the user's content.

#### FILTER TO USER'S CONTENT
index.search('%s author:%s' % (query,user))
					

There's also a special field named "timestamp" that is expected to contain the publication date of the content in seconds since unix epoch (1/1/1970 00:00 UTC). If none is provided, the time of indexation will be used by default.

index.add(id, { 'text': text,
                'timestamp': time.mktime(time.gmtime())
              })
					

Document variables

When you index a document you can define special floating point fields that can be used in the results scoring. The following example can be used in an index that allows 3 or more variables:

#### INDEX DOCUMENT WITH VARIABLES
doc = { 'text': text }
variables = { 0: rating, 1: reputation, 2: visits }

index.add_document(id, doc, variables=variables)
					

You can also update a document variables without having to re-send the entire document. This is much faster and should always be used if no changes were made to the document itself.

#### UPDATE DOCUMENT VARIABLES
variables = { 0: new_rating, 1: new_reputation, 2: new_visits }
index.update_variables(id, variables=variables)
					

Scoring functions

To use the variables in your searches you'll need to define scoring functions. These functions can be defined in your dashboard by clicking "Manage" in your index details or directly through the API client.

# FUNCTION 0 : sorts by most recent
index.add_function(0, "-age")

# FUNCTION 1 : standard textual relevance
index.add_function(1, "relevance")

# FUNCTION 2 : sorts by rating
index.add_function(2, "doc.var[0]")

# FUNCTION 3 : sorts by reputation
index.add_function(3, "d[1]")

# FUNCTION 4 : advanced function
index.add_function(4, "log(d[0]) - age/50000")
					

Read the function definition syntax for help on how to write functions.

If you don't define any functions, you will get the default function 0 which sorts by timestamp (most recent first). By default, searches will use the function 0. You can specify a different function when searching by using the option scoring_function

index.search(query, scoring_function=2)
					

Query variables and Geolocation

Besides the document variables, in the scoring functions you can refer to query variables. These variables are defined at query time and can be different for each query.

A common use-case for query variables is geolocation, where you use two variables for latitude and longitude both in the documents and in the query, and use a distance function to sort by proximity to the user. For this example will assume that every document stores its position in variables 0 and 1 representing latitude and longitude respectively.

Defining a proximity scoring function:

# FUNCTION 5 : inverse distance calculated in miles
index.add_function(5, "-mi(d[0], d[1], q[0], q[1])")
					

Searching from a user's position:

variables = { 0: latitude, 1: longitude }

index.search(query,
             scoring_function=5,
             variables=variables)
					

Faceting

Besides being able to define numeric variables on a document you can tag documents with category values. Each category is defined by string, and its values are alse defined by strings. So for instance, you can define a category named "articleType" and its values can be "camera", "laptop", etc... You can have another category called "priceRange" and its values can be "$0 to $49", "$50 to $100", etc...

Documents can be tagged with a single value for each category, so if a document is in the "$0 to $49" priceRange it can't be in any other, and retagging over the same category results in overwriting the value.

You can tag several categories at once like this:

categories = {
                  'priceRange': '$0 to $299',
                  'articleType': 'camera'
             }
index.update_categories(docid, categories)
					

When searching, you will get an attribute in the results called "facets", and it will contain a dictionary with categories for keys. For each category the value will be another map, with category value as key and occurences as value. So for instance:

{
    'matches': 8,
    'results': [ {'docid': 'doc1'}, ... ],
    'facets': {
        'articleType': {
            'camera': 5,
            'laptop': 3
        },
        'priceRange': {
            '$0 to $299': 4,
            '$300 to $599': 4
        }
    }
}
					

Means that from the matches, 5 are of the "camera" articleType and 3 are "laptop". Also, 4 of them all are in the "$0 to $299" priceRange, and 4 on the "$300 to $599".

Then, you can also filter a query by restricting it to a particular set of category/values. For instance the following will only return results that are of the "camera" articleType and also are either in th "$0 to $299" or "$300 to $599" price range.

index.search(query,
             category_filters={
                'priceRange': ['$0 to $299', '$300 to $599'],
                'articleType': ['camera']
             })
					

Range queries

Document variables and scoring functions can also be used to filter your query results. When performing a search it is possible to add variable and function filters. This will allow you to only retrieve, in the search results, documents whose variable values are within a specific range (e.g.: posts that have more than 10 votes but less than a 100). Or only return documents for the which a certain scoring function returns values within a specific range.

You can specify more than one range for each variable or function (the value must be within at least ONE range) filter, and you can use as many filters as you want in every search (all filters must be met):

"""
  Ranges are specified by string with the format:
   [BOTTOM_LIMIT,TOP_LIMIT]
  Both top and bottom can be replaced by 'None'
  to indicate they should be ignored.

  In this sample, the results will only include documents
  whose variable #0 value is between 5 and 10 or between 15
  and 20, and variable #1 value is less than or equal to 3

"""
index.search(query,
             docvar_filters={
                0: [[5,10], [15,20]],
                1: [[None,3]]
             })

"""
  The same applies to functions
"""
index.search(query,
             function_filters={
                1: [[0.5,None]]
             })
					

Batch indexing

When populating an index for the first time or when a batch task for adding documents makes sense, you can use the batch indexing call.

When using batch indexing, you can add a large batch of documents to the Index with just one call. There is a limit to how many documents you can add in a single call, though. This limit is not related to the number of documents, but to the total size of the resulting HTTP request, which should be less than 1MB.

Making a batch indexing call reduces the number of request needed (reducing the latency introduced by round-trips) and increases the maximum throughput which can be very useful when initially loading a large index.

The indexing of individual documents may fail and your code should handle that and retry indexing them. If there are formal errors in the request, the entire batch will be rejected with an exception.

documents = []
documents.append({ 'docid': 'doc1', 'fields': { 'text': text1 } })
documents.append({ 'docid': 'doc2', 'fields': { 'text': text2 } })
documents.append({ 'docid': 'doc3', 'fields': { 'text': text2 }, 'variables': { 0 : 1.5 } })
documents.append({ 'docid': 'doc4', 'fields': { 'text': text2 }, 'variables': { 0 : 1.5 }, 'categories': { 'Price': '0 to 100' } })

response = index.add_documents(documents)
					

The response will be an array with the same length as the sent batch. Each element will be a map with the key "added" denoting whether the document in this position of the batch was successfully added to the index. If it's false, an error message will also be in the map with the key "error".

failed_documents = [documents[i] for i in xrange(len(response)) if not response[i]['added']]
					

Bulk Delete

With this method, you can delete a batch of documents (reducing the latency introduced by round-trips). The total size of the resulting HTTP request should be less than 1MB.

The deletion of individual documents may fail and your code should handle that and retry deleting them. If there are formal errors in the request, the entire batch will be rejected with an exception.

docids = ['doc1', 'doc2', 'doc3', 'doc4']
response = index.delete_documents(docids)
					

The response will be an array with the same length as the sent batch. Each element will be a map with the key "deleted" denoting whether the document with the id in this position of the batch was successfully deleted from the index. If it's false, an error message will also be in the map with the key "error".

failed_documents = [docids[i] for i in xrange(len(response)) if not response[i]['deleted']]
					

Delete by Search

With this method, you can delete a batch of documents that match a particular search query. You can use many of the same arguments applied to a normal search - start (which will preserve the results found before the value of start), scoring function, category filters, variables, and docvar filters.

query = '<YOUR QUERY STRING>'
index.delete_by_search(query)
					

Index management

You can create and delete indexes directly with the API client. These methods are equivalent to their corresponding actions in the dashboard. Keep in mind that index creation may take a few seconds.

The create_index methods will return the new index's client:

from indextank.client import ApiClient
import time

api = ApiClient('<YOUR API URL>')

# this parameter allows you to create indexes with public search enabled.
# default is False.
public_search_enabled = False

index = api.create_index('test_name', public_search_enabled)

while not index.has_started():
    time.sleep(0.5)
					

The delete_index method completely removes the index represented by the object.

api.delete_index('test_name')
					

Real Time Web Analytics