How to output response of type atom/xml feed (from arxiv call) into Jquery DataTable?
I have the datatable working for a simple json from Ajax call to flask server example.
When i try to do it with the xml from an arxiv api response, i cant seem to get it to display in the datatable (though i can just print the raw xml using <pre lang="xml" > or json).
I also tried to convert to json first via python dictionary, but still couldnt get it formatted into datatable as im unsure how to access the properties properly in the Ajax call when theyre deeper than the first level as in the basic example linked.
The HTML in template:
<table id="arxivtable" class="display" style="width:100%">
<thead>
<tr>
<th>title</th>
<th>id</th>
<th>link</th>
<th>author</th>
<th>published</th>
</tr>
</thead>
</table>
I tried via xml :
$('#arxivtable').DataTable({
"ajax": {
// "url": "static/objects2.txt", // This works for the static file
"url": "/get_arxivpapers", // This now works too thanks to @kthorngren
"dataType": "xml",
"type":"GET",
"dataSrc": "{{name}}",
"contentType":"application/atom+xml"
},
"columns": [
{"data": "title"},
{
"data": "link",
"render": function(data, type, row, meta){
if(type === 'display'){
data = '<a href="' + data + '">' + data + '</a>';
}
return data;
}
},
{ "data": "id" },
{ "data": "link" },
{ "data": "author" },
{ "data": "journal" },
{ "data": "published" },
{ "data": "summary" }
]
});
JSON from AJAX call:
{
"feed": {
"@xmlns": "http://www.w3.org/2005/Atom",
"link": {
"@href": "http://arxiv.org/api/query?search_query%3Dall%3Aeinstein%26id_list%3D%26start%3D0%26max_results%3D2",
"@rel": "self",
"@type": "application/atom+xml"
},
"title": {
"@type": "html",
"#text": "ArXiv Query: search_query=all:einstein&id_list=&start=0&max_results=2"
},
"id": "http://arxiv.org/api/vehKAQR+bheXtHwJw3qx/OG/XXw",
"updated": "2022-06-14T00:00:00-04:00",
"opensearch:totalResults": {
"@xmlns:opensearch": "http://a9.com/-/spec/opensearch/1.1/",
"#text": "36970"
},
"opensearch:startIndex": {
"@xmlns:opensearch": "http://a9.com/-/spec/opensearch/1.1/",
"#text": "0"
},
"opensearch:itemsPerPage": {
"@xmlns:opensearch": "http://a9.com/-/spec/opensearch/1.1/",
"#text": "2"
},
"entry": [
{
"id": "http://arxiv.org/abs/1801.05533v2",
"updated": "2018-11-22T14:04:43Z",
"published": "2018-01-17T03:05:51Z",
"title": "Einstein-Weyl structures on almost cosymplectic manifolds",
"summary": "",
"author": {
"name": "Xiaomin Chen"
},
"arxiv:comment": {
"@xmlns:arxiv": "http://arxiv.org/schemas/atom",
"#text": "accepted by Periodica Mathematica Hungarica, 14 pages, no figures"
},
"link": [
{
"@href": "http://arxiv.org/abs/1801.05533v2",
"@rel": "alternate",
"@type": "text/html"
},
{
"@title": "pdf",
"@href": "http://arxiv.org/pdf/1801.05533v2",
"@rel": "related",
"@type": "application/pdf"
}
],
"arxiv:primary_category": {
"@xmlns:arxiv": "http://arxiv.org/schemas/atom",
"@term": "math.DG",
"@scheme": "http://arxiv.org/schemas/atom"
},
"category": [
{
"@term": "math.DG",
"@scheme": "http://arxiv.org/schemas/atom"
},
{
"@term": "53D10, 53D15",
"@scheme": "http://arxiv.org/schemas/atom"
}
]
},
{
"id": "http://arxiv.org/abs/0802.2137v3",
"updated": "2008-04-01T04:36:21Z",
"published": "2008-02-15T04:40:56Z",
"title": "",
"summary": ".",
"author": {
"name": ""
},
"arxiv:comment": {
"@xmlns:arxiv": "http://arxiv.org/schemas/atom",
"#text": "18 pages, added Theorem 5"
},
"link": [
{
"@href": "http://arxiv.org/abs/0802.2137v3",
"@rel": "alternate",
"@type": "text/html"
},
{
"@title": "pdf",
"@href": "http://arxiv.org/pdf/0802.2137v3",
"@rel": "related",
"@type": "application/pdf"
}
],
"arxiv:primary_category": {
"@xmlns:arxiv": "http://arxiv.org/schemas/atom",
"@term": "math.DG",
"@scheme": "http://arxiv.org/schemas/atom"
},
"category": [
{
"@term": "math.DG",
"@scheme": "http://arxiv.org/schemas/atom"
},
{
"@term": "53C30; 53C25",
"@scheme": "http://arxiv.org/schemas/atom"
}
]
}
]
}
}
Or the original atom/xml:
<feed xmlns="http://www.w3.org/2005/Atom">
<link href="http://arxiv.org/api/query?search_query%3Dall%3Aeinstein%26id_list%3D%26start%3D0%26max_results%3D2" rel="self" type="application/atom+xml">
<title type="html">ArXiv Query: search_query=all:einstein&id_list=&start=0&max_results=2</title>
<id>http://arxiv.org/api/vehKAQR+bheXtHwJw3qx/OG/XXw</id>
<updated>2022-06-14T00:00:00-04:00</updated>
<opensearch:totalresults xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">36970</opensearch:totalresults>
<opensearch:startindex xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">0</opensearch:startindex>
<opensearch:itemsperpage xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/">2</opensearch:itemsperpage>
<entry>
<id>http://arxiv.org/abs/1801.05533v2</id>
<updated>2018-11-22T14:04:43Z</updated>
<published>2018-01-17T03:05:51Z</published>
<title></title>
<summary>
</summary>
<author>
<name></name>
</author>
<arxiv:comment xmlns:arxiv="http://arxiv.org/schemas/atom">accepted by Periodica Mathematica Hungarica, 14 pages, no figures</arxiv:comment>
<link href="http://arxiv.org/abs/1801.05533v2" rel="alternate" type="text/html">
<link title="pdf" href="http://arxiv.org/pdf/1801.05533v2" rel="related" type="application/pdf">
<arxiv:primary_category xmlns:arxiv="http://arxiv.org/schemas/atom" term="math.DG" scheme="http://arxiv.org/schemas/atom">
<category term="math.DG" scheme="http://arxiv.org/schemas/atom">
<category term="53D10, 53D15" scheme="http://arxiv.org/schemas/atom">
</category></category></arxiv:primary_category></entry>
<entry>
<id>http://arxiv.org/abs/0802.2137v3</id>
<updated>2008-04-01T04:36:21Z</updated>
<published>2008-02-15T04:40:56Z</published>
<title></title>
<summary>
</summary>
<author>
<name></name>
</author>
<arxiv:comment xmlns:arxiv="http://arxiv.org/schemas/atom"></arxiv:comment>
<link href="http://arxiv.org/abs/0802.2137v3" rel="alternate" type="text/html">
<link title="pdf" href="http://arxiv.org/pdf/0802.2137v3" rel="related" type="application/pdf">
<arxiv:primary_category xmlns:arxiv="http://arxiv.org/schemas/atom" term="math.DG" scheme="http://arxiv.org/schemas/atom">
<category term="math.DG" scheme="http://arxiv.org/schemas/atom">
<category term="53C30; 53C25" scheme="http://arxiv.org/schemas/atom">
</category></category></arxiv:primary_category></entry>
</feed>
The End Point:
@app.route('/get_arxivpapers')
def getArxivPapers(name="einstein"):
max_results = 2
searchterm = name.replace("_", " ")
url = 'http://export.arxiv.org/api/query?search_query=all:' + searchterm + '&start=0&' + 'max_results='+ str(max_results)
data = urllib.request.urlopen(url)
# data_dict = xmltodict.parse(data)
# json_data = json.dumps(data_dict)
# print(json_data)
# return jsonify(json_data)
return data.read().decode('utf-8')
I will use your JSON source data instead of the XML, since that is easier to handle in DataTables.
Here is a basic demo, to start with, followed by some explanatory notes:
<!doctype html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Demo</title>
<script src="https://code.jquery.com/jquery-3.5.1.min.js"></script>
<script src="https://cdn.datatables.net/1.10.21/js/jquery.dataTables.min.js"></script>
<link rel="stylesheet" type="text/css" href="https://cdn.datatables.net/1.10.21/css/jquery.dataTables.min.css">
</head>
<body>
<div style="margin: 20px;">
<table id="arxivtable" class="display" style="width:100%">
<thead>
<tr>
<th>title</th>
<th>id</th>
<th>link</th>
<th>author</th>
<th>published</th>
<th>summary</th>
</tr>
</thead>
</table>
</div>
<script type="text/javascript">
$(document).ready(function(){
$('#arxivtable').DataTable({
"ajax": {
url: "YOUR_URL_GOES_HERE",
dataSrc: "feed.entry"
},
"columns": [
{"data": "title"},
{ "data": "id" },
{ "data": "link[].@href" },
{ "data": "author.name" },
{ "data": "published" },
{ "data": "summary" }
]
});
});
</script>
</body>
</html>
Notes
1 - Because you have provided hard-coded HTML column headers, you need to make sure the number of those headers matches the number of columns defined in the DataTable. Alternatively, you can remove the HTML <thead> section and use the DataTables columns.title option.
2 - Your Ajax JSON source data contains an array [ ... ]. DataTables needs to know where this array is located in your JSON response, as part of the Ajax handling option, so that it can iterate over that array. Each element in the array will be used to create a row of HTML table data. The ajax.dataSrc option therefore needs to be set accordingly:
dataSrc: "feed.entry"
Once you have set the above Ajax JSON starting point correctly, then you can use field names for each separate column data value - as shown below.
3 - The author JSON value is actually an object:
"author": {
"name": "Xiaomin Chen"
},
Therefore you need to drill down into that to get the field you want to show in the DataTable:
{ "data": "author.name" },
4 - I removed your column renderer function to keep my initial demo simple, but it can be used to access fields and sub-fields - and concatenate strings and other values as needed (as in your example in the question).
5 - The link JSON value is actually an array of objects. For my basic demo, I just accessed the final entry in that array, and then took the href field:
{ "data": "link[].@href" },
This may not be what you want. You may want to only choose links of a certain type, or choose all links, or something different.
This is where DataTables is limited in what it can handle. It cannot display arbitrary nested JSON values of this type (not surprisingly).
In such cases, you would need to re-structure the JSON, prior to sending it to DataTables - or restructure it in a dataSrc function inside DataTables itself:
"dataSrc": function ( json ) { ...transform and return your JSON here... }
6 - I was not sure what you wanted to display for { "data": "journal" }. I did not see anything called journal in the JSON.
7 - Note that all the source JSON data outside of the feed.entry array is also not available to DataTables. DataTables can only iterate over that outer array. Anything you may also need which is not in that outer array would need to be added to the array, to be accessible to DataTables.
See also Nested object data (arrays) and Nested object data (objects) for more related notes.