Difference between revisions of "GCat Service"

From Gcube Wiki
Jump to: navigation, search
(Create (POST))
m
 
(140 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The gCube Science Catalogue Service is a RESTful web service based on the principles defined in [[GCube_Resource_Catalogue | gCube Catalogue]]
+
'''THIS WIKI PAGE IS NO LONGER MAINTAINED. YOU CAN FIND UP-TO-DATE DOCUMENTATION AT THE FOLLOWING LINKS'''
 +
 
 +
'''THE FIRST LINK CONTAINS GENERAL INFORMATION ABOUT THE SERVICE'''
 +
 
 +
https://api.d4science.org/gcat/docs/index.html
 +
 
 +
 
 +
'''THE SECOND LINK CONTAINS API DOCUMENTATION AS WELL AS AN INTERACTIVE INTERFACE PROVIDED BY SWAGGER'''
 +
 
 +
https://api.d4science.org/gcat/api-docs/index.html
 +
 
 +
 
 +
 
 +
 
 +
 
 +
gCat Service is a RESTful service enabling any client to programmatically interact with the gCube Catalogue.
 +
gCat Service uses the principles defined in [[GCat_Background | gCat Background]]
  
 
== Request ==
 
== Request ==
Line 6: Line 22:
 
=== URL ===
 
=== URL ===
  
The URL used to interact with the Science Catalogue Service is composed of two parts:
+
The URL used to interact with gCat is composed of two parts:
  
* Base Service URL (e.g. https:/catalogue.d4science.org/science-catalogue)
+
* '''Base Service URL''': '''https://api.d4science.org/catalogue'''
 
* Specific API (e.g. /organizations)
 
* Specific API (e.g. /organizations)
  
 
D4Science infrastructure uses cloud facilities allowing to replicate a service to achieve failover and load balancing.
 
D4Science infrastructure uses cloud facilities allowing to replicate a service to achieve failover and load balancing.
  
Science Catalogue Service instances can be deployed and undeployed dynamically. For such a reason the Base Service URL MUST NOT be hard-cabled in the code because it can change over time.
+
gCat instances can be deployed and undeployed dynamically. For such a reason the Base Service URL MUST NOT be hard-cabled in the code because it can change over time.
  
 
To dynamically discover the Base Service URL you can use the [https://dev.d4science.org/swagger/registry/ Registry RESTful Service]
 
To dynamically discover the Base Service URL you can use the [https://dev.d4science.org/swagger/registry/ Registry RESTful Service]
  
 
You need to discover gCore Resource having:
 
You need to discover gCore Resource having:
* '''class''' : ''DataCatalogue''
+
* '''class''' : ''DataPublishing''
* '''name''' : ''science-catalogue''
+
* '''name''' : ''gcat''
  
 
=== HTTP Headers ===
 
=== HTTP Headers ===
Line 25: Line 41:
 
==== gCube Authorization Token ====
 
==== gCube Authorization Token ====
  
Any request performed to Science Catalogue MUST contains the gCube Authorization Token.
+
Any request performed to gCat MUST contains the gCube Authorization Token.
  
 
This is done using the HTTP Header ''gcube-token''
 
This is done using the HTTP Header ''gcube-token''
Line 67: Line 83:
 
* '''''404 Not Found''''' used to indicate that the requested instance does not exists [https://tools.ietf.org/html/rfc7231#section-6.5.4 https://tools.ietf.org/html/rfc7231#section-6.5.4];
 
* '''''404 Not Found''''' used to indicate that the requested instance does not exists [https://tools.ietf.org/html/rfc7231#section-6.5.4 https://tools.ietf.org/html/rfc7231#section-6.5.4];
 
* '''''405 Method Not Allowed''''' the used HTTP method is not supported for the requested URL [https://tools.ietf.org/html/rfc7231#section-6.5.5 https://tools.ietf.org/html/rfc7231#section-6.5.5]. <br />The response contains the ''Allow'' HTTP Header indicating the supported HTTP method for such URL [https://tools.ietf.org/html/rfc7231#section-7.4.1 https://tools.ietf.org/html/rfc7231#section-7.4.1];
 
* '''''405 Method Not Allowed''''' the used HTTP method is not supported for the requested URL [https://tools.ietf.org/html/rfc7231#section-6.5.5 https://tools.ietf.org/html/rfc7231#section-6.5.5]. <br />The response contains the ''Allow'' HTTP Header indicating the supported HTTP method for such URL [https://tools.ietf.org/html/rfc7231#section-7.4.1 https://tools.ietf.org/html/rfc7231#section-7.4.1];
 +
* '''''409 Conflict'''''  the request could not be completed due to a conflict with the current state of the target resource (e.g. the name of the resource already exists). [https://tools.ietf.org/html/rfc7231#section-6.5.8 https://tools.ietf.org/html/rfc7231#section-6.5.8].];
 
* '''''500 Internal Server Error''''' indicate a server failure. [https://tools.ietf.org/html/rfc7231#section-6.6.1 https://tools.ietf.org/html/rfc7231#section-6.6.1].
 
* '''''500 Internal Server Error''''' indicate a server failure. [https://tools.ietf.org/html/rfc7231#section-6.6.1 https://tools.ietf.org/html/rfc7231#section-6.6.1].
  
Line 81: Line 98:
 
=== HTTP Methods ===
 
=== HTTP Methods ===
  
To be RESTful compliant Science Catalogue uses standard HTTP Methods to perform a listing of collections and CRUD (Create Read Update Delete) operations on instances.
+
To be RESTful compliant gCat uses standard HTTP Methods to perform a listing of collections and CRUD (Create Read Update Delete) operations on instances.
  
  
Line 90: Line 107:
 
|-
 
|-
 
| List || GET || /{COLLECTION} || 200 OK || Y || Y
 
| List || GET || /{COLLECTION} || 200 OK || Y || Y
 +
|-
 +
| Count || GET || /{COLLECTION}?count=true || 200 OK || Y || Y
 
|-
 
|-
 
| Exists || HEAD || /{COLLECTION} || 204 No Content || Y || Y
 
| Exists || HEAD || /{COLLECTION} || 204 No Content || Y || Y
Line 121: Line 140:
 
* A method is '''''Idempotent''''' if the same operation repeated multiple times has the same side effect than using it one time. "''repeating the request will have the same intended effect, even if the original request succeeded, though the response might differ''" [https://tools.ietf.org/html/rfc7231#section-4.2.2 https://tools.ietf.org/html/rfc7231#section-4.2.2].
 
* A method is '''''Idempotent''''' if the same operation repeated multiple times has the same side effect than using it one time. "''repeating the request will have the same intended effect, even if the original request succeeded, though the response might differ''" [https://tools.ietf.org/html/rfc7231#section-4.2.2 https://tools.ietf.org/html/rfc7231#section-4.2.2].
  
[[#delete|‡]]  DELETE has been defined as idempotent. <span id="non_safe_delete">Allamaraju</span><ref>Allamaraju S. RESTful Web Services Cookbook: Solutions for Improving Scalability and Simplicity . O’Reilly. first ed. 2010</ref> argues that DELETE idempotency should be accomplished client-side. The server should inform the client if a delete succeeded because the resource was really deleted or it was not found i.e., '''''404 Not Found''''' error is suggested instead of '''''204 No Content'''''. The latter situation should be treated as idempotent by the client. For this reason, Science Catalogue does not provide server-side idempotency for DELETE and PURGE operations.
+
[[#delete|‡]]  DELETE has been defined as idempotent. <span id="non_safe_delete">Allamaraju</span><ref>Allamaraju S. RESTful Web Services Cookbook: Solutions for Improving Scalability and Simplicity . O’Reilly. first ed. 2010</ref> argues that DELETE idempotency should be accomplished client-side. The server should inform the client if a delete succeeded because the resource was really deleted or it was not found i.e., '''''404 Not Found''''' error is suggested instead of '''''204 No Content'''''. The latter situation should be treated as idempotent by the client. For this reason, gCat does not provide server-side idempotency for DELETE and PURGE operations.
  
 
You can find more information about HTTP Methods at [https://restfulapi.net/http-methods/ https://restfulapi.net/http-methods/]
 
You can find more information about HTTP Methods at [https://restfulapi.net/http-methods/ https://restfulapi.net/http-methods/]
Line 127: Line 146:
 
Uncommon HTTP Methods:
 
Uncommon HTTP Methods:
 
* PATCH method allows to perform a differential update (i.e. an update which provide only the differences and not the whole new representation) [];
 
* PATCH method allows to perform a differential update (i.e. an update which provide only the differences and not the whole new representation) [];
* PURGE method is not a standard but is a widely used in service which requires this action (e.g. [https://varnish-cache.org/docs/3.0/tutorial/purging.html Varnish], [https://wiki.squid-cache.org/SquidFaq/OperatingSquid#How_can_I_purge_an_object_from_my_cache.3F Squid]). The Science Catalogue provide support for this method but to support a wider range of clients it also provides the Purge action via DELETE with the additional get parameter <pre>purge=true</pre>
+
* PURGE method is not a standard but is a widely used in service which requires this action (e.g. [https://varnish-cache.org/docs/3.0/tutorial/purging.html Varnish], [https://wiki.squid-cache.org/SquidFaq/OperatingSquid#How_can_I_purge_an_object_from_my_cache.3F Squid]). gCat provide support for this method but to support a wider range of clients it also provides the Purge action via DELETE with the additional get parameter <pre>purge=true</pre>
 +
 
 +
=== Roles ===
 +
 
 +
The catalogue uses the following hierarchic roles:
 +
 
 +
* '''Catalogue-Member''': a user with such a role is mainly capable of listing and reading the items;
 +
* '''Catalogue-Editor''': a user with such a role is capable of managing the items he/she creates and capable of using other safe APIs;
 +
* '''Catalogue-Admin''': a user with such a role is capable of administrating many aspects of the catalogue;
 +
* '''Catalogue-Manager''': a user with such a role is capable of using all the APIs exposed by the service except moderation item APIs (i.e. approve, reject).
 +
 
 +
Another role that is not in the role hierarchy:
 +
 
 +
* '''Catalogue-Moderator''': a user with such a role is capable of invoking the item moderation APIs.
 +
 
 +
Please note that not all catalogues/organizations are moderated.
  
 
== Collections ==
 
== Collections ==
  
* [[#Group | Group]]
+
The following collections are available to any users. Non-safe methods can only be invoked by [[GCat_Background#Users.2C_Roles_and_Groups | Catalogue Editor]]
* [[#Item | Item]]
+
 
* [[#Resource | Resource]]
+
* [[#Item_Collection | Item]]
* [[#License | License]]
+
* [[#Resource_Collection | Resource]]
* [[#Organization | Organization]]
+
* [[#Profile_Collection | Profile]]
* [[#User | User]]
+
* [[#Namespace_Collection | Namespace]]
* [[#Profile | Profile]]
+
* [[#License_Collection | License]]
* [[#Item | Namespace]]
+
 
 +
The following collections are available only for [[GCat_Background#Users.2C_Roles_and_Groups | Catalogue Admin]]
 +
 
 +
* [[#Group_Collection | Group]]
 +
* [[#Organization_Collection | Organization]]
 +
* [[#User_Collection | User]]
 +
 
  
 
For each collection HEAD and OPTIONS operations are not shown in tables even there are always available.
 
For each collection HEAD and OPTIONS operations are not shown in tables even there are always available.
 +
 +
To keep this documentation simple, for each collection is presented only the relevant attributes of an instance.
 +
This allows getting the picture of the content of a request or the content of a response.
 +
 +
=== Item Collection ===
 +
 +
{| class="wikitable"
 +
! Operation || HTTP Method || URL || Min Required Role
 +
|-
 +
| List || GET || <code>/items?limit=10&offset=0</code> || Catalogue-Member
 +
|-
 +
| Count || GET || <code>/items?count=true</code> || Catalogue-Member
 +
|-
 +
|
 +
|-
 +
| Bulk Delete || DELETE || <code>/items</code> || rowspan=3 | Catalogue-Editor <br /> (Catalogue-Editor can only delete own items)
 +
|-
 +
| Bulk Purge || DELETE || <code>/items?purge=true</code>
 +
|-
 +
| Bulk Purge || PURGE || <code>/items</code>
 +
|-
 +
|
 +
|-
 +
| Create || POST || <code>/items?social_post=true</code> || Catalogue-Editor
 +
|-
 +
| Read || GET || <code>/items/{NAME}</code> || Catalogue-Member
 +
|-
 +
| Update || PUT || <code>/items/{NAME}</code> || Catalogue-Editor
 +
|-
 +
|
 +
|-
 +
| Moderate || POST || <code>/items/{NAME}</code> || Catalogue-Moderator && Catalogue-Editor(Owner)
 +
|-
 +
|
 +
|-
 +
| Delete || DELETE || <code>/items/{NAME}</code> || Catalogue-Editor
 +
|-
 +
| Purge || DELETE || <code>/items/{NAME}?purge=true</code> || Catalogue-Editor
 +
|-
 +
| Purge || PURGE || <code>/items/{NAME}</code> || Catalogue-Editor
 +
|}
 +
 +
==== List Items ====
 +
 +
The listing API provides paginated results by using the query parameters '''limit''' and '''offset'''. The defaults values are 10 for '''limit''' and 0 for '''offset'''.
 +
To get unlimited results the '''limit''' query parameters must be set to '''-1'''. If the results are too much the operation could fail. It is recommended to request no more than 1000 results.
 +
The '''offset''' parameter indicates the page number taking into account pages containing a number of results equivalent to '''limit'''.
 +
 +
===== Filtering options =====
 +
 +
The listing method offers options to filter the results, thus enacting to search for items including spatial search (see ext_bbox below). 
 +
 +
It accepts the following query parameters (a subset of Solr search query parameters, see [http://www.solrtutorial.com/solr-query-syntax.html Solr Query Syntax]):
 +
* '''q''' (string) – the solr query. Optional. Default: "*:*" See [http://www.solrtutorial.com/solr-query-syntax.html Solr Query Syntax]
 +
** E.g. <code>/items?q=title:foo</code> returns the items with word "foo" in the title;
 +
* '''fq''' (string) – Filter query. A query string that limits the query results without influencing their scores. Note: +site_id:{ckan_site_id} is added to this string prior to the query being executed.
 +
** E.g. <code>/items?q=title:foo&fq=notes:bar</code> returns with word "foo" in the 'title' and the word "bar" in the 'notes';
 +
* '''fq_list''' (list of strings) – additional filter queries to apply.
 +
** E.g. <code>/items?q=title:foo&fq_list=...</code> returns the items with word "foo" in the 'title'
 +
* '''sort''' (string) – sorting of the search results. Optional. Default: 'relevance asc, metadata_modified desc'. As per the solr documentation, this is a comma-separated string of field names and sort-orderings.
 +
** E.g. <code>/items?q=title:foo&sort=name+asc</code> returns the items with word "foo" in the 'title' sorting the results by name ascending;
 +
<!--
 +
* '''facet''' (string) – whether to enable faceted results. Default: True.
 +
* '''facet.mincount''' (int) – the minimum counts for facet fields should be included in the results.
 +
* '''facet.limit''' (int) – the maximum number of values the facet fields return. A negative value means unlimited. This can be set instance-wide with the search.facets.limit config option. Default is 50.
 +
* '''facet.field''' (list of strings) – the fields to facet upon. Default empty. If empty, then the returned facet information is empty.
 +
-->
 +
<!--
 +
* '''include_drafts''' (bool) – if True, draft datasets will be included in the results. A user will only be returned their own draft datasets, and a sysadmin will be returned all draft datasets. Optional, the default is False.
 +
** ... this is a new concept ... I do ignore what is a draft dataset now;
 +
** E.g. <code>/items?q=...&include_drafts</code> returns ...
 +
-->
 +
* '''include_private''' (bool) – if True, private datasets will be included in the results. Only private datasets from the user’s organizations will be returned and sysadmins will be returned all private datasets. Optional, the default is False.
 +
** E.g. <code>/items?include_private=true</code>
 +
<!--
 +
* '''use_default_schema''' (bool) – use default package schema instead of a custom schema defined with an IDatasetForm plugin (default: False)
 +
* '''rows''' (int) – the maximum number of matching rows (datasets) to return. (optional, default: 10, upper limit: 1000)
 +
* '''start''' (int) – the offset in the complete result for where the set of returned datasets should begin.
 +
-->
 +
<!-- ===== Spatial search can be done using ===== -->
 +
* '''ext_bbox''': The coordinates of the upper-right and bottom-left angle of a rectangular to query for. The form is '''Lat,Long,Lat,Long'''
 +
** E.g. <code>/items?limit=10&offset=0&q=Pollution&ext_bbox=-7.535093,49.208494,3.890688,57.372349</code> returns the first 10 items with 'Pollution' having a spatial coverage in the specified bounding box
 +
 +
==== Bulk Delete-Purge ====
 +
 +
The bulk delete API provides a way to delete/purge all the times matching a certain criteria (see [[GCat_Service#Filtering options|Filtering options]])
 +
When invoked with no arguments delete all the items of the organization(s) enabled for the current context.
 +
It deletes only the items of the current user for non-admin users.
 +
 +
For admin users, it is capable of deleting the whole catalogue content. Please use it with care at your risk.
 +
By default, also the admins delete only their items. To specify to delete all the items of all users for the current organizations you must specify the query parameter
 +
 +
<code>own_only=false</code>
 +
 +
It is suggested to check the result of [[GCat_Service#List Items|List Items]] APIs to check what you are going to delete.
 +
 +
==== Create Items ====
 +
 +
The create API allows to create social posts by indicating '''social_post=true''' in the URL. The '''social_post''' query parameter is optional and the default value is '''false'''.
 +
 +
An Item is mainly described by the following attributes (* indicate mandatory attributes):
 +
 +
* '''name*''' (string): the name of the new item, must be between 2 and 100 characters long and contain only lowercase alphanumeric characters, '-' and '_';
 +
* '''title''' (string, defaut="same as name"): the title of the item;
 +
* '''private*''' (bool): If True creates a private item;
 +
 +
* '''maintainer''' (string): the name of the item’s maintainer;
 +
* '''maintainer_email''' (string): the email address of the item’s maintainer;
 +
* '''license_id*''' (license id string): the id of the item’s license, see license_list() for available values;
 +
* '''notes''' (string): a description of the item;
 +
* '''url''' (string): a URL for the item’s source;
 +
* '''version''' (string, no longer than 100 characters): ...;
 +
* '''state''' (string, default='active'): the current state of the item, e.g. 'active' or 'deleted', only active items show up in search results and other lists of items, this parameter will be ignored if you are not authorized to change the state of the item;
 +
* '''groups''' (list of dictionaries): the groups to which the item belongs, each group dictionary should have one or more of the following keys which identify an existing group: 'id' (the id of the group, string), or 'name' (the name of the group, string). To see which groups exist use list method in [[#Group_Collection | Group Collection]]
 +
* '''tags''' (list of tag dictionaries): the item’s tags. The tag is a dictionary in the format:
 +
** ''name'' : the name for the tag, a string between 2 and 100 characters long containing only alphanumeric characters and '-, '_' and '.'.
 +
* '''resources''' (list of resource dictionaries): the item’s resources, see [[#Resource_Collection | Resource Collection]] for the format of resource dictionaries;
 +
* '''extras''' (list of item extra dictionaries): the item’s extras, extras are arbitrary (key: value) metadata items that can be added to items, each extra dictionary should have keys 'key' (a string), 'value' (a string).
 +
 +
Parameter automatically managed:
 +
* '''author''' (string): the name of the item’s author (the owner of the gcube-token);
 +
* '''author_email''' (string): the email address of the item’s author (the email of the owner of gcube-token);
 +
* '''owner_org''' (string): the id of the item’s owning organization, see organization_list() or organization_list_for_user() for available values (the VRE corrsponding to the gcube-token).
 +
 +
<!--
 +
* '''type''' (string): the type of the item, IitemForm plugins associate themselves with different item types and provide custom item handling behaviour for these types;
 +
* '''relationships_as_object''' (list of relationship dictionaries): see package_relationship_create() for the format of relationship dictionaries;
 +
* '''relationships_as_subject''' (list of relationship dictionaries): see package_relationship_create() for the format of relationship dictionaries;
 +
-->
 +
 +
 +
===== Geo-Indexing your datasets =====
 +
 +
In order to make an Item searchable by location, it must have a special extra, with its key named ‘spatial’. The value must be a valid GeoJSON geometry, for example:
 +
 +
<pre>
 +
{
 +
  "type":"Polygon",
 +
  "coordinates":[[[2.05827, 49.8625],[2.05827, 55.7447], [-6.41736, 55.7447], [-6.41736, 49.8625], [2.05827, 49.8625]]]
 +
}
 +
</pre>
 +
or
 +
 +
<pre>
 +
{
 +
  "type": "Point",
 +
  "coordinates": [-3.145,53.078]
 +
}
 +
</pre>
 +
 +
If at least one profile has been defined within this context, then you need to specify the profile's type when creating the item. You need to insert, among the extras of the JSON object describing the item, a '''''system:type''''' property with the proper value (i.e. its value must be equal to the type property contained in the profile). The validation of the submitted request will be performed against the profile whose type has been specified. The other profile's properties need to be specified within the extras field as well.
 +
 +
If no profile has been defined, then no validation will be performed. Thus you do not need to set any '''''system:type''''' property.
 +
 +
Suppose you discover that in the context you are willing to publish are defined the following profiles: 'A', 'B' and 'C'.
 +
 +
You are interested in publishing an item with profile 'A', thus you download the XML of profile 'A' which looks like the following
 +
 +
<source lang="xml">
 +
<metadataformat type="A">
 +
    <metadatafield>
 +
        <fieldName>Input Parameter</fieldName>
 +
        <mandatory>false</mandatory>
 +
        <dataType>String</dataType>
 +
        <note>Input parameter expected for the execution of the process</note>
 +
    </metadatafield>
 +
    <metadatafield>
 +
        <fieldName>Output Parameter</fieldName>
 +
        <mandatory>false</mandatory>
 +
        <dataType>String</dataType>
 +
        <note>Output parameter expected from the execution of the process </note>
 +
    </metadatafield>
 +
    <metadatafield>
 +
        <fieldName>Process Author</fieldName>
 +
        <mandatory>false</mandatory>
 +
        <dataType>String</dataType>
 +
        <maxOccurs>*</maxOccurs>
 +
        <defaultValue/>
 +
        <note>The name of the author, with email and ORCID. The format should be: family, given[, email][, ORCID]. Example: Smith, John, js@acme.org, orcid.org/0000-0002-1825-0097</note>
 +
        <validator>
 +
            <regularExpression>^[a-zA-ZÀ-ÿ .'-]+, [a-zA-ZÀ-ÿ .'-]+[,]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9À-ÿ.-]+)?[,]*(orcid.org\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\d\d\d-\d\d\d[\dX])?$</regularExpression>
 +
        </validator>
 +
    </metadatafield>
 +
</metadataformat>
 +
</source>
 +
 +
The previous information can be easily discovered with the [[#Profile_Collection | Profile Collection]].
 +
Then, the JSON object of the create request will look like
 +
 +
<pre>
 +
{
 +
    "metadataformat": {
 +
        "metadatafield": [
 +
            {
 +
                "note": "Input parameter expected for the execution of the process",
 +
                "fieldName": "Input Parameter",
 +
                "dataType": "String",
 +
                "mandatory": false
 +
            },
 +
            {
 +
                "note": "Output parameter expected from the execution of the process",
 +
                "fieldName": "Output Parameter",
 +
                "dataType": "String",
 +
                "mandatory": false
 +
            },
 +
            {
 +
                "note": "The name of the author, with email and ORCID. The format should\t\t\tbe: family, given[, email][, ORCID]. Example: Smith, John,\t\t\tjs@acme.org, orcid.org/0000-0002-1825-0097",
 +
                "fieldName": "Process Author",
 +
                "defaultValue": "",
 +
                "dataType": "String",
 +
                "validator": {
 +
                    "regularExpression": "^[a-zA-ZÀ-ÿ .'-]+, [a-zA-ZÀ-ÿ .'-]+[,\t\t\t\t]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9À-ÿ.-]+)?[,\t\t\t\t]*(orcid.org\\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\\d\\d\\d-\\d\\d\\d[\\dX])?$"
 +
                },
 +
                "maxOccurs": "*",
 +
                "mandatory": false
 +
            }
 +
        ],
 +
        "type": "DataMiner Process"
 +
    }
 +
}
 +
</pre>
 +
 +
=== Trash Collection ===
 +
 +
{| class="wikitable"
 +
! Operation || HTTP Method || URL || Min Required Role
 +
|-
 +
| List || GET || <code>/trash[?own_only=true]</code> || Catalogue-Editor
 +
|-
 +
| Delete || DELETE <br /> PURGE || <code>/trash[?own_only=true]</code> || Catalogue-Editor
 +
|}
 +
 +
Only Catalogue ADMINs are entitled to list and remove the items he/she doesn't own.
 +
 +
=== Resource Collection ===
 +
 +
{| class="wikitable"
 +
! Operation || HTTP Method || URL || Min Required Role
 +
|-
 +
| List || GET || <code>/items/{ITEM_ID}/resources</code>
 +
|-
 +
| Create || POST || <code>/items/{ITEM_ID}/resources</code>
 +
|-
 +
| Read || GET || <code>/items/{ITEM_ID}/resources/{ID}</code>
 +
|-
 +
| Update || PUT || <code>/items/{ITEM_ID}/resources/{ID}</code>
 +
|-
 +
| Delete || DELETE || <code>/items/{ITEM_ID}/resources/{ID}</code>
 +
|}
 +
 +
 +
An Resource is mainly described by the following attributes (* indicate mandatory attributes):
 +
 +
* '''name*''' (string);
 +
* '''url*''' (string): url of resource;
 +
* <del>'''package_id*''' (string): id of the item that the resource should be added to;</del>
 +
* '''revision_id''' (string);
 +
* '''description''' (string);
 +
* '''format''' (string);
 +
* '''mimetype''' (string);
 +
* '''created''' (iso date string);
 +
* '''last_modified''' (iso date string);
 +
 +
 +
<!--
 +
* '''size''' (int);
 +
* '''mimetype_inner''' (string) – (optional)
 +
* '''cache_url''' (string) – (optional)
 +
* '''resource_type''' (string) – (optional)
 +
* '''hash''' (string)
 +
* '''cache_last_updated''' (iso date string) – (optional)
 +
* '''upload''' (FieldStorage (optional) needs multipart/form-data) – (optional)
 +
-->
 +
 +
=== Profile (or Type) Collection ===
 +
 +
{| class="wikitable"
 +
! Operation || HTTP Method || URL || Min Required Role
 +
|-
 +
| List || GET || /profiles ||
 +
|-
 +
| Count || GET || /profiles?count=true
 +
|-
 +
| Create || PUT || /profiles/{NAME}
 +
|-
 +
| Read || GET || /profiles/{NAME}
 +
|-
 +
| Update || PUT || /profiles/{NAME}
 +
|-
 +
| Delete || DELETE || /profiles/{NAME}
 +
 +
|}
 +
 +
Only a [[GCat_Background#Users.2C_Roles_and_Groups | Catalogue Editor]] can manipulate profiles (Create, Update, Delete). These operation are available starting from gCat 1.1.0.
 +
 +
At creation and update time the provided profile is validated against the defined XSD schema (see [[GCat_Background#Metadata_Profile_v.3 | Item Profile Defintion v3]])
 +
 +
 +
A profile is defined using XML. In place of ''application/json'' Use
 +
 +
<pre>Accept : application/xml</pre>
 +
 +
and
 +
 +
<pre>Content-Type : application/xml</pre>
 +
 +
 +
Listing return a JSON Array. It is also possible to read the representation of a profile as JSON which is an automatic mapping to of the original XML definition to JSON.
 +
 +
An example of Profiles listing  is :
 +
 +
<pre>
 +
[
 +
    "EOSCService",
 +
    "SoBigData.eu: Dataset Metadata",
 +
    "SoBigData.eu: Application Metadata",
 +
    "SoBigData.eu: Method Metadata"
 +
]
 +
</pre>
 +
 +
An example of Profile is :
 +
 +
<source lang="xml">
 +
<metadataformat type="SoBigData Method">
 +
    <metadatafield>
 +
        <fieldName>External Identifier</fieldName>
 +
        <dataType>String</dataType>
 +
        <mandatory>false</mandatory>
 +
        <note>This applies only to methods that have been already published. Insert here a DOI, an handle, and any other Identifier assigned when publishing the dataset alsewhere.</note>
 +
    </metadatafield>
 +
    <metadatafield>
 +
        <fieldName>Creator</fieldName>
 +
        <mandatory>true</mandatory>
 +
        <note>The name of the creator, with email and ORCID. The format should be: family, given[, email][, ORCID]. Example: Smith, John, js@acme.org, orcid.org//0000-0002-1825-0097</note>
 +
        <validator>
 +
            <regularExpression>^[a-zA-Z .'-]+, [a-zA-Z .'-]+[, ]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9.-]+)?[, ]*(orcid.org\/\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\d\d\d-\d\d\d[\dX])?$</regularExpression>
 +
        </validator>
 +
    </metadatafield>
 +
    <metadatafield>
 +
        <fieldName>CreationDate</fieldName>
 +
        <mandatory>true</mandatory>
 +
        <dataType>Time</dataType>
 +
        <note>The date of creation of the dataset (different from the date of registration of the dataset automatically added by the system). Use ISO 8601 Date Format: YYYY-MM-DD[ HH:MM] Ex. 1998-11-10 or 2015-05-29 11:55</note>
 +
        <validator>
 +
            <regularExpression>^(\d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01]))+([ ]+(\d{2}(:?\d{2})?)?)?$</regularExpression>
 +
        </validator>
 +
    </metadatafield>
 +
    <metadatafield>
 +
        <fieldName>Owner</fieldName>
 +
        <mandatory>true</mandatory>
 +
        <note>The name of the owner, with email and ORCID. The format should be: family, given[, email][, ORCID]. Example: Smith, John, js@acme.org, orcid.org//0000-0002-1825-0097</note>
 +
        <validator>
 +
            <regularExpression>^[a-zA-Z .'-]+, [a-zA-Z .'-]+[, ]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9.-]+)?[, ]*(orcid.org\/\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\d\d\d-\d\d\d[\dX])?$</regularExpression>
 +
        </validator>
 +
    </metadatafield>
 +
    ....
 +
    <metadatafield>
 +
        <fieldName>License term</fieldName>
 +
        <mandatory>false</mandatory>
 +
        <dataType>Time_Interval</dataType>
 +
        <note>Period of time during which the dataset may be used. Use ISO 8601 Date Format: YYYY-MM-DD[ HH:MM] Ex. 2016-07-31 or 2015-05-10 12:00</note>
 +
        <vocabulary/>
 +
        <validator>
 +
            <regularExpression>^(\d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01]))+([ ]+(\d{2}(:?\d{2})?)?)?$</regularExpression>
 +
        </validator>
 +
    </metadatafield>
 +
    <metadatafield>
 +
        <fieldName>Requirement of non-disclosure (confidentiality mark)</fieldName>
 +
        <mandatory>false</mandatory>
 +
        <note>Requirement of non-disclosure (confidentiality mark). Whether the dataset bears confidentiality mark/may be used and shared subject to the obligation of non-disclosure</note>
 +
    </metadatafield>
 +
</metadataformat>
 +
</source>
 +
 +
 +
You can find all details about profiles at [[GCat_Background#Metadata | Item Profile]]
 +
 +
=== Namespace Collection ===
 +
 +
{| class="wikitable"
 +
! Operation || HTTP Method || URL || Min Required Role
 +
|-
 +
| List || GET || /namespaces || Catalogue-Member
 +
|}
 +
 +
An example of given result is:
 +
 +
<pre>
 +
[
 +
    {
 +
        "id": "contact",
 +
        "title": "Contact Title",
 +
        "name": "contact",
 +
        "description": "This section is about Contact(s)"
 +
    },
 +
    {
 +
        "id": "developer_information",
 +
        "title": "Developer Information",
 +
        "name": "developer_information",
 +
        "description": "This section is about Developer(s)"
 +
    },
 +
    {
 +
        "id": "extra_information",
 +
        "title": "Extras",
 +
        "name": "extra_information",
 +
        "description": "This section is about Extra(s)"
 +
    },
 +
    {
 +
        "id": "ReportDetails",
 +
        "title": "Report Details",
 +
        "name": "ReportDetails",
 +
        "description": "Report Details"
 +
    }
 +
]
 +
</pre>
 +
 +
=== License Collection ===
 +
 +
{| class="wikitable"
 +
! Operation || HTTP Method || URL || Min Required Role
 +
|-
 +
| List || GET || /licenses || Catalogue-Member
 +
|}
 +
 +
 +
An example of given result is:
 +
 +
<pre>
 +
[
 +
    {
 +
        "status": "active",
 +
        "maintainer": "Creative Commons",
 +
        "od_conformance": "rejected",
 +
        "family": "Creative Commons",
 +
        "osd_conformance": "not reviewed",
 +
        "domain_data": true,
 +
        "title": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0",
 +
        "url": "https://creativecommons.org/licenses/by-nc-sa/4.0/",
 +
        "domain_content": true,
 +
        "is_okd_compliant": false,
 +
        "is_osi_compliant": false,
 +
        "domain_software": false,
 +
        "id": "CC-BY-NC-SA-4.0"
 +
    },
 +
    {
 +
        "status": "active",
 +
        "maintainer": "Creative Commons",
 +
        "od_conformance": "rejected",
 +
        "family": "Creative Commons",
 +
        "osd_conformance": "not reviewed",
 +
        "domain_data": true,
 +
        "title": "Creative Commons Attribution-NonCommercial-NoDerivatives 4.0",
 +
        "url": "https://creativecommons.org/licenses/by-nc-nd/4.0/",
 +
        "domain_content": true,
 +
        "is_okd_compliant": false,
 +
        "is_osi_compliant": false,
 +
        "domain_software": false,
 +
        "id": "CC-BY-NC-ND-4.0"
 +
    },
 +
    ...
 +
    {
 +
        "status": "active",
 +
        "maintainer": "Free Software Foundation",
 +
        "od_conformance": "not reviewed",
 +
        "family": "",
 +
        "osd_conformance": "approved",
 +
        "domain_data": false,
 +
        "title": "GNU Affero General Public License v3",
 +
        "url": "http://www.opensource.org/licenses/AGPL-3.0",
 +
        "domain_content": false,
 +
        "is_okd_compliant": false,
 +
        "is_osi_compliant": true,
 +
        "domain_software": true,
 +
        "id": "AGPL-3.0"
 +
    }
 +
]
 +
</pre>
  
 
=== Group Collection ===
 
=== Group Collection ===
  
 
{| class="wikitable"
 
{| class="wikitable"
! Operation || HTTP Method || URL
+
! Operation || HTTP Method || URL || Min Required Role
 
|-
 
|-
 
| List || GET || /groups
 
| List || GET || /groups
 +
|-
 +
| Count || GET || /groups?count=true
 
|-
 
|-
 
| Create || POST || /groups
 
| Create || POST || /groups
Line 164: Line 683:
 
|}
 
|}
  
Only [[GCube_Resource_Catalogue#Users.2C_Roles_and_Groups CatalogueAdmin]] are able to invoke non-safe methods.
+
Only [[GCat_Background#Users.2C_Roles_and_Groups | Catalogue Admin]] are able to invoke non-safe methods.
  
  
==== Create (POST) ====
+
A group is mainly described by the following attributes ('''*''' indicate mandatory attributes):
 
+
A group is described by the following attributes ('''*''' indicate mandatory attributes):
+
 
* '''name*''' (string): the name of the group, a string between 2 and 100 characters long, containing only lowercase alphanumeric characters, '-' and '_' ;
 
* '''name*''' (string): the name of the group, a string between 2 and 100 characters long, containing only lowercase alphanumeric characters, '-' and '_' ;
 
* '''id''' (string): the id of the group;
 
* '''id''' (string): the id of the group;
Line 187: Line 704:
 
-->
 
-->
  
The content body of the request should be something like this
+
==== Representation ====
  
<pre>
+
An example of Group representation is:
{
+
    "name": "group_name",
+
    "title": "Group Title"
+
}
+
</pre>
+
 
+
The obtained result is something like this:
+
  
 
<pre>
 
<pre>
Line 211: Line 721:
 
}
 
}
 
</pre>
 
</pre>
 
=== Item Collection ===
 
 
{| class="wikitable"
 
! Operation || HTTP Method || URL
 
|-
 
| List || GET || /items
 
|-
 
| Create || POST || /items
 
|-
 
| Read || GET || /items/{NAME}
 
|-
 
| Update || PUT || /items/{NAME}
 
|-
 
| Delete || DELETE || /items/{NAME}
 
|-
 
| Purge || PURGE || /items/{NAME}
 
|-
 
| Purge || DELETE || /items/{NAME}?purge=true
 
|}
 
 
=== Resource Collection ===
 
 
{| class="wikitable"
 
! Operation || HTTP Method || URL
 
|-
 
| List || GET || /items/{NAME}/resources
 
|-
 
| Create || POST || /items/{NAME}/resources
 
|-
 
| Read || GET || /items/{NAME}/resources/{ID}
 
|-
 
| Update || PUT || /items/{NAME}/resources/{ID}
 
|-
 
| Delete || DELETE || /items/{NAME}/resources/{ID}
 
|}
 
 
=== License Collection ===
 
 
{| class="wikitable"
 
! Operation || HTTP Method || URL
 
|-
 
| List || GET || /licenses
 
|}
 
  
 
=== Organization Collection ===
 
=== Organization Collection ===
  
 
{| class="wikitable"
 
{| class="wikitable"
! Operation || HTTP Method || URL
+
! Operation || HTTP Method || URL || Min Required Role
 
|-
 
|-
 
| List || GET || /organizations
 
| List || GET || /organizations
 +
|-
 +
| Count || GET || /organizations?count=true
 
|-
 
|-
 
| Create || POST || /organizations
 
| Create || POST || /organizations
Line 278: Line 746:
 
|}
 
|}
  
Only [[https://wiki.gcube-system.org/gcube/GCube_Resource_Catalogue#Users.2C_Roles_and_Groups Catalogue_Admin]] are able to invoke non-safe methods.
+
Only [[GCat_Background#Users.2C_Roles_and_Groups | Catalogue Admin]] are able to invoke non-safe methods.
 +
 
 +
An Organization is mainly described by the following attributes (* indicate mandatory attributes):
 +
 
 +
 
 +
* '''name*''' (string): the name of the organization, a string between 2 and 100 characters long, containing only lowercase alphanumeric characters, '-' and '_';
 +
* '''id''' (string): the id of the organization;
 +
* '''title''' (string): the title of the organization;
 +
* '''description''' (string): the description of the organization;
 +
* '''image_url''' (string): the URL to an image to be displayed on the organization’s page;
 +
* '''state''' (string, default: 'active'): the current state of the organization, e.g. 'active' or 'deleted', only active organizations show up in search results and other lists of organizations, this parameter will be ignored if you are not authorized to change the state of the organization;
 +
* '''approval_status''' (string);
 +
* '''extras''' (list of dataset extra dictionaries): the organization’s extras (optional), extras are arbitrary (key: value) metadata items that can be added to organizations, each extra dictionary should have keys 'key' (a string), 'value' (a string), and optionally 'deleted'.
 +
 
 +
<!--
 +
* '''packages (list of dictionaries): the items that belong to the organization, a list of dictionaries each with keys 'name' (string, the id or name of the dataset) and optionally 'title' (string, the title of the dataset)
 +
* '''users (list of dictionaries): the users that belong to the organization, a list of dictionaries each with key 'name' (string, the id or name of the user) and optionally 'capacity' (string, the capacity in which the user is a member of the organization)
 +
-->
  
 
=== User Collection ===
 
=== User Collection ===
  
 
{| class="wikitable"
 
{| class="wikitable"
! Operation || HTTP Method || URL
+
! Operation || HTTP Method || URL || Min Required Role
 
|-
 
|-
 
| List || GET || /users
 
| List || GET || /users
Line 296: Line 781:
 
|}
 
|}
  
Only [[https://wiki.gcube-system.org/gcube/GCube_Resource_Catalogue#Users.2C_Roles_and_Groups Catalogue_Admin]] are able to invoke non-safe methods.
+
Only [[GCat_Background#Users.2C_Roles_and_Groups | Catalogue Admin]] are able to invoke non-safe methods.
  
=== Profile Collection ===
+
A User is mainly described by the following attributes (* indicate mandatory attributes):
 +
* '''name*''' (string): the name of the user, a string between 2 and 100 characters in length, containing only lowercase alphanumeric characters, '-' and '_';
 +
* '''id''' (string): the id of the user;
 +
* '''fullname''' (string): the full name of the user;
 +
* '''email''' (string): the email address for the user;
 +
* '''password''' (string): the password of the user, a string of at least 4 characters (parameter only used to create a new user);
 +
* '''about''' (string): a description of the user.
  
{| class="wikitable"
 
! Operation || HTTP Method || URL
 
|-
 
| List || GET || /profiles
 
|-
 
| Read || GET || /profiles/{NAME}
 
|}
 
  
A profile is defined using XML. It is possible to get the original XML definition using 'Accept' HTTP Header.
+
<pre>
 +
{
 +
    "name": "luca_frosini",
 +
    "id": " 5f5c2bc0-bcd4-460a-9548-52cd4d33b263",
 +
    "fullname": "Luca Frosini"
 +
}
 +
</pre>
  
<pre>Accept : application/xml</pre>
 
  
instead of
 
  
<pre>Accept : application/json</pre>
 
  
=== Namespace Collection ===
+
=== Dev and Pre Users used for moderation tests ===
 +
 
 +
To perform moderation tests in dev and preproduction infrastructure we use different users with the indicated roles.
  
 
{| class="wikitable"
 
{| class="wikitable"
! Operation || HTTP Method || URL
+
! User || Username || Role
 +
|-
 +
| Mister White || mister.white || Catalogue-Manager
 +
|-
 +
| Mister Blue || mister.blue ||  Catalogue-Admin
 +
|-
 +
| Mister Orange || mister.orange || Catalogue-Editor
 +
|-
 +
| Mister Pink || mister.pink || NO ROLE (means Catalogue-Member)
 +
|-
 +
| Mister Blonde || mister.blonde || Catalogue-Admin + Catalogue-Moderator
 
|-
 
|-
| List || GET || /namespaces
+
| Mister Brown || mister.brown || Catalogue-Moderator
 
|}
 
|}

Latest revision as of 08:03, 17 April 2024

THIS WIKI PAGE IS NO LONGER MAINTAINED. YOU CAN FIND UP-TO-DATE DOCUMENTATION AT THE FOLLOWING LINKS

THE FIRST LINK CONTAINS GENERAL INFORMATION ABOUT THE SERVICE

https://api.d4science.org/gcat/docs/index.html


THE SECOND LINK CONTAINS API DOCUMENTATION AS WELL AS AN INTERACTIVE INTERFACE PROVIDED BY SWAGGER

https://api.d4science.org/gcat/api-docs/index.html



gCat Service is a RESTful service enabling any client to programmatically interact with the gCube Catalogue. gCat Service uses the principles defined in gCat Background

Request

URL

The URL used to interact with gCat is composed of two parts:

D4Science infrastructure uses cloud facilities allowing to replicate a service to achieve failover and load balancing.

gCat instances can be deployed and undeployed dynamically. For such a reason the Base Service URL MUST NOT be hard-cabled in the code because it can change over time.

To dynamically discover the Base Service URL you can use the Registry RESTful Service

You need to discover gCore Resource having:

  • class : DataPublishing
  • name : gcat

HTTP Headers

gCube Authorization Token

Any request performed to gCat MUST contains the gCube Authorization Token.

This is done using the HTTP Header gcube-token

gcube-token: YOUR-TOKEN

The gcube-token HTTP header acts as the standard Authorization HTTP header. Plese note that Authorization HTTP header cannot be used in place of gcube-token HTTP header.

Retrieve your gCube Authorization Token

the gCube Authorization Token is a UUID bound to yourself and a given Infrastructure context. To retrieve it, you just need to go to a VRE for which you are interested and use the Authorisation Options portlet (see below)

Authorisation option.png

Click on Show button and select the token.

Content Type

Any request must contain the indication of the interested content type.

For any operation returning a result, the client must specify the Accept HTTP Header.

Accept: application/json

For any operation sending content to the service, it is necessary to specify the Content-Type HTTP Header.

Content-Type: application/json

Actually, the service accepts and returns only JSON objects.

Except for a profile which can be also requested in XML.

HTTP Statuses

Any successful operation returns 200 OK status code except for create operation which return 201 Created or for the operations which do not provide any content and returns 204 No Content.

The most common error status a client can obtain are

A complete list of HTTP Status can be found here: https://httpstatuses.com/

You can report a 500 Internal Server Error the ticketing system. Please use this checklist before reporting an error:

  • replicate the request
  • the failure could be temporal due to network error, server issue so please retry the request after a certain amount of time
  • indicate how to replicate the error
  • indicate the time when the error occurred (this simplify the identification of the issue)

HTTP Methods

To be RESTful compliant gCat uses standard HTTP Methods to perform a listing of collections and CRUD (Create Read Update Delete) operations on instances.


Operation HTTP Method URL Success HTTP Status Safe Idempotent
Supported
HTTP Methods
OPTIONS /{COLLECTION} 204 No Content
(Supported HTTP Methods in Allow HTTP Header)
Y Y
List GET /{COLLECTION} 200 OK Y Y
Count GET /{COLLECTION}?count=true 200 OK Y Y
Exists HEAD /{COLLECTION} 204 No Content Y Y
Create POST /{COLLECTION} 201 Created N N
Supported
HTTP Methods
OPTIONS /{COLLECTION}/{INSTANCE_ID} 204 No Content
(Supported HTTP Methods in Allow HTTP Header)
Y Y
Exist HEAD /{COLLECTION}/{INSTANCE_ID} 204 No Content Y Y
Read GET /{COLLECTION}/{INSTANCE_ID} 200 OK Y Y
Update PUT /{COLLECTION}/{INSTANCE_ID} 200 OK N Y
Patch PATCH /{COLLECTION}/{INSTANCE_ID} 200 OK N Y
Delete DELETE /{COLLECTION}/{INSTANCE_ID} 204 No Content N N
Purge PURGE /{COLLECTION}/{INSTANCE_ID} 204 No Content N N
Purge DELETE /{COLLECTION}/{INSTANCE_ID}?purge=true 204 No Content N N


About URL:

  • {COLLECTION} is the plural name of the entity type;
  • {INSTANCE_ID} is an identification which enables to univocally identify the instance in the collection.

About Safety and Idempotency properties:

  • A method is Safe if it does not produce any side effects. "This does not prevent an implementation from including behaviour that is potentially harmful, that is not entirely read-only, or that causes side effects while invoking a safe method" https://tools.ietf.org/html/rfc7231#section-4.2.1;
  • A method is Idempotent if the same operation repeated multiple times has the same side effect than using it one time. "repeating the request will have the same intended effect, even if the original request succeeded, though the response might differ" https://tools.ietf.org/html/rfc7231#section-4.2.2.

DELETE has been defined as idempotent. Allamaraju[1] argues that DELETE idempotency should be accomplished client-side. The server should inform the client if a delete succeeded because the resource was really deleted or it was not found i.e., 404 Not Found error is suggested instead of 204 No Content. The latter situation should be treated as idempotent by the client. For this reason, gCat does not provide server-side idempotency for DELETE and PURGE operations.

You can find more information about HTTP Methods at https://restfulapi.net/http-methods/

Uncommon HTTP Methods:

  • PATCH method allows to perform a differential update (i.e. an update which provide only the differences and not the whole new representation) [];
  • PURGE method is not a standard but is a widely used in service which requires this action (e.g. Varnish, Squid). gCat provide support for this method but to support a wider range of clients it also provides the Purge action via DELETE with the additional get parameter
    purge=true

Roles

The catalogue uses the following hierarchic roles:

  • Catalogue-Member: a user with such a role is mainly capable of listing and reading the items;
  • Catalogue-Editor: a user with such a role is capable of managing the items he/she creates and capable of using other safe APIs;
  • Catalogue-Admin: a user with such a role is capable of administrating many aspects of the catalogue;
  • Catalogue-Manager: a user with such a role is capable of using all the APIs exposed by the service except moderation item APIs (i.e. approve, reject).

Another role that is not in the role hierarchy:

  • Catalogue-Moderator: a user with such a role is capable of invoking the item moderation APIs.

Please note that not all catalogues/organizations are moderated.

Collections

The following collections are available to any users. Non-safe methods can only be invoked by Catalogue Editor

The following collections are available only for Catalogue Admin


For each collection HEAD and OPTIONS operations are not shown in tables even there are always available.

To keep this documentation simple, for each collection is presented only the relevant attributes of an instance. This allows getting the picture of the content of a request or the content of a response.

Item Collection

Operation HTTP Method URL Min Required Role
List GET /items?limit=10&offset=0 Catalogue-Member
Count GET /items?count=true Catalogue-Member
Bulk Delete DELETE /items Catalogue-Editor
(Catalogue-Editor can only delete own items)
Bulk Purge DELETE /items?purge=true
Bulk Purge PURGE /items
Create POST /items?social_post=true Catalogue-Editor
Read GET /items/{NAME} Catalogue-Member
Update PUT /items/{NAME} Catalogue-Editor
Moderate POST /items/{NAME} Catalogue-Moderator && Catalogue-Editor(Owner)
Delete DELETE /items/{NAME} Catalogue-Editor
Purge DELETE /items/{NAME}?purge=true Catalogue-Editor
Purge PURGE /items/{NAME} Catalogue-Editor

List Items

The listing API provides paginated results by using the query parameters limit and offset. The defaults values are 10 for limit and 0 for offset. To get unlimited results the limit query parameters must be set to -1. If the results are too much the operation could fail. It is recommended to request no more than 1000 results. The offset parameter indicates the page number taking into account pages containing a number of results equivalent to limit.

Filtering options

The listing method offers options to filter the results, thus enacting to search for items including spatial search (see ext_bbox below).

It accepts the following query parameters (a subset of Solr search query parameters, see Solr Query Syntax):

  • q (string) – the solr query. Optional. Default: "*:*" See Solr Query Syntax
    • E.g. /items?q=title:foo returns the items with word "foo" in the title;
  • fq (string) – Filter query. A query string that limits the query results without influencing their scores. Note: +site_id:{ckan_site_id} is added to this string prior to the query being executed.
    • E.g. /items?q=title:foo&fq=notes:bar returns with word "foo" in the 'title' and the word "bar" in the 'notes';
  • fq_list (list of strings) – additional filter queries to apply.
    • E.g. /items?q=title:foo&fq_list=... returns the items with word "foo" in the 'title'
  • sort (string) – sorting of the search results. Optional. Default: 'relevance asc, metadata_modified desc'. As per the solr documentation, this is a comma-separated string of field names and sort-orderings.
    • E.g. /items?q=title:foo&sort=name+asc returns the items with word "foo" in the 'title' sorting the results by name ascending;
  • include_private (bool) – if True, private datasets will be included in the results. Only private datasets from the user’s organizations will be returned and sysadmins will be returned all private datasets. Optional, the default is False.
    • E.g. /items?include_private=true
  • ext_bbox: The coordinates of the upper-right and bottom-left angle of a rectangular to query for. The form is Lat,Long,Lat,Long
    • E.g. /items?limit=10&offset=0&q=Pollution&ext_bbox=-7.535093,49.208494,3.890688,57.372349 returns the first 10 items with 'Pollution' having a spatial coverage in the specified bounding box

Bulk Delete-Purge

The bulk delete API provides a way to delete/purge all the times matching a certain criteria (see Filtering options) When invoked with no arguments delete all the items of the organization(s) enabled for the current context. It deletes only the items of the current user for non-admin users.

For admin users, it is capable of deleting the whole catalogue content. Please use it with care at your risk. By default, also the admins delete only their items. To specify to delete all the items of all users for the current organizations you must specify the query parameter

own_only=false

It is suggested to check the result of List Items APIs to check what you are going to delete.

Create Items

The create API allows to create social posts by indicating social_post=true in the URL. The social_post query parameter is optional and the default value is false.

An Item is mainly described by the following attributes (* indicate mandatory attributes):

  • name* (string): the name of the new item, must be between 2 and 100 characters long and contain only lowercase alphanumeric characters, '-' and '_';
  • title (string, defaut="same as name"): the title of the item;
  • private* (bool): If True creates a private item;
  • maintainer (string): the name of the item’s maintainer;
  • maintainer_email (string): the email address of the item’s maintainer;
  • license_id* (license id string): the id of the item’s license, see license_list() for available values;
  • notes (string): a description of the item;
  • url (string): a URL for the item’s source;
  • version (string, no longer than 100 characters): ...;
  • state (string, default='active'): the current state of the item, e.g. 'active' or 'deleted', only active items show up in search results and other lists of items, this parameter will be ignored if you are not authorized to change the state of the item;
  • groups (list of dictionaries): the groups to which the item belongs, each group dictionary should have one or more of the following keys which identify an existing group: 'id' (the id of the group, string), or 'name' (the name of the group, string). To see which groups exist use list method in Group Collection
  • tags (list of tag dictionaries): the item’s tags. The tag is a dictionary in the format:
    • name : the name for the tag, a string between 2 and 100 characters long containing only alphanumeric characters and '-, '_' and '.'.
  • resources (list of resource dictionaries): the item’s resources, see Resource Collection for the format of resource dictionaries;
  • extras (list of item extra dictionaries): the item’s extras, extras are arbitrary (key: value) metadata items that can be added to items, each extra dictionary should have keys 'key' (a string), 'value' (a string).

Parameter automatically managed:

  • author (string): the name of the item’s author (the owner of the gcube-token);
  • author_email (string): the email address of the item’s author (the email of the owner of gcube-token);
  • owner_org (string): the id of the item’s owning organization, see organization_list() or organization_list_for_user() for available values (the VRE corrsponding to the gcube-token).


Geo-Indexing your datasets

In order to make an Item searchable by location, it must have a special extra, with its key named ‘spatial’. The value must be a valid GeoJSON geometry, for example:

{
  "type":"Polygon",
  "coordinates":[[[2.05827, 49.8625],[2.05827, 55.7447], [-6.41736, 55.7447], [-6.41736, 49.8625], [2.05827, 49.8625]]]
}

or

{
  "type": "Point",
  "coordinates": [-3.145,53.078]
}

If at least one profile has been defined within this context, then you need to specify the profile's type when creating the item. You need to insert, among the extras of the JSON object describing the item, a system:type property with the proper value (i.e. its value must be equal to the type property contained in the profile). The validation of the submitted request will be performed against the profile whose type has been specified. The other profile's properties need to be specified within the extras field as well.

If no profile has been defined, then no validation will be performed. Thus you do not need to set any system:type property.

Suppose you discover that in the context you are willing to publish are defined the following profiles: 'A', 'B' and 'C'.

You are interested in publishing an item with profile 'A', thus you download the XML of profile 'A' which looks like the following

<metadataformat type="A">
    <metadatafield>
        <fieldName>Input Parameter</fieldName>
        <mandatory>false</mandatory>
        <dataType>String</dataType>
        <note>Input parameter expected for the execution of the process</note>
    </metadatafield>
    <metadatafield>
        <fieldName>Output Parameter</fieldName>
        <mandatory>false</mandatory>
        <dataType>String</dataType>
        <note>Output parameter expected from the execution of the process </note>
    </metadatafield>
    <metadatafield>
        <fieldName>Process Author</fieldName>
        <mandatory>false</mandatory>
        <dataType>String</dataType>
        <maxOccurs>*</maxOccurs>
        <defaultValue/>
        <note>The name of the author, with email and ORCID. The format should be: family, given[, email][, ORCID]. Example: Smith, John, js@acme.org, orcid.org/0000-0002-1825-0097</note>
        <validator>
            <regularExpression>^[a-zA-ZÀ-ÿ .'-]+, [a-zA-ZÀ-ÿ .'-]+[,]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9À-ÿ.-]+)?[,]*(orcid.org\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\d\d\d-\d\d\d[\dX])?$</regularExpression>
        </validator>
    </metadatafield>
</metadataformat>

The previous information can be easily discovered with the Profile Collection. Then, the JSON object of the create request will look like

{
    "metadataformat": {
        "metadatafield": [
            {
                "note": "Input parameter expected for the execution of the process",
                "fieldName": "Input Parameter",
                "dataType": "String",
                "mandatory": false
            },
            {
                "note": "Output parameter expected from the execution of the process",
                "fieldName": "Output Parameter",
                "dataType": "String",
                "mandatory": false
            },
            {
                "note": "The name of the author, with email and ORCID. The format should\t\t\tbe: family, given[, email][, ORCID]. Example: Smith, John,\t\t\tjs@acme.org, orcid.org/0000-0002-1825-0097",
                "fieldName": "Process Author",
                "defaultValue": "",
                "dataType": "String",
                "validator": {
                    "regularExpression": "^[a-zA-ZÀ-ÿ .'-]+, [a-zA-ZÀ-ÿ .'-]+[,\t\t\t\t]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9À-ÿ.-]+)?[,\t\t\t\t]*(orcid.org\\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\\d\\d\\d-\\d\\d\\d[\\dX])?$"
                },
                "maxOccurs": "*",
                "mandatory": false
            }
        ],
        "type": "DataMiner Process"
    }
}

Trash Collection

Operation HTTP Method URL Min Required Role
List GET /trash[?own_only=true] Catalogue-Editor
Delete DELETE
PURGE
/trash[?own_only=true] Catalogue-Editor

Only Catalogue ADMINs are entitled to list and remove the items he/she doesn't own.

Resource Collection

Operation HTTP Method URL Min Required Role
List GET /items/{ITEM_ID}/resources
Create POST /items/{ITEM_ID}/resources
Read GET /items/{ITEM_ID}/resources/{ID}
Update PUT /items/{ITEM_ID}/resources/{ID}
Delete DELETE /items/{ITEM_ID}/resources/{ID}


An Resource is mainly described by the following attributes (* indicate mandatory attributes):

  • name* (string);
  • url* (string): url of resource;
  • package_id* (string): id of the item that the resource should be added to;
  • revision_id (string);
  • description (string);
  • format (string);
  • mimetype (string);
  • created (iso date string);
  • last_modified (iso date string);


Profile (or Type) Collection

Operation HTTP Method URL Min Required Role
List GET /profiles
Count GET /profiles?count=true
Create PUT /profiles/{NAME}
Read GET /profiles/{NAME}
Update PUT /profiles/{NAME}
Delete DELETE /profiles/{NAME}

Only a Catalogue Editor can manipulate profiles (Create, Update, Delete). These operation are available starting from gCat 1.1.0.

At creation and update time the provided profile is validated against the defined XSD schema (see Item Profile Defintion v3)


A profile is defined using XML. In place of application/json Use

Accept : application/xml

and

Content-Type : application/xml


Listing return a JSON Array. It is also possible to read the representation of a profile as JSON which is an automatic mapping to of the original XML definition to JSON.

An example of Profiles listing is :

[
    "EOSCService",
    "SoBigData.eu: Dataset Metadata",
    "SoBigData.eu: Application Metadata",
    "SoBigData.eu: Method Metadata"
]

An example of Profile is :

<metadataformat type="SoBigData Method">
    <metadatafield>
        <fieldName>External Identifier</fieldName>
        <dataType>String</dataType>
        <mandatory>false</mandatory>
        <note>This applies only to methods that have been already published. Insert here a DOI, an handle, and any other Identifier assigned when publishing the dataset alsewhere.</note>
    </metadatafield>
    <metadatafield>
        <fieldName>Creator</fieldName>
        <mandatory>true</mandatory>
        <note>The name of the creator, with email and ORCID. The format should be: family, given[, email][, ORCID]. Example: Smith, John, js@acme.org, orcid.org//0000-0002-1825-0097</note>
        <validator>
            <regularExpression>^[a-zA-Z .'-]+, [a-zA-Z .'-]+[, ]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9.-]+)?[, ]*(orcid.org\/\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\d\d\d-\d\d\d[\dX])?$</regularExpression>
        </validator>
    </metadatafield>
    <metadatafield>
        <fieldName>CreationDate</fieldName>
        <mandatory>true</mandatory>
        <dataType>Time</dataType>
        <note>The date of creation of the dataset (different from the date of registration of the dataset automatically added by the system). Use ISO 8601 Date Format: YYYY-MM-DD[ HH:MM] Ex. 1998-11-10 or 2015-05-29 11:55</note>
        <validator>
            <regularExpression>^(\d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01]))+([ ]+(\d{2}(:?\d{2})?)?)?$</regularExpression>
        </validator>
    </metadatafield>
    <metadatafield>
        <fieldName>Owner</fieldName>
        <mandatory>true</mandatory>
        <note>The name of the owner, with email and ORCID. The format should be: family, given[, email][, ORCID]. Example: Smith, John, js@acme.org, orcid.org//0000-0002-1825-0097</note>
        <validator>
            <regularExpression>^[a-zA-Z .'-]+, [a-zA-Z .'-]+[, ]*([a-zA-Z0-9_!#$%’*+=?`{|}~^.-]+@[a-zA-Z0-9.-]+)?[, ]*(orcid.org\/\/0000-000(1-[5-9]|2-[0-9]|3-[0-4])\d\d\d-\d\d\d[\dX])?$</regularExpression>
        </validator>
    </metadatafield>
    ....
    <metadatafield>
        <fieldName>License term</fieldName>
        <mandatory>false</mandatory>
        <dataType>Time_Interval</dataType>
        <note>Period of time during which the dataset may be used. Use ISO 8601 Date Format: YYYY-MM-DD[ HH:MM] Ex. 2016-07-31 or 2015-05-10 12:00</note>
        <vocabulary/>
        <validator>
            <regularExpression>^(\d{4}\-(0?[1-9]|1[012])\-(0?[1-9]|[12][0-9]|3[01]))+([ ]+(\d{2}(:?\d{2})?)?)?$</regularExpression>
        </validator>
    </metadatafield>
    <metadatafield>
        <fieldName>Requirement of non-disclosure (confidentiality mark)</fieldName>
        <mandatory>false</mandatory>
        <note>Requirement of non-disclosure (confidentiality mark). Whether the dataset bears confidentiality mark/may be used and shared subject to the obligation of non-disclosure</note>
    </metadatafield>
</metadataformat>


You can find all details about profiles at Item Profile

Namespace Collection

Operation HTTP Method URL Min Required Role
List GET /namespaces Catalogue-Member

An example of given result is:

[
    {
        "id": "contact",
        "title": "Contact Title",
        "name": "contact",
        "description": "This section is about Contact(s)"
    },
    {
        "id": "developer_information",
        "title": "Developer Information",
        "name": "developer_information",
        "description": "This section is about Developer(s)"
    },
    {
        "id": "extra_information",
        "title": "Extras",
        "name": "extra_information",
        "description": "This section is about Extra(s)"
    },
    {
        "id": "ReportDetails",
        "title": "Report Details",
        "name": "ReportDetails",
        "description": "Report Details"
    }
]

License Collection

Operation HTTP Method URL Min Required Role
List GET /licenses Catalogue-Member


An example of given result is:

[
    {
        "status": "active",
        "maintainer": "Creative Commons",
        "od_conformance": "rejected",
        "family": "Creative Commons",
        "osd_conformance": "not reviewed",
        "domain_data": true,
        "title": "Creative Commons Attribution-NonCommercial-ShareAlike 4.0",
        "url": "https://creativecommons.org/licenses/by-nc-sa/4.0/",
        "domain_content": true,
        "is_okd_compliant": false,
        "is_osi_compliant": false,
        "domain_software": false,
        "id": "CC-BY-NC-SA-4.0"
    },
    {
        "status": "active",
        "maintainer": "Creative Commons",
        "od_conformance": "rejected",
        "family": "Creative Commons",
        "osd_conformance": "not reviewed",
        "domain_data": true,
        "title": "Creative Commons Attribution-NonCommercial-NoDerivatives 4.0",
        "url": "https://creativecommons.org/licenses/by-nc-nd/4.0/",
        "domain_content": true,
        "is_okd_compliant": false,
        "is_osi_compliant": false,
        "domain_software": false,
        "id": "CC-BY-NC-ND-4.0"
    },
    ...
    {
        "status": "active",
        "maintainer": "Free Software Foundation",
        "od_conformance": "not reviewed",
        "family": "",
        "osd_conformance": "approved",
        "domain_data": false,
        "title": "GNU Affero General Public License v3",
        "url": "http://www.opensource.org/licenses/AGPL-3.0",
        "domain_content": false,
        "is_okd_compliant": false,
        "is_osi_compliant": true,
        "domain_software": true,
        "id": "AGPL-3.0"
    }
]

Group Collection

Operation HTTP Method URL Min Required Role
List GET /groups
Count GET /groups?count=true
Create POST /groups
Read GET /groups/{NAME}
Update PUT /groups/{NAME}
Patch PATCH /groups/{NAME}
Delete DELETE /groups/{NAME}
Purge PURGE /groups/{NAME}
Purge DELETE /groups/{NAME}?purge=true

Only Catalogue Admin are able to invoke non-safe methods.


A group is mainly described by the following attributes (* indicate mandatory attributes):

  • name* (string): the name of the group, a string between 2 and 100 characters long, containing only lowercase alphanumeric characters, '-' and '_' ;
  • id (string): the id of the group;
  • title (string): the title of the group;
  • description (string): the description of the group;
  • image_url (string): the URL to an image to be displayed on the group’s page;
  • state (string, default: 'active'): the current state of the group, e.g. 'active' or 'deleted', only active groups show up in search results and other lists of groups, this parameter will be ignored if you are not authorized to change the state of the group;
  • extras (list of dataset extra dictionaries): the group’s extras, extras are arbitrary (key: value) metadata items that can be added to groups, each extra dictionary should have keys 'key' (a string), 'value' (a string), and optionally 'deleted'.


Representation

An example of Group representation is:

{
    "name": "group_name",
    "id": "fd72502e-ddfa-4121-a12a-bb42e572d649",
    "title": "Group Title",
    "description": "",
    "image_url": "",
    "state": "active",
    "extras": [],
    "created": "2018-03-22T11:48:42.612063",
    "revision_id": "0b5dda21-9bb0-4993-896b-d865e0d2f1f3"
}

Organization Collection

Operation HTTP Method URL Min Required Role
List GET /organizations
Count GET /organizations?count=true
Create POST /organizations
Read GET /organizations/{NAME}
Update PUT /organizations/{NAME}
Patch PATCH /organizations/{NAME}
Delete DELETE /organizations/{NAME}
Purge PURGE /organizations/{NAME}
Purge DELETE /organizations/{NAME}?purge=true

Only Catalogue Admin are able to invoke non-safe methods.

An Organization is mainly described by the following attributes (* indicate mandatory attributes):


  • name* (string): the name of the organization, a string between 2 and 100 characters long, containing only lowercase alphanumeric characters, '-' and '_';
  • id (string): the id of the organization;
  • title (string): the title of the organization;
  • description (string): the description of the organization;
  • image_url (string): the URL to an image to be displayed on the organization’s page;
  • state (string, default: 'active'): the current state of the organization, e.g. 'active' or 'deleted', only active organizations show up in search results and other lists of organizations, this parameter will be ignored if you are not authorized to change the state of the organization;
  • approval_status (string);
  • extras (list of dataset extra dictionaries): the organization’s extras (optional), extras are arbitrary (key: value) metadata items that can be added to organizations, each extra dictionary should have keys 'key' (a string), 'value' (a string), and optionally 'deleted'.


User Collection

Operation HTTP Method URL Min Required Role
List GET /users
Create POST /users
Read GET /users/{NAME}
Update PUT /users/{NAME}
Delete DELETE /users/{NAME}

Only Catalogue Admin are able to invoke non-safe methods.

A User is mainly described by the following attributes (* indicate mandatory attributes):

  • name* (string): the name of the user, a string between 2 and 100 characters in length, containing only lowercase alphanumeric characters, '-' and '_';
  • id (string): the id of the user;
  • fullname (string): the full name of the user;
  • email (string): the email address for the user;
  • password (string): the password of the user, a string of at least 4 characters (parameter only used to create a new user);
  • about (string): a description of the user.


{
    "name": "luca_frosini",
    "id": " 5f5c2bc0-bcd4-460a-9548-52cd4d33b263",
    "fullname": "Luca Frosini"
}



Dev and Pre Users used for moderation tests

To perform moderation tests in dev and preproduction infrastructure we use different users with the indicated roles.

User Username Role
Mister White mister.white Catalogue-Manager
Mister Blue mister.blue Catalogue-Admin
Mister Orange mister.orange Catalogue-Editor
Mister Pink mister.pink NO ROLE (means Catalogue-Member)
Mister Blonde mister.blonde Catalogue-Admin + Catalogue-Moderator
Mister Brown mister.brown Catalogue-Moderator
  1. Allamaraju S. RESTful Web Services Cookbook: Solutions for Improving Scalability and Simplicity . O’Reilly. first ed. 2010